v1.18: Stanza Upgrade, Refactoring, and Locking Improvements
commit : 76161bd8451e581a8f9aa11c1f37d14863585d29
author : David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 19:17:39 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 19:17:39 -0400
Bug Fixes:
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
Features:
* Added the stanza-upgrade command to provide a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. (Contributed by Cynthia Shang.)
* Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section. (Contributed by Cynthia Shang.)
Refactoring:
* Simplify locking scheme. Now, only the master process will hold write locks (for archive-push and backup commands) and not all local and remote worker processes as before.
* Refactor Ini.pm to facilitate testing.
* Do not set timestamps of files in the backup directories to match timestamps in the cluster directory. This was originally done to enable backup resume, but that process is now implemented with checksums.
* Improved error message when the restore command detects the presence of postmaster.pid. (Suggested by Yogesh Sharma.)
* Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. (Suggested by Yogesh Sharma.)
* The backup and restore commands no longer copy via temp files. In both cases the files are checksummed on resume so there's no danger of partial copies.
* Allow functions to accept optional parameters as a hash.
* Refactor File->list() and fileList() to accept optional parameters.
* Refactor backupLabel() and add unit tests.
* Silence some perl critic warnings. (Contributed by Cynthia Shang.)
M README.md
M doc/release.pl
M doc/resource/exe.cache
M doc/xml/index.xml
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M libc/lib/pgBackRest/LibC.pm
Go back to using static user for documentation.
commit : 5c635e0f0a1f5dc8ab8ef8c65204e81d3495afcb
author : David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 18:36:33 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 18:36:33 -0400
Making this dynamic in commit 5d2e792 broke doc builds from cache. The long-term solution is to create a special user for doc builds but that’s beyond the scope of this release.
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/xml/user-guide.xml
Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions.
commit : f207dc71238097a0fded50cc87bb7ef2afb006de
author : David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 12:41:34 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 12:41:34 -0400
Suggested by Yogesh Sharma.
M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Common/Exception.pm
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Config/Config.pm
M lib/pgBackRest/Protocol/CommandMaster.pm
M test/expect/archive-get-001.log
M test/expect/archive-get-002.log
M test/expect/archive-get-003.log
M test/expect/archive-get-004.log
M test/expect/archive-get-005.log
M test/expect/archive-get-006.log
M test/expect/archive-get-007.log
M test/expect/archive-get-008.log
M test/expect/archive-push-001.log
M test/expect/archive-push-002.log
M test/expect/archive-push-003.log
M test/expect/archive-push-004.log
M test/expect/archive-push-005.log
M test/expect/archive-push-006.log
M test/expect/archive-push-007.log
M test/expect/archive-push-008.log
M test/expect/archive-stop-001.log
M test/expect/archive-stop-002.log
M test/expect/archive-stop-003.log
M test/expect/archive-stop-004.log
M test/expect/archive-stop-005.log
M test/expect/archive-stop-006.log
M test/expect/full-real-001.log
M test/expect/full-real-007.log
M test/expect/full-synthetic-001.log
M test/expect/full-synthetic-002.log
M test/expect/full-synthetic-003.log
M test/expect/full-synthetic-004.log
M test/expect/full-synthetic-005.log
M test/expect/full-synthetic-006.log
M test/expect/full-synthetic-007.log
M test/expect/full-synthetic-008.log
M test/expect/stanza-create-001.log
M test/expect/stanza-create-002.log
M test/expect/stanza-upgrade-001.log
M test/expect/stanza-upgrade-002.log
Improved error message when the restore command detects the presence of postmaster.pid.
commit : f5f8d51366faae6ac31f355d021eaff67bef8f7f
author : David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 09:53:07 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 09:53:07 -0400
Suggested by Yogesh Sharma.
M doc/xml/release.xml
M lib/pgBackRest/Restore.pm
M test/expect/full-synthetic-001.log
M test/lib/pgBackRestTest/Full/FullSyntheticTest.pm
Do not set timestamps of files in the backup directories to match timestamps in the cluster directory.
commit : b0119914d4633431b208a5a327a3758a06258087
author : David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 09:22:51 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 09:22:51 -0400
This was originally done to enable backup resume, but that process is now implemented with checksums.
M doc/xml/release.xml
M lib/pgBackRest/Backup.pm
M lib/pgBackRest/BackupFile.pm
Backup and restore commands no longer copy via temp files.
commit : fe02b40293fec08c819c5f41110203b8919fbae4
author : David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 09:05:57 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 12 Apr 2017 09:05:57 -0400
In both cases the files are checksummed on resume so there's no danger of partial copies.
M doc/xml/release.xml
M lib/pgBackRest/BackupFile.pm
M lib/pgBackRest/File.pm
M lib/pgBackRest/FileCommon.pm
M lib/pgBackRest/RestoreFile.pm
M test/expect/archive-get-002.log
M test/expect/archive-get-004.log
M test/expect/archive-get-006.log
M test/expect/archive-get-008.log
M test/expect/archive-push-001.log
M test/expect/archive-push-003.log
M test/expect/archive-push-005.log
M test/expect/archive-push-007.log
M test/expect/full-synthetic-001.log
M test/expect/full-synthetic-002.log
M test/expect/full-synthetic-003.log
M test/expect/full-synthetic-004.log
M test/expect/full-synthetic-005.log
M test/expect/full-synthetic-006.log
M test/expect/full-synthetic-007.log
M test/expect/full-synthetic-008.log
M test/expect/stanza-create-001.log
M test/expect/stanza-create-002.log
Silence some perl critic warnings.
commit : 21a000d002bae2fa87cd09952740cbe14770aada
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Mon, 10 Apr 2017 17:23:39 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 10 Apr 2017 17:23:39 -0400
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/lib/BackRestDoc/Common/DocRender.pm
M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/xml/release.xml
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/File.pm
M lib/pgBackRest/Info.pm
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Expire/ExpireEnvTest.pm
M test/lint/perlcritic.policy
Ignore clock skew in container libc/package builds using make.
commit : 8546571461215eb65ffcff649c6baa17bc57d605
author : David Steele <david@pgmasters.net>
date : Mon, 10 Apr 2017 17:20:21 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 10 Apr 2017 17:20:21 -0400
It is common for containers to have clock skew so the build process takes care of this issue independently.
M doc/xml/release.xml
M test/test.pl
Added ability to test warning messages.
commit : b63ede5614fed0dce32449f1450b7a5627eeeb3d
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Mon, 10 Apr 2017 13:53:19 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 10 Apr 2017 13:53:19 -0400
Contributed by Cynthia Shang.
M doc/xml/release.xml
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Config/Config.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Config/ConfigUnitTest.pm
Improvements to Ini.pm.
commit : 3d84f2ce5e02c5bd7f9bdcbb3b4579b146bf7ccf
author : David Steele <david@pgmasters.net>
date : Mon, 10 Apr 2017 13:24:45 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 10 Apr 2017 13:24:45 -0400
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/xml/release.xml
M lib/pgBackRest/Archive/ArchiveInfo.pm
M lib/pgBackRest/Backup.pm
M lib/pgBackRest/BackupInfo.pm
M lib/pgBackRest/Common/Ini.pm
M lib/pgBackRest/Config/Config.pm
M lib/pgBackRest/Manifest.pm
M lib/pgBackRest/Restore.pm
M lib/pgBackRest/Stanza.pm
M test/expect/full-synthetic-001.log
M test/expect/full-synthetic-002.log
M test/expect/full-synthetic-003.log
M test/expect/full-synthetic-004.log
M test/expect/full-synthetic-005.log
M test/expect/full-synthetic-006.log
M test/expect/full-synthetic-007.log
M test/expect/full-synthetic-008.log
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Common/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Config/ConfigConfigTest.pm
M test/lib/pgBackRestTest/Full/FullSyntheticTest.pm
A test/lib/pgBackRestTest/Info/InfoIniUnitTest.pm
Coverage testing always enabled on Debian-based containers.
commit : 367e06f4be9133f399d4b7380ace8c5ca087c4b6
author : David Steele <david@pgmasters.net>
date : Mon, 10 Apr 2017 12:31:30 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 10 Apr 2017 12:31:30 -0400
* Full coverage is verified when specified.
* Modules marked with partial coverage will error if they are actually fully covered.
* Simplified test representation is DefineTest.
* Added new representation for queries in DefineTest and added API functions.
* Update modules using DefineTest to use new API.
M .travis.yml
M doc/xml/release.xml
M lib/pgBackRest/Common/String.pm
M test/Vagrantfile
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/test.pl
Fixed locking bug and refactoring.
commit : eb4918d01881c20448e37da4d931d6e8a19f659a
author : David Steele <david@pgmasters.net>
date : Fri, 7 Apr 2017 10:31:49 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 7 Apr 2017 10:31:49 -0400
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
* Simplify locking scheme. Now, only the master process will hold write locks (archive-push, backup) and not all the local and remote worker processes as before.
M bin/pgbackrest
M doc/xml/release.xml
M lib/pgBackRest/Archive/ArchivePushAsync.pm
M lib/pgBackRest/Common/Lock.pm
M lib/pgBackRest/Protocol/LocalMinion.pm
M lib/pgBackRest/Protocol/Protocol.pm
M test/expect/archive-get-001.log
M test/expect/archive-get-002.log
M test/expect/archive-get-003.log
M test/expect/archive-get-004.log
M test/expect/archive-get-005.log
M test/expect/archive-get-006.log
M test/expect/archive-get-007.log
M test/expect/archive-get-008.log
M test/expect/archive-push-001.log
M test/expect/archive-push-003.log
M test/expect/archive-push-005.log
M test/expect/archive-push-007.log
M test/expect/archive-stop-004.log
M test/expect/archive-stop-006.log
M test/expect/full-synthetic-001.log
M test/expect/full-synthetic-002.log
M test/expect/full-synthetic-003.log
M test/expect/full-synthetic-004.log
M test/expect/full-synthetic-005.log
M test/expect/full-synthetic-006.log
M test/expect/full-synthetic-007.log
M test/expect/full-synthetic-008.log
M test/expect/stanza-create-001.log
M test/expect/stanza-create-002.log
M test/lib/pgBackRestTest/Archive/ArchivePushUnitTest.pm
Update wording for release note sections.
commit : edfb0962b678a5ebb3021d37d487fafb843dbf2a
author : David Steele <david@pgmasters.net>
date : Thu, 6 Apr 2017 21:47:43 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 6 Apr 2017 21:47:43 -0400
M doc/lib/BackRestDoc/Custom/DocCustomRelease.pm
M doc/xml/release.xml
Improvements and additional tests for stanza-upgrade.
commit : b620e6b282d3fc0bd1354b9490fbd84ee8d22d6e
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Thu, 6 Apr 2017 19:36:37 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 6 Apr 2017 19:36:37 -0400
Contributed by Cynthia Shang.
M doc/xml/user-guide.xml
M lib/pgBackRest/Backup.pm
M lib/pgBackRest/BackupInfo.pm
M lib/pgBackRest/Expire.pm
M test/expect/expire-expire-001.log
M test/expect/expire-expire-002.log
M test/expect/full-synthetic-001.log
M test/expect/full-synthetic-002.log
M test/expect/full-synthetic-003.log
M test/expect/full-synthetic-004.log
M test/expect/full-synthetic-005.log
M test/expect/full-synthetic-006.log
M test/expect/full-synthetic-007.log
M test/expect/full-synthetic-008.log
M test/expect/stanza-upgrade-001.log
M test/expect/stanza-upgrade-002.log
A test/lib/pgBackRestTest/Backup/BackupInfoUnitTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Expire/ExpireExpireTest.pm
M test/lib/pgBackRestTest/Stanza/StanzaUnitTest.pm
M test/lib/pgBackRestTest/Stanza/StanzaUpgradeTest.pm
Require description in every call to testResult() and make iWaitSeconds optional.
commit : 0da216c034d484927fa8ad0845e08f25c5227b5a
author : David Steele <david@crunchydata.com>
date : Wed, 5 Apr 2017 10:50:32 -0400
committer: David Steele <david@crunchydata.com>
date : Wed, 5 Apr 2017 10:50:32 -0400
M doc/xml/release.xml
M test/lib/pgBackRestTest/Archive/ArchivePushTest.pm
M test/lib/pgBackRestTest/Archive/ArchiveStopTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/File/FileOwnerTest.pm
M test/lib/pgBackRestTest/File/FileUnitTest.pm
Refactor backupLabel() and add unit tests.
commit : c7b61a200258b2f10d1c62f2aedc7441e27bb6b1
author : David Steele <david@crunchydata.com>
date : Wed, 5 Apr 2017 10:16:16 -0400
committer: David Steele <david@crunchydata.com>
date : Wed, 5 Apr 2017 10:16:16 -0400
M bin/pgbackrest
M doc/xml/release.xml
M lib/pgBackRest/Backup.pm
M lib/pgBackRest/BackupCommon.pm
M test/lib/pgBackRestTest/Backup/BackupUnitTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
Allow functions to accept optional parameters as a hash.
commit : b2f43b56014f3d43404a17531887a09a78eb0db1
author : David Steele <david@pgmasters.net>
date : Tue, 4 Apr 2017 21:17:19 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 4 Apr 2017 21:17:19 -0400
Refactor File->list() and fileList() to accept optional parameters.
M doc/xml/release.xml
M lib/pgBackRest/Archive/ArchiveCommon.pm
M lib/pgBackRest/Archive/ArchiveInfo.pm
M lib/pgBackRest/Archive/ArchivePush.pm
M lib/pgBackRest/Backup.pm
M lib/pgBackRest/BackupInfo.pm
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Expire.pm
M lib/pgBackRest/File.pm
M lib/pgBackRest/FileCommon.pm
M lib/pgBackRest/Info.pm
M lib/pgBackRest/Stanza.pm
M test/lib/pgBackRestTest/Archive/ArchiveStopTest.pm
M test/lib/pgBackRestTest/Common/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/File/FileListTest.pm
Fixed flapping archive stop tests.
commit : 9eae36a7028a7c08869ec2e7c2f31f0aa49ecb4c
author : David Steele <david@pgmasters.net>
date : Tue, 4 Apr 2017 21:15:22 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 4 Apr 2017 21:15:22 -0400
M doc/xml/release.xml
M test/lib/pgBackRestTest/Archive/ArchiveStopTest.pm
Added stanza-upgrade command.
commit : beead043ac1f988e3757dc8689638ee5d443d9f9
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Mon, 3 Apr 2017 10:42:55 -0400
committer: David Steele <david@crunchydata.com>
date : Mon, 3 Apr 2017 10:42:55 -0400
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.
Contributed by Cynthia Shang.
M bin/pgbackrest
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Archive/ArchiveInfo.pm
M lib/pgBackRest/BackupInfo.pm
M lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Config/Config.pm
M lib/pgBackRest/Config/ConfigHelpData.pm
M lib/pgBackRest/Expire.pm
M lib/pgBackRest/Info.pm
A lib/pgBackRest/InfoCommon.pm
M lib/pgBackRest/Stanza.pm
A test/data/backup.pg_control_92.bin
M test/data/backup.pg_control_93.bin
A test/data/backup.pg_control_95.bin
A test/data/backup.wal1_92.bin
A test/data/backup.wal1_93.bin
A test/data/backup.wal1_95.bin
M test/expect/expire-expire-001.log
A test/expect/expire-expire-002.log
M test/expect/full-real-001.log
M test/expect/full-real-007.log
M test/expect/full-synthetic-001.log
M test/expect/full-synthetic-002.log
M test/expect/full-synthetic-003.log
M test/expect/full-synthetic-004.log
M test/expect/full-synthetic-005.log
M test/expect/full-synthetic-006.log
M test/expect/full-synthetic-007.log
M test/expect/full-synthetic-008.log
M test/expect/help-help-001.log
M test/expect/stanza-create-001.log
M test/expect/stanza-create-002.log
A test/expect/stanza-upgrade-001.log
A test/expect/stanza-upgrade-002.log
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/Env/EnvHostTest.pm
M test/lib/pgBackRestTest/Common/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Expire/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Expire/ExpireExpireTest.pm
M test/lib/pgBackRestTest/Full/FullRealTest.pm
M test/lib/pgBackRestTest/Full/FullSyntheticTest.pm
M test/lib/pgBackRestTest/Info/InfoUnitTest.pm
M test/lib/pgBackRestTest/Stanza/StanzaCreateTest.pm
A test/lib/pgBackRestTest/Stanza/StanzaUnitTest.pm
A test/lib/pgBackRestTest/Stanza/StanzaUpgradeTest.pm
Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section.
commit : af7e4f4b4b6070e566865872e5240b824a14983f
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Wed, 22 Mar 2017 19:03:28 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 22 Mar 2017 19:03:28 -0400
Contributed by Cynthia Shang.
M doc/xml/release.xml
M lib/pgBackRest/Config/Config.pm
M lib/pgBackRest/Version.pm
M libc/lib/pgBackRest/LibC.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/lib/pgBackRestTest/Config/ConfigUnitTest.pm
Fixed user change in docs missed in vagrant upgrade.
commit : 5d2e79242da79854ee922c1a84dec0e67d52f090
author : David Steele <david@pgmasters.net>
date : Thu, 16 Mar 2017 19:50:59 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 16 Mar 2017 19:50:59 -0400
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/xml/user-guide.xml
Updated vagrant to new version and image.
commit : cbd879a9830d2585b3df3209f59c4a9581a3c166
author : David Steele <david@pgmasters.net>
date : Tue, 14 Mar 2017 22:05:17 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 14 Mar 2017 22:05:17 -0400
M .gitignore
A doc/.gitignore
M doc/xml/release.xml
A test/.gitignore
M test/Vagrantfile