pgBackRest 1.13 commit log

v1.13: Parallel Archiving, Stanza Create, Improved Info and Check

  
commit   : 41eba95155b1fab4c5f98dab3327c1563968fcab    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 20:23:03 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 20:23:03 -0500    

Click here for diff

  
IMPORTANT NOTE: The new implementation of asynchronous archiving no longer copies WAL to a separate queue. If there is any WAL left over in the old queue after upgrading to 1.13, it will be abandoned and not pushed to the repository.  
  
To prevent this outcome, stop archiving by setting archive_command = false. Next, drain the async queue by running pgbackrest --stanza=[stanza-name] archive-push and wait for the process to complete. Check that the queue in [spool-path]/archive/[stanza-name]/out is empty. Finally, install 1.13 and restore the original archive_command.  
  
IMPORTANT NOTE: The stanza-create command is not longer optional and must be executed before backup or archiving can be performed on a new stanza. Pre-existing stanzas do not require stanza-create to be executed.  
  
Bug Fixes:  
  
* Fixed const assignment giving compiler warning in C library. (Fixed by Adrian Vondendriesch.)  
* Fixed a few directory syncs that were missed for the --repo-sync option.  
* Fixed an issue where a missing user/group on restore could cause an "uninitialized value" error in File->owner(). (Reported by Leonardo Avellar.)  
* Fixed an issue where protocol mismatch errors did not output the expected value.  
* Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination.  
  
Features:  
  
* Improved, multi-process implementation of asynchronous archiving.  
* Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. (Contributed by Cynthia Shang.)  
* Improved check command to run on a standby, though only basic checks are done because pg_switch_xlog() cannot be executed on a replica. (Contributed by Cynthia Shang.)  
* Added archive and backup WAL ranges to the info command.  
* Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2. (Contributed by blogh.)  
* Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations. (Suggested by Michael Vitale.)  
  
Refactoring:  
  
* Refactor File and BackupCommon modules to improve test coverage.  
* Return proper error code when unable to convert a relative path to an absolute path. (Suggested by Yogesh Sharma.)  
* Log file banner is not output until the first log entry is written. (Suggested by Jens Wilke.)  
* Moved File->manifest() into the FileCommon.pm module.  
* Moved the Archive modules to the Archive directory and split the archive-get and archive-push commands into separate modules.  
* Split the check command out of the Archive.pm module.  
* Allow logging to be suppressed via logDisable() and logEnable().  
* Allow for locks to be taken more than once in the same process without error.  
* Lock directories can be created when more than one directory level is required.  
* Clean up optionValid()/optionTest() logic in Lock.pm.  
* Added Exception::exceptionCode() and Exception::exceptionMessage() to simplify error handling logic.  
* Represent .gz extension with a constant.  
* Allow empty files to be created with FileCommon::fileStringWrite() and use temp files to avoid partial reads.  
* Refactor process IO and process master/minion code out from the common protocol code.  
* Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.  
* Remove Intel-specific optimization from C library build flags. (Contributed by Adrian Vondendriesch.)  
* Removed --lock option. This option was introduced before the lock directory could be located outside the repository and is now obsolete.  
* Added --log-timestamp option to allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.  
* Fixed alignment issues with multiline logging.  
  

Fixed alignment issues with multiline logging.

  
commit   : 0daeace1cd2e2ef55c3df3e43b4b622418fe625b    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 19:58:52 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 19:58:52 -0500    

Click here for diff

  
  

Added –log-timestamp option.

  
commit   : cded35791d160a6fc2f99daaaaabf0db21c9bb38    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 10:59:49 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 10:59:49 -0500    

Click here for diff

  
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.  
  

Log file banner is not output until the first log entry is written.

  
commit   : 55feae645e92eda928ad0eb5be74431807f3c604    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 10:37:54 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 10:37:54 -0500    

Click here for diff

  
Suggested by Jens Wilke.  
  

Added documentation for parallel archive-push.

  
commit   : 2237c3fc1bafd620f5cafda4d38acbfc7b490aa5    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 10:35:53 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 10:35:53 -0500    

Click here for diff

  
  

Added archive and backup WAL ranges to the info command.

  
commit   : 8b5f37c2a87b221fe40f1883f6583ee31ad977d7    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 10:31:29 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 5 Feb 2017 10:31:29 -0500    

Click here for diff

  
  

Fixed –no-online tests to suppress expected errors.

  
commit   : 1666ee177f059c3566e531fa697cb229eb68c0c4    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 2 Feb 2017 20:44:42 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 2 Feb 2017 20:44:42 -0500    

Click here for diff

  
  

Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations.

  
commit   : efdf64506ae075cadc24de0805c8238478dfd28f    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 2 Feb 2017 20:42:45 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 2 Feb 2017 20:42:45 -0500    

Click here for diff

  
Suggested by Michael Vitale.  
  

Removed –lock option.

  
commit   : 3e54aad88f473db619dc7546cf144c3987224943    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 31 Jan 2017 10:38:45 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 31 Jan 2017 10:38:45 -0500    

Click here for diff

  
This option was introduced before the lock directory could be located outside the repository and is now obsolete.  
  

Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination.

  
commit   : 7418559a6751959727836e462002a14c41461198    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 31 Jan 2017 10:36:59 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 31 Jan 2017 10:36:59 -0500    

Click here for diff

  
  

Remove Intel-specific optimization from C library build flags.

  
commit   : 9b92edddbe506c3ebaa6f970e62270facdcc5a27    
  
author   : Adrian Vondendriesch <adrian.vondendriesch@credativ.de>    
date     : Mon, 30 Jan 2017 17:06:35 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 17:06:35 -0500    

Click here for diff

  
Contributed by Adrian Vondendriesch.  
  

Missed expect logs from commit 455c604.

  
commit   : eb1a1e53080375d947b94e58f917bc2816528264    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 16:22:12 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 16:22:12 -0500    

Click here for diff

  
  

Fixed minor typos.

  
commit   : 34e4cfaeec473ba1c90f8c203f82b1146510deee    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 15:48:10 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 15:48:10 -0500    

Click here for diff

  
  

C Library builds immune to clock differences between VM and host.

  
commit   : 455c6048ee4794a2d334067e4651460b3ad2b416    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 14:00:40 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 14:00:40 -0500    

Click here for diff

  
  

Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.

  
commit   : 6b99d770b5203680ac53813d9c3c73f476d895bb    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 13:59:00 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 30 Jan 2017 13:59:00 -0500    

Click here for diff

  
  

Improved, multi-process implementation of asynchronous archiving.

  
commit   : 43ada9fb33b51479b959ea6a93fcb53fa0803e25    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 11:02:27 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 11:02:27 -0500    

Click here for diff

  
  

Refactor process IO and process master/minion code out from the common protocol code.

  
commit   : 2a9ba877e1dbce234f1b44d8f3163ce38f9c1781    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 10:06:16 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 10:06:16 -0500    

Click here for diff

  
  

Allow empty files to be created with FileCommon::fileStringWrite() and use temp files to avoid partial reads.

  
commit   : 0163d3b3e02f22179f9854246e6f5009e8ff4e40    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 10:04:41 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 10:04:41 -0500    

Click here for diff

  
  

Represent .gz extension with a constant.

  
commit   : 2553c7a9982f630226923ec14b0f56350607db44    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 10:02:00 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 10:02:00 -0500    

Click here for diff

  
  

Added Exception::exceptionCode() and Exception::exceptionMessage() to simplify error handling logic.

  
commit   : f3ccf3b897e9bab58e2ab1f15dabf54fef9b34db    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 09:56:26 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 09:56:26 -0500    

Click here for diff

  
  

Various improvements in Lock.pm:

  
commit   : 6620594b7724f5950b20c1c103a2b0fee194e816    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 09:53:46 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 09:53:46 -0500    

Click here for diff

  
* Allow for locks to be taken more than once in the same process without error.  
* Lock directories can be created when more than one directory level is required.  
* Clean up optionValid()/optionTest() logic in Lock.pm.  
  

Various improvements to the test suite:

  
commit   : 18db178ac3a03540bcaf4bd5c23f6a34ab419f55    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 09:42:30 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 27 Jan 2017 09:42:30 -0500    

Click here for diff

  
* Allow logging to be suppressed via logDisable() and logEnable().  
* Added more flexibility in initializing and cleaning up after modules and tests.  
* testResult() suppresses logging and reports exceptions.  
* testException() allows messages to be matched with regular expressions.  
* Refactor name/locations of common modules that setup test environments.  
  

Fixed an issue where protocol mismatch errors did not output the expected value.

  
commit   : e876a222fcb68fb4a95455113d3378d5e6b81c7a    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 24 Jan 2017 14:58:55 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 24 Jan 2017 14:58:55 -0500    

Click here for diff

  
  

Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2.

  
commit   : b6fa146f9b1b9e622f456d73585423538c04ad17    
  
author   : Benoit Lobréau (blogh) <blo.capdata@gmail.com>    
date     : Sun, 22 Jan 2017 19:29:56 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 22 Jan 2017 19:29:56 -0500    

Click here for diff

  
Contributed by blogh.  
  

Split the archive-get and archive-push commands into separate modules.

  
commit   : 7d1347cdec4f4c563d5a6fa0a1d66297a2742f0f    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 19:59:32 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 19:59:32 -0500    

Click here for diff

  
  

Split the check command out of the Archive.pm module.

  
commit   : 155c8081632d1081da8b7e94ebd3a7d422f0f280    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 19:54:51 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 19:54:51 -0500    

Click here for diff

  
  

Moved the Archive modules to the Archive directory.

  
commit   : b58045f5757b03a1e15e1955402917b9003c5692    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 19:51:20 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 19:51:20 -0500    

Click here for diff

  
  

C Library builds only run when C library has actually changed.

  
commit   : 2ab63f27b23b8932e9685c5179284fddbb6cf49c    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 13:12:53 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 13:12:53 -0500    

Click here for diff

  
  

Moved File->manifest() into the FileCommon.pm module.

  
commit   : 0b4055e588331ec70d789ced6e5685f54def197b    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 13:08:03 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Jan 2017 13:08:03 -0500    

Click here for diff

  
  

Added integration for testing coverage with Devel::Cover.

  
commit   : 0e4f51c27110f2d3eb6f3d1d15dc3f8ac4675e34    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 9 Jan 2017 20:49:04 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 9 Jan 2017 20:49:04 -0500    

Click here for diff

  
  

Fixed an issue where a missing user/group on restore could cause an “uninitialized value” error in File->owner().

  
commit   : 39744a4f1dbbe4e593bebaa37a6be93dae0da214    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 5 Jan 2017 15:19:56 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 5 Jan 2017 15:19:56 -0500    

Click here for diff

  
Reported by Leonardo Avellar.  
  

Return proper error code when unable to convert a relative path to an absolute path.

  
commit   : 4ff2714bf00063534ddf90d4995f9f1abcb60c99    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 18:39:50 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 18:39:50 -0500    

Click here for diff

  
Suggested by Yogesh Sharma.  
  

Moved tests into new modules to better reflect actual usage.

  
commit   : c8dfc673b87bc04f87e7e3c01f5e227a71f415d8    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 18:31:16 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 18:31:16 -0500    

Click here for diff

  
  

Allow multiple –module, –test, and –run options to be used for test.pl.

  
commit   : e3d63c0252bfbd0cccb746d405c7e3cdeb8f75c1    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 18:29:13 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 18:29:13 -0500    

Click here for diff

  
  

Improvements to stanza-create:

  
commit   : be337c951aee0121dc3d1a251bdb5218d72769a0    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 4 Jan 2017 10:12:29 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 10:12:29 -0500    

Click here for diff

  
* Force only required when a change is needed for backup.info or archive.info.  
* Identical files are not overwritten.  
  

Added expect log expression to replace year subdirectories in backup.history.

  
commit   : b91ca113298104a3c718db93a1dad0fa7336c8c6    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 10:09:21 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 10:09:21 -0500    

Click here for diff

  
  

Update LICENSE.txt for 2017.

  
commit   : 314ef2c8b317c7d56ae2770231e49748183fdd96    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 10:05:43 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 4 Jan 2017 10:05:43 -0500    

Click here for diff

  
  

Added documentation to the User Guide for the process-max option.

  
commit   : d5ee7d3535ea3961db2cb73120ee94632463e737    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Fri, 23 Dec 2016 16:50:02 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 23 Dec 2016 16:50:02 -0500    

Click here for diff

  
Contributed by Cynthia Shang.  
  

Refactor File module to improve test coverage.

  
commit   : 83beab7ec368ffebb5d8526e06eb65a9abd568c2    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 23 Dec 2016 11:43:26 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 23 Dec 2016 11:43:26 -0500    

Click here for diff

  
Added unit tests for low-level functions in the BackupCommon module.  
  

Refactor File module to improve test coverage.

  
commit   : 5d3473b52dfe1f67d557d434a9ee373b6317dcfc    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 23 Dec 2016 08:30:34 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 23 Dec 2016 08:30:34 -0500    

Click here for diff

  
  

Split test modules into separate files.

  
commit   : 6b2666a9d7ee179e237a4ad34117024f2ed62f7f    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 23 Dec 2016 08:22:59 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 23 Dec 2016 08:22:59 -0500    

Click here for diff

  
Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.  
  

Improved check command to run on a standby.

  
commit   : 56144c99c0570daf7f1dba85968b79cbc3a84b15    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 21 Dec 2016 14:35:20 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 21 Dec 2016 14:35:20 -0500    

Click here for diff

  
Only basic checks are done because pg_switch_xlog() cannot be executed on a replica.  
  
Contributed by Cynthia Shang.  
  

Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory.

  
commit   : 974a02dbe41372d28d2c234ebea8a8ac6352865a    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Tue, 20 Dec 2016 16:52:20 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 20 Dec 2016 16:52:20 -0500    

Click here for diff

  
Contributed by Cynthia Shang.  
  

Fixed a few directory syncs that were missed for the –repo-sync option.

  
commit   : aecc0f737e40dfb7304753e37612b11402e4c51c    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 19 Dec 2016 14:40:22 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 19 Dec 2016 14:40:22 -0500    

Click here for diff

  
  

Fixed const assignment giving compiler warning in C library.

  
commit   : 4e1b74688df11f31e2d5caf76bf7c1d53623ae45    
  
author   : Adrian Vondendriesch <adrian.vondendriesch@credativ.de>    
date     : Tue, 13 Dec 2016 08:22:36 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 13 Dec 2016 08:22:36 -0500    

Click here for diff

  
Fixed by Adrian Vondendriesch.