v1.29: Critical Bug Fix for Backup Resume
commit : f738d268fec21cb697eb460b82240f2c2b0833f5
author : David Steele <[email protected]>
date : Thu, 5 Jul 2018 18:36:34 -0400
committer: David Steele <[email protected]>
date : Thu, 5 Jul 2018 18:36:34 -0400 IMPORTANT NOTE: This release fixes a critical bug in the backup resume feature. All resumed backups prior to this release should be considered inconsistent. A backup will be resumed after a prior backup fails, unless resume=n has been specified. A resumed backup can be identified by checking the backup log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". If the message exists, do not use this backup or any backup in the same set for a restore and check the restore logs to see if a resumed backup was restored. If so, there may be inconsistent data in the cluster.
Bug Fixes:
* Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. See note above for more details. (Reported by David Youatt, Yogesh Sharma, Stephen Frost.)
* Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz.)
* Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
* Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate. (Reported by Brad Nicholson.)
* Fix archive-copy throwing "path not found" error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
* Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
* Fix delta restore failing when a linked file was missing.
* Fix error in selective restore when only one user database exists in the cluster. (Fixed by Cynthia Shang. Reported by Nj Baliyan.)
Improvements:
* Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)
* Improve performance of HTTPS client. Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
* Improve S3 delete performance. The constant S3_BATCH_MAX had been replaced with a hard-coded value of 2, probably during testing.
* Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path. Remove recursive path sync functionality since it is no longer used. M README.md
M doc/release.pl
M doc/resource/exe.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M libc/lib/pgBackRest/LibCAuto.pm
Remove call to lscpu which can vary widely by build host.
commit : cc564ae0aa031abe353cd3354f4be73c6f23b92c
author : David Steele <[email protected]>
date : Thu, 5 Jul 2018 17:05:15 -0400
committer: David Steele <[email protected]>
date : Thu, 5 Jul 2018 17:05:15 -0400 M doc/xml/release.xml
M doc/xml/user-guide.xml
Add zero-length file to mock/all test.
commit : 3afaccf91c3ee8de1daed1ab4b992a4e2052fdec
author : David Steele <[email protected]>
date : Thu, 5 Jul 2018 16:08:21 -0400
committer: David Steele <[email protected]>
date : Thu, 5 Jul 2018 16:08:21 -0400 M doc/xml/release.xml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume.
commit : ab33d30bc48d792cc05358ba1eeba556c6aeef47
author : David Steele <[email protected]>
date : Wed, 4 Jul 2018 10:07:21 -0400
committer: David Steele <[email protected]>
date : Wed, 4 Jul 2018 10:07:21 -0400 Reported by David Youatt, Yogesh Sharma, Stephen Frost. M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Backup/File.pm
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
Add new test for Common::Io::Process to show that output on stderr will raise an exception on close() even if the exit code is 0.
commit : d728050a0129e8c179db5d0a975f06e8cdc63e8d
author : David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:54:33 -0400
committer: David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:54:33 -0400 M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Module/Common/CommonIoProcessTest.pm
Update docs with 32-bit support and caveats.
commit : e8ffefe8862d17b3814d070cb720238c8fe4f91d
author : David Steele <[email protected]>
date : Thu, 5 Jul 2018 10:50:21 -0400
committer: David Steele <[email protected]>
date : Thu, 5 Jul 2018 10:50:21 -0400 32-bit support was added in v1.26.
Reported by Viorel Tabara. M doc/xml/release.xml
M doc/xml/user-guide.xml
Update out-of-date description for the spool-path option.
commit : 0c92430954c496b1cb405c47d0009bf0b1b8b1bd
author : David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:40:37 -0400
committer: David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:40:37 -0400 M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/Config/ConfigHelpData.pm
Clarify that S3 buckets must be created by the user.
commit : 971bd78832ab6a878fc2206339a0edf42fcc2e69
author : David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:35:58 -0400
committer: David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:35:58 -0400 Suggested by David Youatt. M doc/xml/release.xml
M doc/xml/user-guide.xml
Fix error in selective restore when only one user database exists in the cluster.
commit : edc1b11ba77e825e9ed673bc1ae1321a8b69bf0d
author : Cynthia Shang <[email protected]>
date : Wed, 4 Jul 2018 09:29:49 -0400
committer: David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:29:49 -0400 Fixed by Cynthia Shang.
Reported by Nj Baliyan. M doc/xml/release.xml
M lib/pgBackRest/Restore.pm
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory.
commit : b94c75a83779adbc7f6556149e870cff458694e9
author : David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:24:59 -0400
committer: David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:24:59 -0400 Reported by Vitaliy Kukharik. M doc/xml/release.xml
M lib/pgBackRest/Manifest.pm
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
Fix delta restore failing when a linked file was missing.
commit : 66cdc1f1a0fcbd9ff5bf06b79ae715a1424796c0
author : David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:11:28 -0400
committer: David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:11:28 -0400 M doc/xml/release.xml
M lib/pgBackRest/RestoreFile.pm
Fix archive-copy throwing "path not found" error for incr/diff backups.
commit : af7594f47c8e57616081de81a193c5171d19ffc5
author : David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:08:45 -0400
committer: David Steele <[email protected]>
date : Wed, 4 Jul 2018 09:08:45 -0400 Reported by yummyliu, Vitaliy Kukharik. M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate.
commit : ca2f8af196f02f944164e56e990d9d4049eb804b
author : David Steele <[email protected]>
date : Tue, 3 Jul 2018 21:10:44 -0400
committer: David Steele <[email protected]>
date : Tue, 3 Jul 2018 21:10:44 -0400 Reported by Brad Nicholson. M doc/xml/release.xml
M lib/pgBackRest/Config/Data.pm
M src/config/define.auto.c
Add log-level-stderr option for stanza-* commands.
commit : 327e7c58968e2ee92c869bb9e462fba6ce4127f6
author : David Steele <[email protected]>
date : Tue, 3 Jul 2018 20:56:46 -0400
committer: David Steele <[email protected]>
date : Tue, 3 Jul 2018 20:56:46 -0400 M doc/xml/release.xml
M lib/pgBackRest/Config/Data.pm
M src/config/define.auto.c
M test/expect/archive-stop-001.log
M test/expect/archive-stop-002.log
M test/expect/archive-stop-003.log
M test/expect/archive-stop-004.log
M test/expect/archive-stop-005.log
M test/expect/archive-stop-006.log
M test/expect/archive-stop-007.log
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
Improve efficiency of path syncs.
commit : 445d1dcd954a45ec9f09aa3e07e96a0df325675c
author : David Steele <[email protected]>
date : Tue, 3 Jul 2018 14:07:42 -0400
committer: David Steele <[email protected]>
date : Tue, 3 Jul 2018 14:07:42 -0400 * Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
* Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path. Remove recursive path sync functionality since it is no longer used.
* Make backup.history sync more efficient. Only the backup.history/[year] directory was being synced, so check if the backup.history is newly created and sync it as well. M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Restore.pm
M lib/pgBackRest/Storage/Base.pm
M lib/pgBackRest/Storage/Local.pm
M lib/pgBackRest/Storage/Posix/Driver.pm
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
S3 fixes and improvements.
commit : fa4fd3d6e56c413afe389ab53a9c80430794a297
author : David Steele <[email protected]>
date : Tue, 3 Jul 2018 12:20:01 -0400
committer: David Steele <[email protected]>
date : Tue, 3 Jul 2018 12:20:01 -0400 * Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz.)
* Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)
* Improve performance of HTTPS client. Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
* Improve S3 delete performance. The constant S3_BATCH_MAX had been replaced with a hard-coded value of 2, probably during testing. M doc/xml/release.xml
M lib/pgBackRest/Common/Http/Client.pm
M lib/pgBackRest/Common/Io/Buffered.pm
M lib/pgBackRest/Storage/S3/Auth.pm
M lib/pgBackRest/Storage/S3/Driver.pm
M lib/pgBackRest/Version.pm
M libc/lib/pgBackRest/LibCAuto.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Module/Common/CommonHttpClientTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageS3AuthTest.pm
Change backup test user from backrest to pgbackrest.
commit : 8b01e4e27ada2fcfd9100476e8742f79a9d32150
author : David Steele <[email protected]>
date : Tue, 3 Jul 2018 11:09:42 -0400
committer: David Steele <[email protected]>
date : Tue, 3 Jul 2018 11:09:42 -0400 M .travis.yml
M doc/xml/release.xml
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Common/ContainerTest.pm
Make ls ordering deterministic in mock/all test.
commit : dc94a6899eea54e8fb70741c32d4ca8890559f84
author : David Steele <[email protected]>
date : Tue, 3 Jul 2018 11:07:06 -0400
committer: David Steele <[email protected]>
date : Tue, 3 Jul 2018 11:07:06 -0400 M doc/xml/release.xml
M test/expect/mock-all-001.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
Fixes to get CI tests passing.
commit : a52c658c8ff282ae75f33673e641e13a69a5b504
author : David Steele <[email protected]>
date : Tue, 3 Jul 2018 10:18:52 -0400
committer: David Steele <[email protected]>
date : Tue, 3 Jul 2018 10:18:52 -0400 * Disable package build tests since v1 will no longer be packaged. Users installing packages should update to v2. v1 builds are intended for users installing from source.
* Update SSL error message test on CentOS 7. M doc/xml/release.xml
M test/lib/pgBackRestTest/Module/Storage/StorageS3CertTest.pm
M test/travis.pl
Add contributor.
commit : a96e7e02974fee88569c259ff07487505790f006
author : David Steele <[email protected]>
date : Fri, 9 Feb 2018 22:27:21 -0500
committer: David Steele <[email protected]>
date : Fri, 9 Feb 2018 22:27:21 -0500 M doc/xml/release.xml
Update URL for Debian package repository.
commit : f09f5ad90db6ecf1469d5e0d073121e24c51697a
author : David Steele <[email protected]>
date : Fri, 9 Feb 2018 23:23:47 -0500
committer: David Steele <[email protected]>
date : Fri, 9 Feb 2018 23:23:47 -0500 M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
Update path for prior documentation deployment.
commit : e7d42ec3b4ec6601b1358ccfbb5cc3d7354be451
author : David Steele <[email protected]>
date : Thu, 1 Feb 2018 17:49:21 -0500
committer: David Steele <[email protected]>
date : Thu, 1 Feb 2018 17:49:21 -0500 M doc/release.pl