v1.25: S3 Performance Improvements
commit : 85b35f30a4a4ea07e92ad5ded6ce73bc2937f0ef
author : David Steele <david@pgmasters.net>
date : Tue, 24 Oct 2017 17:36:21 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 24 Oct 2017 17:36:21 -0400
Bug Fixes:
* Fix custom settings for compress-level option being ignored. (Reported by Jens Wilke.)
* Remove error when overlapping timelines are detected. Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios. (Reported by blogh.)
* Fix instances where database-id was not rendered as an integer in JSON info output. (Fixed by Cynthia Shang. Reported by Jason O'Donnell.)
Features:
* Improve performance of list requests on S3. Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting. (Suggested by Mihail Shvein.)
M README.md
M doc/resource/exe.cache
M doc/xml/release.xml
M lib/pgBackRest/Config/ConfigHelpData.pm
M lib/pgBackRest/Version.pm
Allow functions with sensitive options to be logged at debug level with redactions.
commit : 8674a4f7aea3bea2db7e92be599c034170b81706
author : David Steele <david@pgmasters.net>
date : Tue, 24 Oct 2017 12:35:36 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 24 Oct 2017 12:35:36 -0400
Previously, functions with sensitive options had to be logged at trace level to avoid exposing them. Trace level logging may still expose secrets so use with caution.
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Config/Config.pm
M lib/pgBackRest/Storage/Posix/Driver.pm
M lib/pgBackRest/Storage/S3/Auth.pm
M lib/pgBackRest/Storage/S3/Request.pm
M test/expect/archive-stop-007.log
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
Improve performance of test code.
commit : b2cc65d886a5c1fe4845f655c53b0c809beefb72
author : David Steele <david@pgmasters.net>
date : Tue, 24 Oct 2017 10:35:42 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 24 Oct 2017 10:35:42 -0400
Wait when all tests have been assigned to reduce CPU load.
M doc/xml/release.xml
M test/test.pl
Replace dynamically built class hierarchies in I/O layer with fixed parent() calls.
commit : d989cf8ac21e0978174cc94747965743f252a45d
author : David Steele <david@pgmasters.net>
date : Sun, 22 Oct 2017 19:07:17 -0400
committer: David Steele <david@pgmasters.net>
date : Sun, 22 Oct 2017 19:07:17 -0400
M doc/xml/release.xml
M lib/pgBackRest/Backup/Filter/PageChecksum.pm
M lib/pgBackRest/Common/Http/Client.pm
M lib/pgBackRest/Common/Io/Base.pm
M lib/pgBackRest/Common/Io/Buffered.pm
A lib/pgBackRest/Common/Io/Filter.pm
M lib/pgBackRest/Common/Io/Process.pm
M lib/pgBackRest/Protocol/Storage/File.pm
M lib/pgBackRest/Storage/Filter/Gzip.pm
M lib/pgBackRest/Storage/Filter/Sha.pm
M lib/pgBackRest/Storage/Posix/FileRead.pm
M lib/pgBackRest/Storage/Posix/FileWrite.pm
M lib/pgBackRest/Storage/S3/FileRead.pm
M lib/pgBackRest/Storage/S3/FileWrite.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Module/Archive/ArchivePushTest.pm
M test/lib/pgBackRestTest/Module/Common/CommonIoHandleTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageFilterShaTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageS3Test.pm
Fix custom settings for compress-level option being ignored.
commit : d82fe88031a3c9ca88bd70652bc4f9fc0e12d4bf
author : David Steele <david@pgmasters.net>
date : Sun, 22 Oct 2017 18:05:46 -0400
committer: David Steele <david@pgmasters.net>
date : Sun, 22 Oct 2017 18:05:46 -0400
Reported by Jens Wilke.
M doc/xml/release.xml
M lib/pgBackRest/Archive/Push/Async.pm
M lib/pgBackRest/Archive/Push/File.pm
M lib/pgBackRest/Archive/Push/Push.pm
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Backup/File.pm
M test/expect/archive-stop-001.log
M test/expect/archive-stop-002.log
M test/expect/archive-stop-003.log
M test/expect/archive-stop-004.log
M test/expect/archive-stop-005.log
M test/expect/archive-stop-006.log
M test/expect/archive-stop-007.log
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageFilterGzipTest.pm
Add I/O performance tests.
commit : 3553ccae449b7b9ad71290234e55c57d9c9455a5
author : David Steele <david@pgmasters.net>
date : Sun, 22 Oct 2017 13:27:47 -0400
committer: David Steele <david@pgmasters.net>
date : Sun, 22 Oct 2017 13:27:47 -0400
M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/lib/pgBackRestTest/Module/Performance/PerformanceIoTest.pm
Run all tests on tempfs rather than local disk.
commit : 9b98c2e7d6492b626267ffa0c50c67922dd43bda
author : David Steele <david@pgmasters.net>
date : Sun, 22 Oct 2017 13:11:03 -0400
committer: David Steele <david@pgmasters.net>
date : Sun, 22 Oct 2017 13:11:03 -0400
M doc/xml/release.xml
M test/Vagrantfile
M test/test.pl
Improve labeling for errors in helper processes.
commit : e1cb9ae86a596d0c1e833ba60730e65805c4ef4f
author : David Steele <david@pgmasters.net>
date : Sat, 21 Oct 2017 14:40:14 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 21 Oct 2017 14:40:14 -0400
M doc/xml/release.xml
M lib/pgBackRest/Common/Io/Process.pm
M lib/pgBackRest/Protocol/Base/Master.pm
M lib/pgBackRest/Protocol/Local/Master.pm
M lib/pgBackRest/Protocol/Remote/Master.pm
M test/expect/archive-stop-001.log
M test/expect/archive-stop-002.log
M test/expect/archive-stop-003.log
M test/expect/archive-stop-004.log
M test/expect/archive-stop-005.log
M test/expect/archive-stop-006.log
M test/expect/archive-stop-007.log
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Archive/ArchivePushTest.pm
M test/lib/pgBackRestTest/Module/Protocol/ProtocolHelperTest.pm
Improve protocol error handling.
commit : 44c7c0475ad30873075751ea23083861fe53278a
author : David Steele <david@pgmasters.net>
date : Sat, 21 Oct 2017 14:11:33 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 21 Oct 2017 14:11:33 -0400
In particular, "stop" errors are no longer reported as "unexpected".
M bin/pgbackrest
M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Common/Io/Process.pm
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Protocol/Base/Master.pm
M lib/pgBackRest/Protocol/Base/Minion.pm
M test/expect/mock-all-002.log
Improve performance of list requests on S3.
commit : 1f120f3fce1f8f30739a0cc178e8eea2053fd2f2
author : David Steele <david@pgmasters.net>
date : Fri, 20 Oct 2017 14:10:16 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 20 Oct 2017 14:10:16 -0400
Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting.
Suggested by Mihail Shvein.
M doc/xml/release.xml
M lib/pgBackRest/Common/String.pm
M lib/pgBackRest/Storage/Local.pm
M lib/pgBackRest/Storage/S3/Driver.pm
Add base64 encode/decode.
commit : f4524aeaa92baf2cdae2125c87661036e2bdc0a0
author : David Steele <david@crunchydata.com>
date : Wed, 18 Oct 2017 11:14:26 -0400
committer: David Steele <david@crunchydata.com>
date : Wed, 18 Oct 2017 11:14:26 -0400
M doc/xml/release.xml
M libc/LibC.xs
M libc/Makefile.PL
A libc/xs/common/encode.xs
A libc/xs/common/encode.xsh
A src/common/encode.c
A src/common/encode.h
A src/common/encode/base64.c
A src/common/encode/base64.h
M src/common/errorType.c
M src/common/errorType.h
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/lib/pgBackRestTest/Module/Common/CommonEncodePerlTest.pm
A test/src/module/common/encodeTest.c
Fix instances where database-id was not rendered as an integer in JSON info output.
commit : 2d56de00fcb04f2f949c1855f68cbeee25cdee6f
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Wed, 18 Oct 2017 08:42:32 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 18 Oct 2017 08:42:32 -0400
Fixed by Cynthia Shang.
Reported by Jason O'Donnell.
M doc/xml/release.xml
M lib/pgBackRest/Info.pm
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
Add PostgreSQL versions to Debian VMs for testing.
commit : b1efd598cfe82c02db1a5943cd900dfc4be28e05
author : David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 18:21:47 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 18:21:47 -0400
M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/VmTest.pm
Remove error when overlapping timelines are detected.
commit : c94e52f697cdae67c35fe8cb0f95088029231b95
author : David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 14:34:44 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 14:34:44 -0400
Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios.
Reported by blogh.
M doc/xml/release.xml
M lib/pgBackRest/Archive/Common.pm
M lib/pgBackRest/Backup/Backup.pm
M test/lib/pgBackRestTest/Module/Archive/ArchiveCommonTest.pm
Add C memory contexts.
commit : 583a76f605cd49d03fe2fe4a756e7de08966a62d
author : David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 11:25:49 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 11:25:49 -0400
M doc/xml/release.xml
A src/common/memContext.c
A src/common/memContext.h
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/src/module/common/memContextTest.c
Convert config tests into C unit tests.
commit : 6f5186f9e620693815133dc1444c0b627a3ab6a1
author : David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 11:13:51 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 11:13:51 -0400
M doc/xml/release.xml
D libc/t/config.t
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/src/module/config/ruleTest.c
Improvements to page checksum module and tests:
commit : 9d78948a14b20813201b5bf7f21e3525577aa7dc
author : David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 11:03:06 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 11:03:06 -0400
* Page checksum module uses new C error handler.
* Convert page checksum tests into C unit tests.
M doc/xml/release.xml
D libc/t/data/page.bin
D libc/t/pageChecksum.t
M libc/xs/postgres/pageChecksum.xs
M src/postgres/pageChecksum.c
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
A test/src/module/postgres/pageChecksumTest.c
Add test macros for C results and errors.
commit : 80758f002311dd58497f95a5fd9a8f9de92920e8
author : David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 10:54:35 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 10:54:35 -0400
M doc/xml/release.xml
M test/src/common/harnessTest.h
Perl error handler recognizes errors thrown from the C library.
commit : 904b83747cd8140aa64a0eb2ac598cc8c1cf310b
author : David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 10:47:31 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 10:47:31 -0400
M doc/doc.pl
M doc/release.pl
M doc/xml/release.xml
M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Backup/Info.pm
M lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Common/Exception.pm
M lib/pgBackRest/Common/Exit.pm
M lib/pgBackRest/Protocol/Base/Minion.pm
M lib/pgBackRest/Protocol/Command/Master.pm
M lib/pgBackRest/Protocol/Local/Process.pm
M lib/pgBackRest/Stanza.pm
A libc/LibC.h
M libc/LibC.xs
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Env/ConfigEnvTest.pm
M test/test.pl
M test/travis.pl
Add C error handler.
commit : f1e739b4aa85f4b57613b8191e29c2abd303ed8e
author : David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 10:09:56 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Oct 2017 10:09:56 -0400
Basic try ... catch ... finally pattern to simplify error handling in C.
M doc/xml/release.xml
M lib/pgBackRest/Common/Exception.pm
M libc/LibC.xs
A src/common/error.c
A src/common/error.h
A src/common/errorType.c
A src/common/errorType.h
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/src/common/harnessTest.h
A test/src/module/common/errorTest.c
Add C unit test infrastructure.
commit : 10dfbd90b5c3af6ebb6c651760d2d62ced738b84
author : David Steele <david@pgmasters.net>
date : Thu, 12 Oct 2017 12:55:48 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 12 Oct 2017 12:55:48 -0400
M doc/xml/release.xml
M libc/Makefile.PL
M libc/t/sanity.t
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
A test/src/common/harnessTest.c
A test/src/common/harnessTest.h
A test/src/module/common/typeTest.c
A test/src/test.c
M test/test.pl
Add LibC.template.pm to simplify LibC module generation.
commit : 08d6f14603e7248dd7cc865129527963d9e3df49
author : David Steele <david@pgmasters.net>
date : Tue, 10 Oct 2017 18:21:44 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 10 Oct 2017 18:21:44 -0400
M doc/xml/release.xml
A libc/LibC.template.pm
M libc/Makefile.PL
Remove Debian test repo after PostgreSQL 10 release.
commit : abea4d1fd5f8cd2a8e0c8625dbb3d700b5317cc7
author : David Steele <david@crunchydata.com>
date : Thu, 5 Oct 2017 12:33:13 -0400
committer: David Steele <david@crunchydata.com>
date : Thu, 5 Oct 2017 12:33:13 -0400
M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
Warnings in C builds treated as errors.
commit : d89b7d5862201051732274f622742888dbf09315
author : David Steele <david@pgmasters.net>
date : Sat, 30 Sep 2017 10:48:25 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 30 Sep 2017 10:48:25 -0400
M doc/xml/release.xml
M test/test.pl
Improvements in C codebase:
commit : 8efcc38304dae4e224ae1ac0211ed03691ae5c8a
author : David Steele <david@pgmasters.net>
date : Sat, 30 Sep 2017 10:44:03 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 30 Sep 2017 10:44:03 -0400
* Update C naming conventions.
* Use int datatype wherever possible.
* Better separation of C source from Perl interface.
M build/lib/pgBackRestBuild/Build.pm
M build/lib/pgBackRestBuild/CodeGen/Common.pm
M build/lib/pgBackRestBuild/CodeGen/Lookup.pm
M build/lib/pgBackRestBuild/CodeGen/Switch.pm
M build/lib/pgBackRestBuild/Config/Build.pm
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
D libc/LibC.h
M libc/LibC.xs
M libc/Makefile.PL
M libc/t/config.t
R090 libc/t/pgBackRest-LibC.t libc/t/pageChecksum.t
A libc/t/sanity.t
M libc/xs/config/config.xs
M libc/xs/config/configRule.xs
M libc/xs/postgres/pageChecksum.xs
M src/common/type.h
M src/config/config.auto.md
M src/config/config.c
M src/config/config.h
M src/config/configRule.auto.md
M src/config/configRule.c
M src/config/configRule.h
M src/postgres/pageChecksum.c
M src/postgres/pageChecksum.h