pgBackRest 1.25 commit log · Life with Open Source

24 Oct 2017, 17:36 by Yogesh Sharma

`v1.25: S3 Performance Improvements`

commit   : 85b35f30a4a4ea07e92ad5ded6ce73bc2937f0ef    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 24 Oct 2017 17:36:21 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 24 Oct 2017 17:36:21 -0400

Click here for diff

Bug Fixes:  
  
* Fix custom settings for compress-level option being ignored. (Reported by Jens Wilke.)  
* Remove error when overlapping timelines are detected. Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios. (Reported by blogh.)  
* Fix instances where database-id was not rendered as an integer in JSON info output. (Fixed by Cynthia Shang. Reported by Jason O'Donnell.)  
  
Features:  
  
* Improve performance of list requests on S3. Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting. (Suggested by Mihail Shvein.)

M README.md
M doc/resource/exe.cache
M doc/xml/release.xml
M lib/pgBackRest/Config/ConfigHelpData.pm
M lib/pgBackRest/Version.pm

`Allow functions with sensitive options to be logged at debug level with redactions.`

commit   : 8674a4f7aea3bea2db7e92be599c034170b81706    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 24 Oct 2017 12:35:36 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 24 Oct 2017 12:35:36 -0400

Click here for diff

Previously, functions with sensitive options had to be logged at trace level to avoid exposing them. Trace level logging may still expose secrets so use with caution.

M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Config/Config.pm
M lib/pgBackRest/Storage/Posix/Driver.pm
M lib/pgBackRest/Storage/S3/Auth.pm
M lib/pgBackRest/Storage/S3/Request.pm
M test/expect/archive-stop-007.log
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log

`Improve performance of test code.`

commit   : b2cc65d886a5c1fe4845f655c53b0c809beefb72    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 24 Oct 2017 10:35:42 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 24 Oct 2017 10:35:42 -0400

Click here for diff

Wait when all tests have been assigned to reduce CPU load.

M doc/xml/release.xml
M test/test.pl

`Replace dynamically built class hierarchies in I/O layer with fixed parent() calls.`

commit   : d989cf8ac21e0978174cc94747965743f252a45d    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 22 Oct 2017 19:07:17 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 22 Oct 2017 19:07:17 -0400

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Backup/Filter/PageChecksum.pm
M lib/pgBackRest/Common/Http/Client.pm
M lib/pgBackRest/Common/Io/Base.pm
M lib/pgBackRest/Common/Io/Buffered.pm
A lib/pgBackRest/Common/Io/Filter.pm
M lib/pgBackRest/Common/Io/Process.pm
M lib/pgBackRest/Protocol/Storage/File.pm
M lib/pgBackRest/Storage/Filter/Gzip.pm
M lib/pgBackRest/Storage/Filter/Sha.pm
M lib/pgBackRest/Storage/Posix/FileRead.pm
M lib/pgBackRest/Storage/Posix/FileWrite.pm
M lib/pgBackRest/Storage/S3/FileRead.pm
M lib/pgBackRest/Storage/S3/FileWrite.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Module/Archive/ArchivePushTest.pm
M test/lib/pgBackRestTest/Module/Common/CommonIoHandleTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageFilterShaTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageS3Test.pm

`Fix custom settings for compress-level option being ignored.`

commit   : d82fe88031a3c9ca88bd70652bc4f9fc0e12d4bf    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 22 Oct 2017 18:05:46 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 22 Oct 2017 18:05:46 -0400

Click here for diff

Reported by Jens Wilke.

M doc/xml/release.xml
M lib/pgBackRest/Archive/Push/Async.pm
M lib/pgBackRest/Archive/Push/File.pm
M lib/pgBackRest/Archive/Push/Push.pm
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Backup/File.pm
M test/expect/archive-stop-001.log
M test/expect/archive-stop-002.log
M test/expect/archive-stop-003.log
M test/expect/archive-stop-004.log
M test/expect/archive-stop-005.log
M test/expect/archive-stop-006.log
M test/expect/archive-stop-007.log
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageFilterGzipTest.pm

`Add I/O performance tests.`

commit   : 3553ccae449b7b9ad71290234e55c57d9c9455a5    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 22 Oct 2017 13:27:47 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 22 Oct 2017 13:27:47 -0400

Click here for diff

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/lib/pgBackRestTest/Module/Performance/PerformanceIoTest.pm

`Run all tests on tempfs rather than local disk.`

commit   : 9b98c2e7d6492b626267ffa0c50c67922dd43bda    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 22 Oct 2017 13:11:03 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 22 Oct 2017 13:11:03 -0400

Click here for diff

M doc/xml/release.xml
M test/Vagrantfile
M test/test.pl

`Improve labeling for errors in helper processes.`

commit   : e1cb9ae86a596d0c1e833ba60730e65805c4ef4f    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 21 Oct 2017 14:40:14 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 21 Oct 2017 14:40:14 -0400

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Common/Io/Process.pm
M lib/pgBackRest/Protocol/Base/Master.pm
M lib/pgBackRest/Protocol/Local/Master.pm
M lib/pgBackRest/Protocol/Remote/Master.pm
M test/expect/archive-stop-001.log
M test/expect/archive-stop-002.log
M test/expect/archive-stop-003.log
M test/expect/archive-stop-004.log
M test/expect/archive-stop-005.log
M test/expect/archive-stop-006.log
M test/expect/archive-stop-007.log
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Archive/ArchivePushTest.pm
M test/lib/pgBackRestTest/Module/Protocol/ProtocolHelperTest.pm

`Improve protocol error handling.`

commit   : 44c7c0475ad30873075751ea23083861fe53278a    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 21 Oct 2017 14:11:33 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 21 Oct 2017 14:11:33 -0400

Click here for diff

In particular, "stop" errors are no longer reported as "unexpected".

M bin/pgbackrest
M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Common/Io/Process.pm
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Protocol/Base/Master.pm
M lib/pgBackRest/Protocol/Base/Minion.pm
M test/expect/mock-all-002.log

`Improve performance of list requests on S3.`

commit   : 1f120f3fce1f8f30739a0cc178e8eea2053fd2f2    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 20 Oct 2017 14:10:16 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 20 Oct 2017 14:10:16 -0400

Click here for diff

Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting.  
  
Suggested by Mihail Shvein.

M doc/xml/release.xml
M lib/pgBackRest/Common/String.pm
M lib/pgBackRest/Storage/Local.pm
M lib/pgBackRest/Storage/S3/Driver.pm

`Add base64 encode/decode.`

commit   : f4524aeaa92baf2cdae2125c87661036e2bdc0a0    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 18 Oct 2017 11:14:26 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 18 Oct 2017 11:14:26 -0400

Click here for diff

M doc/xml/release.xml
M libc/LibC.xs
M libc/Makefile.PL
A libc/xs/common/encode.xs
A libc/xs/common/encode.xsh
A src/common/encode.c
A src/common/encode.h
A src/common/encode/base64.c
A src/common/encode/base64.h
M src/common/errorType.c
M src/common/errorType.h
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/lib/pgBackRestTest/Module/Common/CommonEncodePerlTest.pm
A test/src/module/common/encodeTest.c

`Fix instances where database-id was not rendered as an integer in JSON info output.`

commit   : 2d56de00fcb04f2f949c1855f68cbeee25cdee6f    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 18 Oct 2017 08:42:32 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 18 Oct 2017 08:42:32 -0400

Click here for diff

Fixed by Cynthia Shang.  
Reported by Jason O'Donnell.

M doc/xml/release.xml
M lib/pgBackRest/Info.pm
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log

`Add PostgreSQL versions to Debian VMs for testing.`

commit   : b1efd598cfe82c02db1a5943cd900dfc4be28e05    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 18:21:47 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 18:21:47 -0400

Click here for diff

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/VmTest.pm

`Remove error when overlapping timelines are detected.`

commit   : c94e52f697cdae67c35fe8cb0f95088029231b95    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 14:34:44 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 14:34:44 -0400

Click here for diff

Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios.  
  
Reported by blogh.

M doc/xml/release.xml
M lib/pgBackRest/Archive/Common.pm
M lib/pgBackRest/Backup/Backup.pm
M test/lib/pgBackRestTest/Module/Archive/ArchiveCommonTest.pm

`Add C memory contexts.`

commit   : 583a76f605cd49d03fe2fe4a756e7de08966a62d    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 11:25:49 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 11:25:49 -0400

Click here for diff

M doc/xml/release.xml
A src/common/memContext.c
A src/common/memContext.h
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/src/module/common/memContextTest.c

`Convert config tests into C unit tests.`

commit   : 6f5186f9e620693815133dc1444c0b627a3ab6a1    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 11:13:51 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 11:13:51 -0400

Click here for diff

M doc/xml/release.xml
D libc/t/config.t
M test/lib/pgBackRestTest/Common/DefineTest.pm
A test/src/module/config/ruleTest.c

`Improvements to page checksum module and tests:`

commit   : 9d78948a14b20813201b5bf7f21e3525577aa7dc    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 11:03:06 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 11:03:06 -0400

Click here for diff

* Page checksum module uses new C error handler.  
* Convert page checksum tests into C unit tests.

M doc/xml/release.xml
D libc/t/data/page.bin
D libc/t/pageChecksum.t
M libc/xs/postgres/pageChecksum.xs
M src/postgres/pageChecksum.c
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
A test/src/module/postgres/pageChecksumTest.c

`Add test macros for C results and errors.`

commit   : 80758f002311dd58497f95a5fd9a8f9de92920e8    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 10:54:35 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 10:54:35 -0400

Click here for diff

M doc/xml/release.xml
M test/src/common/harnessTest.h

`Perl error handler recognizes errors thrown from the C library.`

commit   : 904b83747cd8140aa64a0eb2ac598cc8c1cf310b    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 10:47:31 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 10:47:31 -0400

Click here for diff

M doc/doc.pl
M doc/release.pl
M doc/xml/release.xml
M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Backup/Info.pm
M lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Common/Exception.pm
M lib/pgBackRest/Common/Exit.pm
M lib/pgBackRest/Protocol/Base/Minion.pm
M lib/pgBackRest/Protocol/Command/Master.pm
M lib/pgBackRest/Protocol/Local/Process.pm
M lib/pgBackRest/Stanza.pm
A libc/LibC.h
M libc/LibC.xs
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Env/ConfigEnvTest.pm
M test/test.pl
M test/travis.pl

`Add C error handler.`

commit   : f1e739b4aa85f4b57613b8191e29c2abd303ed8e    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 10:09:56 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 16 Oct 2017 10:09:56 -0400

Click here for diff

Basic try ... catch ... finally pattern to simplify error handling in C.

M doc/xml/release.xml
M lib/pgBackRest/Common/Exception.pm
M libc/LibC.xs
A src/common/error.c
A src/common/error.h
A src/common/errorType.c
A src/common/errorType.h
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/src/common/harnessTest.h
A test/src/module/common/errorTest.c

`Add C unit test infrastructure.`

commit   : 10dfbd90b5c3af6ebb6c651760d2d62ced738b84    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 12 Oct 2017 12:55:48 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 12 Oct 2017 12:55:48 -0400

Click here for diff

M doc/xml/release.xml
M libc/Makefile.PL
M libc/t/sanity.t
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
A test/src/common/harnessTest.c
A test/src/common/harnessTest.h
A test/src/module/common/typeTest.c
A test/src/test.c
M test/test.pl

`Add LibC.template.pm to simplify LibC module generation.`

commit   : 08d6f14603e7248dd7cc865129527963d9e3df49    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Oct 2017 18:21:44 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Oct 2017 18:21:44 -0400

Click here for diff

M doc/xml/release.xml
A libc/LibC.template.pm
M libc/Makefile.PL

`Remove Debian test repo after PostgreSQL 10 release.`

commit   : abea4d1fd5f8cd2a8e0c8625dbb3d700b5317cc7    
  
author   : David Steele <david@crunchydata.com>    
date     : Thu, 5 Oct 2017 12:33:13 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Thu, 5 Oct 2017 12:33:13 -0400

Click here for diff

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm

`Warnings in C builds treated as errors.`

commit   : d89b7d5862201051732274f622742888dbf09315    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 30 Sep 2017 10:48:25 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 30 Sep 2017 10:48:25 -0400

Click here for diff

M doc/xml/release.xml
M test/test.pl

`Improvements in C codebase:`

commit   : 8efcc38304dae4e224ae1ac0211ed03691ae5c8a    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 30 Sep 2017 10:44:03 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 30 Sep 2017 10:44:03 -0400

Click here for diff

* Update C naming conventions.  
* Use int datatype wherever possible.  
* Better separation of C source from Perl interface.

M build/lib/pgBackRestBuild/Build.pm
M build/lib/pgBackRestBuild/CodeGen/Common.pm
M build/lib/pgBackRestBuild/CodeGen/Lookup.pm
M build/lib/pgBackRestBuild/CodeGen/Switch.pm
M build/lib/pgBackRestBuild/Config/Build.pm
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
D libc/LibC.h
M libc/LibC.xs
M libc/Makefile.PL
M libc/t/config.t
R090 libc/t/pgBackRest-LibC.t libc/t/pageChecksum.t
A libc/t/sanity.t
M libc/xs/config/config.xs
M libc/xs/config/configRule.xs
M libc/xs/postgres/pageChecksum.xs
M src/common/type.h
M src/config/config.auto.md
M src/config/config.c
M src/config/config.h
M src/config/configRule.auto.md
M src/config/configRule.c
M src/config/configRule.h
M src/postgres/pageChecksum.c
M src/postgres/pageChecksum.h