pgBackRest 2.17 commit log · Life with Open Source

05 Aug 2019, 12:03 by Yogesh Sharma

`v2.16: C Migrations and Bug Fixes`

commit   : 9e730c1bd61b00fa506ea443cdd6161e88ed31ae    
  
author   : David Steele <[email protected]>    
date     : Mon, 5 Aug 2019 12:03:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 5 Aug 2019 12:03:04 -0400

Click here for diff

Bug Fixes:  
  
* Retry S3 RequestTimeTooSkewed errors instead of immediately terminating. (Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.)  
* Fix incorrect handling of transfer-encoding response to HEAD request. (Reported by Pavel Suderevsky.)  
* Fix scoping violations exposed by optimizations in gcc 9. (Reported by Christian Lange, Ned T. Crigler.)  
  
Features:  
  
* Add repo-s3-port option for setting a non-standard S3 service port.  
  
Improvements:  
  
* The local command for backup is implemented entirely in C. (Contributed by David Steele, Cynthia Shang.)  
* The check command is implemented partly in C. (Reviewed by Cynthia Shang.)

M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml

`The check command is implemented partly in C.`

commit   : 3d3003e9ca6dd449ea16c6812d1cabc392c9d294    
  
author   : David Steele <[email protected]>    
date     : Thu, 1 Aug 2019 20:35:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 1 Aug 2019 20:35:01 -0400

Click here for diff

Implement switch WAL and archive check in C but leave the rest in Perl for now.  
  
The main idea was to have some real integration tests for the new database code so the rest of the migration can wait.  
  
Reviewed by Cynthia Shang.

M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Db.pm
M src/Makefile.in
A src/command/check/check.c
A src/command/check/check.h
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
A test/src/module/command/checkTest.c

`Add Db object to encapsulate PostgreSQL queries and commands.`

commit   : e4901d50d5e99168f7962b93f7a327e45f7b7fed    
  
author   : David Steele <[email protected]>    
date     : Thu, 1 Aug 2019 15:38:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 1 Aug 2019 15:38:27 -0400

Click here for diff

Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check.  
  
Add the dbGet() helper function to get Db objects easily.  
  
Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together.  
  
Reviewed by Cynthia Shang.

M doc/xml/release.xml
M src/Makefile.in
M src/command/remote/remote.c
A src/db/db.c
A src/db/db.h
A src/db/helper.c
A src/db/helper.h
A src/db/protocol.c
A src/db/protocol.h
M src/postgres/client.c
M src/postgres/client.h
M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/version.h
M test/define.yaml
M test/src/common/harnessPq.c
M test/src/common/harnessPq.h
A test/src/module/db/dbTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/postgres/interfaceTest.c

`Retry S3 RequestTimeTooSkewed errors instead of immediately terminating.`

commit   : f9e1f3a79823ee1bd6656e4f7a8fb23e735b8ccf    
  
author   : David Steele <[email protected]>    
date     : Thu, 1 Aug 2019 14:28:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 1 Aug 2019 14:28:30 -0400

Click here for diff

The cause of this error seems to be that a failed request takes so long that a subsequent retry at the http level uses outdated headers.  
  
We're not sure if pgBackRest it to blame here (in one case a kernel downgrade fixed it, in another case an incorrect network driver was the problem) so add retries to hopefully deal with the issue if it is not too persistent.  If SSL_write() has long delays before reporting an error then this will obviously affect backup performance.  
  
Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.

M doc/xml/release.xml
M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

`Improve error handling for SSL_write().`

commit   : 2eb3c9f95f6e57072b65da81cb07a1268ebbf38e    
  
author   : David Steele <[email protected]>    
date     : Thu, 1 Aug 2019 09:58:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 1 Aug 2019 09:58:24 -0400

Click here for diff

Error codes were not being caught for SSL_write() so it was hard to see exactly what was happening in error cases.  Report errors to aid in debugging.  
  
Also add a retry for SSL_ERROR_WANT_READ.  Even though we have not been able to reproduce this case it is required by SSL_write() so go ahead and implement it.

M src/common/io/tls/client.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c

`Improve multi-host handling in protocol helper.`

commit   : 89c67287bcb8bc7884b5dd37703b0d7786c68b95    
  
author   : David Steele <[email protected]>    
date     : Wed, 31 Jul 2019 20:44:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 31 Jul 2019 20:44:49 -0400

Click here for diff

Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.

M src/protocol/client.c
M src/protocol/helper.c
M src/protocol/helper.h
M src/protocol/server.c
M src/storage/helper.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c

`Add timeout to walSegmentFind().`

commit   : 893ae24284e506c551fbc28bec8719881a856e2a    
  
author   : David Steele <[email protected]>    
date     : Wed, 31 Jul 2019 19:58:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 31 Jul 2019 19:58:57 -0400

Click here for diff

Keep trying to locate the WAL segment until timeout.  This is useful for the check and backup commands which must wait for segments to arrive in the archive.

M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/file.c
M src/command/archive/push/file.c
M test/src/module/command/archiveCommonTest.c

`Rename control/control module to control/common.`

commit   : 03b28da1cac8f39c7f57d2c15a83ad667a51b997    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 31 Jul 2019 11:35:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 31 Jul 2019 11:35:58 -0400

Click here for diff

This is more consistent with how other common modules are named.  
  
Contributed by Cynthia Shang.

M src/Makefile.in
M src/command/archive/get/file.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
R096 src/command/control/control.c src/command/control/common.c
R086 src/command/control/control.h src/command/control/common.h
M test/code-count/file-type.yaml
M test/define.yaml

`Fix incorrect comment (obviously pasted from S3 code).`

commit   : a04baa1bdc07f85041bfaf910f122b7ae278d6aa    
  
author   : David Steele <[email protected]>    
date     : Tue, 30 Jul 2019 11:42:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 30 Jul 2019 11:42:56 -0400

Click here for diff

M src/storage/remote/storage.c

`Remove extra linefeed.`

commit   : 3d892cfb75c3303b0b592d4481d415d25c3f0df9    
  
author   : David Steele <[email protected]>    
date     : Tue, 30 Jul 2019 11:39:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 30 Jul 2019 11:39:01 -0400

Click here for diff

M src/command/backup/file.h

`Don't pass local config* options to the remote.`

commit   : 88c1929ec5a41915753e3d3078bdaae62d14bf0f    
  
author   : David Steele <[email protected]>    
date     : Fri, 26 Jul 2019 08:37:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 26 Jul 2019 08:37:58 -0400

Click here for diff

The remotes have their own config options (repo-host-config, etc.) so don't pass the local config* options.  
  
This was a regression from the behavior of the Perl code and while there have been no field reports it caused breakage on test systems with multiple configurations.

M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

`Allow modules to be included for testing without requiring coverage.`

commit   : f8b0676fd6edbe915bc23d2f082b4862004ca151    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 20:15:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 20:15:06 -0400

Click here for diff

Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested.  The include directive allows this.  
  
Update modules that had previously been added to coverage that only need to be included.

M test/define.yaml
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm

`Add repo-s3-port option for setting a non-standard S3 service port.`

commit   : 554d98746a96d5a93e3449d62c81c9b7a7c84e9d    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 17:36:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 17:36:51 -0400

Click here for diff

If this option is set then ports appended to repo-s3-endpoint or repo-s3-host will be ignored.  
  
Setting this option explicitly may be the only way to use a bare ipv6 address with S3 (since multiple colons confuse the parser) but we plan to improve this in the future.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/perl/embed.auto.c
M src/storage/helper.c
M src/storage/s3/storage.h
M test/src/module/command/helpTest.c
M test/src/module/storage/s3Test.c

`Add Perl interface to C PgQuery object.`

commit   : d8ca0e5c5bc2af6b85b20f48b40b83762d415b7a    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 17:05:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 17:05:39 -0400

Click here for diff

This validates that all current queries work with the new interface and removes the dependency on DBD::Pg.

M doc/xml/user-guide.xml
M lib/pgBackRest/Db.pm
M libc/LibC.xs
M libc/Makefile.PL
M libc/typemap
A libc/xs/postgres/client.xs
A libc/xs/postgres/client.xsh
M src/Makefile.in
M src/perl/embed.auto.c
M src/perl/libc.auto.c
M test/container.yaml
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Module/Stanza/StanzaAllPerlTest.pm
M test/patch/debian-package.patch
M test/patch/rhel-package.patch

`Add PostgreSQL query client.`

commit   : 415542b4a3589ff7b25dbd97d1041a9b1ff87815    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 14:50:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 14:50:02 -0400

Click here for diff

This direct interface to libpq allows simple queries to be run against PostgreSQL and supports timeouts.  
  
Testing is performed using a shim that can use scripted responses to test all aspects of the client code.  The shim will be very useful for testing backup scenarios on complex topologies.  
  
Reviewed by Cynthia Shang.

M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/Makefile.in
M src/configure
M src/configure.ac
A src/postgres/client.c
A src/postgres/client.h
M test/Vagrantfile
M test/define.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/patch/debian-package.patch
A test/src/common/harnessPq.c
A test/src/common/harnessPq.h
A test/src/module/postgres/clientTest.c

`The local command for backup is implemented entirely in C.`

commit   : 59f135340d4f76c522c8b24ecb23d56b57b2b0f8    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 14:34:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 25 Jul 2019 14:34:16 -0400

Click here for diff

The local process is now entirely migrated to C.  Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C.  
  
Contributed by David Steele, Cynthia Shang.

M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Backup/File.pm
M lib/pgBackRest/Main.pm
D lib/pgBackRest/Protocol/Local/Minion.pm
M libc/Makefile.PL
M libc/xs/storage/storage.xsh
M src/Makefile.in
A src/command/backup/file.c
A src/command/backup/file.h
A src/command/backup/protocol.c
A src/command/backup/protocol.h
M src/command/local/local.c
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
M test/lib/pgBackRestTest/Module/Backup/BackupFileUnitPerlTest.pm
A test/src/module/command/backupTest.c

`Add int rendering to jsonFromVar().`

commit   : 54ec8f151e4164c3f363f417bece7c4b1533dfd6    
  
author   : David Steele <[email protected]>    
date     : Wed, 24 Jul 2019 19:45:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 24 Jul 2019 19:45:35 -0400

Click here for diff

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

`Add new types to JSON render.`

commit   : 615735e7eed32c429a13559584a7268e4dbff3fc    
  
author   : David Steele <[email protected]>    
date     : Wed, 24 Jul 2019 06:52:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 24 Jul 2019 06:52:49 -0400

Click here for diff

Add bool, array, and int64 as valid array subtypes.  
  
Pretty print for the array subtype is not correct but is currently not in use (this can be seen at line 328 in typeJsonTest.c).

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

`Add lock bot configuration.`

commit   : 62f0c7fb37e160d4a8a2dbe410fe8a569bc4ba38    
  
author   : David Steele <[email protected]>    
date     : Mon, 22 Jul 2019 13:09:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 22 Jul 2019 13:09:51 -0400

Click here for diff

Lock closed issues after ninety days to prevent posting on old issues.

A .github/lock.yml

`Add IoSink filter.`

commit   : 38ba458616f450232dc1090306e651561c9b3076    
  
author   : David Steele <[email protected]>    
date     : Thu, 18 Jul 2019 08:42:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 18 Jul 2019 08:42:42 -0400

Click here for diff

Discard all data passed to the filter.  Useful for calculating size/checksum on a remote system when no data needs to be returned.  
  
Update ioReadDrain() to automatically use the IoSink filter.

M libc/Makefile.PL
M src/Makefile.in
M src/common/io/filter/filter.c
A src/common/io/filter/sink.c
A src/common/io/filter/sink.h
M src/common/io/io.c
M src/storage/remote/protocol.c
M test/define.yaml
M test/src/module/storage/remoteTest.c

`Remove never-used infoBackupCheckPg() function.`

commit   : d1dd6add4853a77d2ebbf27c326bbb4670b78955    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 17 Jul 2019 16:55:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 17 Jul 2019 16:55:21 -0400

Click here for diff

Contributed by Cynthia Shang.

M src/info/infoBackup.c
M src/info/infoBackup.h
M test/src/module/info/infoBackupTest.c

`Fix incorrect handling of transfer-encoding response to HEAD request.`

commit   : 3bdba4933d45451a3fa6f5dafcb79fd976bffef6    
  
author   : David Steele <[email protected]>    
date     : Wed, 17 Jul 2019 16:49:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 17 Jul 2019 16:49:42 -0400

Click here for diff

The HTTP server can use either content-length or transfer-encoding to indicate that there is content in the response.  HEAD requests do not include content but return all the same headers as GET.  In the HEAD case we were ignoring content-length but not transfer-encoding which led to unexpected eof errors on AWS S3.  Our test server, minio, uses content-length so this was not caught in integration testing.  
  
Ignore all content for HEAD requests (no matter how it is reported) and add a unit test for transfer-encoding to prevent a regression.  
  
Found by Pavel Suderevsky.

M doc/xml/release.xml
M src/common/io/http/client.c
M test/src/module/common/ioHttpTest.c

`Remove obsolete ignoreMissing parameter.`

commit   : 6f981c53bb63d3a644a8573cb9edb162c443d03b    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 17 Jul 2019 15:44:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 17 Jul 2019 15:44:55 -0400

Click here for diff

Contributed by Cynthia Shang.

M test/src/module/storage/remoteTest.c

`Fix comment typos and clarify HEAD response behavior.`

commit   : 7662d32e6020148d822fd5080f2a5ebda7ebe103    
  
author   : David Steele <[email protected]>    
date     : Wed, 17 Jul 2019 15:42:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 17 Jul 2019 15:42:37 -0400

Click here for diff

M src/common/io/http/client.c
M test/src/module/common/ioHttpTest.c

`Allow pg storage to be remote.`

commit   : eee67db4d611bb18950daf7bddb84217b010d767    
  
author   : David Steele <[email protected]>    
date     : Wed, 17 Jul 2019 14:09:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 17 Jul 2019 14:09:50 -0400

Click here for diff

None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.

M src/Makefile.in
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/helper.c
M src/storage/remote/protocol.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

`Add compressed storage feature.`

commit   : 30f55a3c2a139ffd59547fd8e422c3fae3feecd3    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 Jul 2019 17:36:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Jul 2019 17:36:24 -0400

Click here for diff

This feature denotes storage that can compress files so that they take up less space than what was written.  Currently this includes the Posix and CIFS drivers.  The stored size of the file will be rechecked after write to determine if the reported size is different.  This check would be wasted on object stores such as S3, and they might not report the file as existing immediately after write.  
  
Also add tests to each storage driver to check features.

M src/storage/posix/storage.c
M src/storage/storage.h
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

`Allow multiple filters to be pushed to the remote and return results.`

commit   : 3e1062825dde7cab506225a3b0658552b44de7ce    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 Jul 2019 16:49:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Jul 2019 16:49:46 -0400

Click here for diff

Previously only a single filter could be pushed to the remote since order was not being maintained.  Now the filters are strictly ordered.  
  
Results are returned from the remote and set in the local IoFilterGroup so they can be retrieved.  
  
Expand remote filter support to include all filters.

M src/Makefile.in
M src/command/backup/pageChecksum.c
M src/command/backup/pageChecksum.h
M src/common/crypto/cipherBlock.c
M src/common/crypto/cipherBlock.h
M src/common/crypto/hash.c
M src/common/crypto/hash.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/write.c
M test/src/module/command/backupCommonTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/ioTest.c
M test/src/module/storage/remoteTest.c

`Add ioReadDrain().`

commit   : d5654375a5152764f76335eb5265eb994d1df6bd    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 Jul 2019 08:44:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Jul 2019 08:44:41 -0400

Click here for diff

Read all data from an IoRead object and discard it.  This is handy for calculating size, hash, etc. when the output is not needed.  
  
Update code where a loop was used before.

M src/command/archive/push/file.c
M src/command/restore/file.c
M src/common/io/io.c
M src/common/io/io.h
M test/src/module/common/ioTest.c

`Add constants for path and archive.info/backup.info combinations.`

commit   : cdb75ac8b38810e2e7c6f5d0161443650f94bb31    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 15 Jul 2019 07:13:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Jul 2019 07:13:36 -0400

Click here for diff

Contributed by Cynthia Shang.

M src/command/archive/get/file.c
M src/command/archive/push/push.c
M src/command/expire/expire.c
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h

`Allow NULL in JSON list.`

commit   : ede7df9fb1d937f228e982d987d306345007dfe0    
  
author   : David Steele <[email protected]>    
date     : Sun, 14 Jul 2019 15:53:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 14 Jul 2019 15:53:31 -0400

Click here for diff

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

`Add lstClear() to List object.`

commit   : c836c483dc7198a685991753300bae49d3caa73c    
  
author   : David Steele <[email protected]>    
date     : Sun, 14 Jul 2019 15:42:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 14 Jul 2019 15:42:55 -0400

Click here for diff

M src/common/type/list.c
M src/common/type/list.h
M test/src/module/common/typeListTest.c

`Fix incorrect offline upper bound for ignoring page checksum errors.`

commit   : e10577d0b0c8054a08513b069e2f2647ddd51fb7    
  
author   : David Steele <[email protected]>    
date     : Thu, 11 Jul 2019 09:13:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 11 Jul 2019 09:13:56 -0400

Click here for diff

For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX.  This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode.  
  
Online mode is not affected since the upper bound is retrieved from pg_start_backup().

M lib/pgBackRest/Backup/Backup.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

`Fix links broken by non-standard version.`

commit   : 2fd0ebb78aabe882d4b080d4014d7ecabbded3da    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 15:08:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 15:08:35 -0400

Click here for diff

Using version 2.15.1 fixed the duplicate tarball problem but broke the auto-generated links.  Fix them manually since this should not be a common problem.  
  
Reported by Mohamad El-Rifai.

M README.md

`Revert a2dcdc07.`

commit   : 6a89c1526e1ecc2940ed2bf42258ceb9ef85f0e0    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 10 Jul 2019 12:04:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 12:04:25 -0400

Click here for diff

It is simpler to implement the required logic in stanza-delete rather than add complexity to this function.  
  
Contributed by Cynthia Shang.

M doc/xml/release.xml
M src/command/archive/get/file.c
M src/command/archive/push/push.c
M src/command/control/control.c
M src/command/control/control.h
M test/src/module/command/controlTest.c

`Remove extraneous test macro.`

commit   : 04646599a776348914b3e9bb9b707d2845540aac    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 06:17:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 06:17:33 -0400

Click here for diff

M src/config/config.c

`Clarify that return statements are not removed in production builds.`

commit   : 4e7db608dc6488feb99e5f2707f06e3ca39d78e8    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 06:11:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 06:11:21 -0400

Click here for diff

M src/common/debug.h

`Update contributor name.`

commit   : a22a6dc08c25a2af9ac9a6f2331144f2bae8dcbe    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 06:06:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Jul 2019 06:06:07 -0400

Click here for diff

M doc/xml/release.xml

`Update lockStopTest() to optionally return a result rather than error.`

commit   : a2dcdc0711a8dfb7d562b2ee0a8586de9676669f    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 9 Jul 2019 16:41:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 9 Jul 2019 16:41:58 -0400

Click here for diff

Some commands (e.g. stanza-delete) would prefer to throw a customized error.  
  
Contributed by Cynthia Shang.

M doc/xml/release.xml
M src/command/archive/get/file.c
M src/command/archive/push/push.c
M src/command/control/control.c
M src/command/control/control.h
M test/src/module/command/controlTest.c

`Exclude more build files from rsync between tests.`

commit   : 27b3246e852e98902f21589f5cdfece9ebb27b39    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Jul 2019 08:29:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Jul 2019 08:29:25 -0400

Click here for diff

Files (especially build.auto.h) were being removed and forcing a full build between separate invocations of test.pl.  
  
This affected ad-hoc testing at the command-line, not a full test run in CI.

M test/lib/pgBackRestTest/Common/JobTest.pm

`Remove clang static analysis.`

commit   : 5e1ed2e8a52c2c4d2151ef17496f0847cd9ae939    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Jul 2019 18:34:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Jul 2019 18:34:15 -0400

Click here for diff

This analysis never produced anything but false positives (var might be NULL) but took over a minute per test run and added 600MB to the test container.

M doc/RELEASE.md
M doc/release.pl
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/test.pl
M test/travis.pl

`Force PostgreSQL versions to string for newer versions of JSON:PP.`

commit   : 488fb672948c07abfd7e38604eacbd0f82610b1a    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Jul 2019 17:25:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Jul 2019 17:25:01 -0400

Click here for diff

Since 2.91 JSON::PP has a bias for saving variables that look like numbers as numbers even if they were declared as strings.  
  
Force versions to strings where needed by appending ''.  
  
Update the json-pp-perl package on Ubuntu 18.04 to 2.97 to provide test coverage.

M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Backup/Info.pm
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm

`Remove perl critic and coverage.`

commit   : 9836578520e3fb5c038230ebfb123001a7eb32fa    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Jul 2019 16:55:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Jul 2019 16:55:17 -0400

Click here for diff

No new Perl code is being developed, so these tools are just taking up time and making migrations to newer platforms harder.  There are only a few Perl tests remaining with full coverage so the coverage tool does not warn of loss of coverage in most cases.  
  
Remove both tools and associated libraries.

M .travis.yml
M doc/lib/BackRestDoc/Common/DocManifest.pm
M lib/pgBackRest/LibC.pm
M src/perl/embed.auto.c
M test/Vagrantfile
M test/define.yaml
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
D test/lint/perlcritic.policy
D test/package/d8-libdevel-cover-perl_1.23-2_amd64.deb
D test/package/d9-libdevel-cover-perl_1.23-2_amd64.deb
D test/package/u12-libdevel-cover-perl_1.23-2_amd64.deb
D test/package/u14-libdevel-cover-perl_1.23-2_amd64.deb
D test/package/u14-libdevel-cover-perl_1.29-2_amd64.deb
D test/package/u16-libdevel-cover-perl_1.23-2_amd64.deb
D test/package/u18-libdevel-cover-perl_1.29-2_amd64.deb
M test/test.pl

`Fix scoping violations exposed by optimizations in gcc 9.`

commit   : fc2101352206a87f8471fd37455d9cd990fce95d    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Jul 2019 16:25:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Jul 2019 16:25:28 -0400

Click here for diff

gcc < 9 makes all compound literals function scope, even though the C spec requires them to be invalid outside the current scope.  Since the compiler and valgrind were not enforcing this we had a few violations which caused problems in gcc >= 9.  
  
Even though we are not quite ready to support gcc 9 officially, fix the scoping violations that currently exist in the codebase.  
  
Reported by chrlange, Ned T. Crigler.

M doc/xml/release.xml
M libc/xs/storage/storage.xsh
M src/command/archive/common.c
M src/command/help/help.c
M src/common/lock.c
M src/config/parse.c
M src/perl/config.c
M src/storage/storage.c

`Use minio for integration testing.`

commit   : 1708f1d1514b3f2afb69664317df705a1fccfaf7    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 Jul 2019 22:20:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 Jul 2019 22:20:35 -0400

Click here for diff

ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing.  Replace it with minio which starts quickly and ships as a single executable or a tiny container.  
  
Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly.  
  
This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.

M src/common/io/tls/client.c
M test/container.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm
D test/lib/pgBackRestTest/Env/S3EnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
A test/patch/debian-package.patch
A test/patch/rhel-package.patch

`Updates for openssl 1.1.1.`

commit   : b9b21315ead6610bb41ee19794763555a7265261    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 Jul 2019 22:09:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 Jul 2019 22:09:12 -0400

Click here for diff

Some HTTP error tests were failing after the upgrade to openssl 1.1.1, though the rest of the unit and integration tests worked fine.  This seemed to be related to the very small messages used in the error testing, but it pointed to an issue with the code not being fully compliant, made worse by auto-retry being enabled by default.  
  
Disable auto-retry and implement better error handling to bring the code in line with openssl recommendations.  
  
There's no evidence this is a problem in the field, but having all the tests pass seems like a good idea and the new code is certainly more robust.  
  
Coverage will be complete in the next commit when openssl 1.1.1 is introduced.

M src/common/io/tls/client.c
M test/define.yaml
M test/src/module/common/ioTlsTest.c

`Community yum package can be installed with --var=package=yum.`

commit   : c55009d0f9ebfea746ab270be167dc2d78976ce9    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Jun 2019 14:39:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Jun 2019 14:39:11 -0400

Click here for diff

Like apt, the community yum package can now be installed instead of a user-specified package.

M doc/xml/user-guide.xml

`Remove Debian package patch merged to upstream.`

commit   : b0728c33dbd7c99a7b785c152f20d5cdfa902bff    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Jun 2019 14:30:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Jun 2019 14:30:20 -0400

Click here for diff

D test/patch/debian-package.patch

`Update release notes to explicitly select release commit when tagging.`

commit   : 020101b30bfe9f2d34b878fcd51fad932781d9f8    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Jun 2019 09:38:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Jun 2019 09:38:40 -0400

Click here for diff

M doc/RELEASE.md

`Reverse loop in infoPgSave() to be consistent with infoPgNewLoad().`

commit   : 33e3d316d63a41cd64d217abe30786cfb48bf09e    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 26 Jun 2019 19:52:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Jun 2019 19:52:04 -0400

Click here for diff

Contributed by Cynthia Shang.

M src/info/infoPg.c
M test/src/module/info/infoPgTest.c

`Add test function to create the S3 bucket instead of using aws cli.`

commit   : 4bffa0c5bb7d551c60b8d8bcff9a3bbe6e02eddd    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Jun 2019 15:02:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Jun 2019 15:02:30 -0400

Click here for diff

Eventually the idea is to remove the dependency on aws cli since Python is a big install.

M libc/xs/storage/storage.xs
M libc/xs/storage/storage.xsh
M src/Makefile.in
M src/perl/libc.auto.c
M test/lib/pgBackRestTest/Env/HostEnvTest.pm

`Add Perl interface to C storage layer.`

commit   : 4815752ccc46ff742f67b369bc75ad3efcf11204    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Jun 2019 08:24:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Jun 2019 08:24:58 -0400

Click here for diff

Maintaining the storage layer/drivers in two languages is burdensome.  Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C.  Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents.  
  
The goal is to move the integration tests to C so this interface will eventually be removed.  That being the case, the interface was designed for maximum compatibility to ease the transition.  The result looks a bit hacky but we'll improve it as needed until it can be retired.

M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Archive/Get/File.pm
M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Backup/File.pm
D lib/pgBackRest/Backup/Filter/PageChecksum.pm
D lib/pgBackRest/Common/Http/Client.pm
D lib/pgBackRest/Common/Http/Common.pm
D lib/pgBackRest/Common/Xml.pm
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Protocol/Local/Minion.pm
M lib/pgBackRest/Protocol/Remote/Minion.pm
M lib/pgBackRest/Protocol/Storage/Helper.pm
M lib/pgBackRest/Protocol/Storage/Remote.pm
M lib/pgBackRest/Restore.pm
M lib/pgBackRest/RestoreFile.pm
M lib/pgBackRest/Stanza.pm
M lib/pgBackRest/Storage/Base.pm
D lib/pgBackRest/Storage/Cifs/Driver.pm
D lib/pgBackRest/Storage/Filter/CipherBlock.pm
D lib/pgBackRest/Storage/Filter/Gzip.pm
D lib/pgBackRest/Storage/Filter/Sha.pm
M lib/pgBackRest/Storage/Helper.pm
D lib/pgBackRest/Storage/Posix/Driver.pm
D lib/pgBackRest/Storage/Posix/FileRead.pm
D lib/pgBackRest/Storage/Posix/FileWrite.pm
D lib/pgBackRest/Storage/S3/Auth.pm
D lib/pgBackRest/Storage/S3/Driver.pm
D lib/pgBackRest/Storage/S3/FileRead.pm
D lib/pgBackRest/Storage/S3/FileWrite.pm
D lib/pgBackRest/Storage/S3/Info.pm
D lib/pgBackRest/Storage/S3/Request.pm
R055 lib/pgBackRest/Storage/Local.pm lib/pgBackRest/Storage/Storage.pm
A lib/pgBackRest/Storage/StorageRead.pm
A lib/pgBackRest/Storage/StorageWrite.pm
M libc/LibC.h
M libc/LibC.xs
M libc/Makefile.PL
M libc/build/lib/pgBackRestLibC/Build.pm
M libc/typemap
D libc/xs/crypto/cipherBlock.xs
D libc/xs/crypto/cipherBlock.xsh
M libc/xs/crypto/hash.xs
M libc/xs/crypto/hash.xsh
M libc/xs/postgres/pageChecksum.xs
M libc/xs/storage/storage.xs
A libc/xs/storage/storage.xsh
A libc/xs/storage/storageRead.xs
A libc/xs/storage/storageRead.xsh
A libc/xs/storage/storageWrite.xs
A libc/xs/storage/storageWrite.xsh
M src/Makefile.in
M src/command/restore/file.c
M src/common/error.c
M src/perl/embed.auto.c
M src/perl/libc.auto.c
M src/storage/storage.c
M src/storage/storage.intern.h
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Module/Backup/BackupFileUnitPerlTest.pm
D test/lib/pgBackRestTest/Module/Common/CommonHttpClientPerlTest.pm
M test/lib/pgBackRestTest/Module/Common/CommonIniPerlTest.pm
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
D test/lib/pgBackRestTest/Module/Performance/PerformanceIoTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/lib/pgBackRestTest/Module/Stanza/StanzaAllPerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageFilterCipherBlockPerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageFilterGzipPerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageFilterShaPerlTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageHelperPerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageLocalPerlTest.pm
A test/lib/pgBackRestTest/Module/Storage/StoragePerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StoragePosixPerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageS3AuthPerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageS3CertPerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageS3PerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageS3RequestPerlTest.pm
M test/src/module/storage/posixTest.c

`Fix missing dash in site name.`

commit   : bd6c0941e9e3aa2dab84f7404da1f9dc60cd693b    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 Jun 2019 17:27:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 Jun 2019 17:27:19 -0400

Click here for diff

Without this the project name and tagline just run together.

M doc/lib/BackRestDoc/Html/DocHtmlPage.pm

`Begin v2.16 development.`

commit   : 466602387bc279aa44a811dc2948b893e4127b2c    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 Jun 2019 08:42:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 Jun 2019 08:42:20 -0400

Click here for diff

M doc/RELEASE.md
M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h