pgBackRest 2.28 commit log

v2.27: Expiration Improvements and Compression Drivers

commit   : d8214e0d78ca8922eecb4da03008034c6a258b3b    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 May 2020 08:11:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 May 2020 08:11:50 -0400    

Click here for diff

Bug Fixes:  
  
* Fix issue checking if file links are contained in path links. (Reviewed by Cynthia Shang. Reported by Christophe CavalliƩ.)  
* Allow pg-path1 to be optional for synchronous archive-push. (Reviewed by Cynthia Shang. Reported by Jerome Peng.)  
* The expire command now checks if a stop file is present. (Fixed by Cynthia Shang. Reviewed by David Steele.)  
* Handle missing reason phrase in HTTP response. (Reviewed by Cynthia Shang. Reported by Tenuun.)  
* Increase buffer size for lz4 compression flush. (Reviewed by Cynthia Shang. Reported by Eric Radman.)  
* Ignore pg-host* and repo-host* options for the remote command. (Reviewed by Cynthia Shang. Reported by Pavel Suderevsky.)  
* Fix possibly missing pg1-* options for the remote command. (Reviewed by Cynthia Shang. Reported by Andrew L'Ecuyer.)  
  
Features:  
  
* Time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. (Contributed by Cynthia Shang, Pierre Ducroquet. Reviewed by David Steele.)  
* Ad hoc backup expiration. Allow the user to remove a specified backup regardless of retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)  
* Zstandard compression support. Note that setting compress-type=zst will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)  
* bzip2 compression support. Note that setting compress-type=bz2 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Contributed by Stephen Frost. Reviewed by David Steele, Cynthia Shang.)  
* Add backup/expire running status to the info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)  
  
Improvements:  
  
* Expire WAL archive only when repo-retention-archive threshold is met. WAL prior to the first full backup was previously expired after the first full backup. Now it is preserved according to retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)  
* Add local MD5 implementation so S3 works when FIPS is enabled. (Reviewed by Cynthia Shang, Stephen Frost. Suggested by Brian Almeida, John Kelley.)  
* PostgreSQL 13 beta1 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. (Reviewed by Cynthia Shang.)  
* Reduce buffer-size default to 1MiB. (Reviewed by Stephen Frost.)  
* Throw user-friendly error if expire is not run on repository host. (Contributed by Cynthia Shang. Reviewed by David Steele.)  

M CONTRIBUTING.md
M README.md
M doc/RELEASE.md
M doc/release.pl
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Ignore pg-host* and repo-host* options for the remote command.

commit   : 20d8c76b6cbf4d803b155680802e8dd245f3d107    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 May 2020 13:51:26 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 22 May 2020 13:51:26 -0400    

Click here for diff

The purpose of the remote command is to get access to local resources, so a remote should never start another remote. However, this could happen if there were host settings on the remote host, which ended badly with lock errors, loops, etc.
  

  
Add pg-local and repo-local options to indicate that the resource is local even if there are host settings.
  

  
Note that for the time being these options are internal and not intended for general usage. However, this is likely the direction needed to allow for more symmetric and manageable configurations.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/protocol/helper.c
M test/src/module/config/parseTest.c
M test/src/module/protocol/protocolTest.c

Remove extra spaces.

commit   : 35ab61da708c6b42341449ed5fb68b24bd3a854c    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 May 2020 09:28:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 May 2020 09:28:50 -0400    

Click here for diff

These spaces crept in over time and then got copy-pasted all over the place.  

M doc/xml/release.xml

Fix possibly missing pg1-* options for the remote command.

commit   : ae75ffc1731e2d3e7f230d8cceee1e6050b37a33    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 May 2020 16:09:23 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 21 May 2020 16:09:23 -0400    

Click here for diff

Some pg1-* options are required by the remote so if they are not provided in the remote's configuration file then it may cause a configuration error, depending on the operation. This currently only applies to the pg1-path option.
  

  
This is still an issue for repo-* options but the same solution cannot be applied because some repo-* options are secure and cannot be passed on the command-line.

M doc/xml/release.xml
M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

PostgreSQL 13 beta1 support.

commit   : ec7b7c5a3e2062de05e68baec987c35eef4d0a27    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 May 2020 13:46:16 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 21 May 2020 13:46:16 -0400    

Click here for diff

There don't appear to be any behavioral changes since PostgreSQL 12 and all the tests pass.
  

  
Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.

M doc/xml/release.xml
M src/Makefile.in
M src/postgres/interface.c
A src/postgres/interface/v130.c
M src/postgres/interface/version.h
M src/postgres/interface/version.vendor.h
M src/postgres/version.h
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm

Revert PostgreSQL comment to original source version.

commit   : ed814321514259a55e84607cd28358ec8cfae754    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 May 2020 13:43:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 May 2020 13:43:12 -0400    

Click here for diff

Not sure how "iff" got changed to "if". Perhaps an unintentional spell check.  

M src/postgres/interface/static.vendor.h

Add local MD5 implementation so S3 works when FIPS is enabled.

commit   : f15d6104d245330db0de59714e9872a18a65e658    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 May 2020 14:56:13 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 20 May 2020 14:56:13 -0400    

Click here for diff

S3 requires the Content-MD5 header for many requests but MD5 is not available via OpenSSL when FIPS is enabled because it is considered to be insecure.
  

  
Even though our usage does not present any security risks a local M5 implementation is required to circumvent the over-broad FIPS restriction.
  

  
Vendorize the MD5 implementation found at https://openwall.info/wiki/people/solar/software/public-domain-source-code/md5 and add full coverage for the module in the common/crypto unit tests.

M doc/xml/release.xml
M src/common/crypto/hash.c
A src/common/crypto/md5.vendor.c
M test/define.yaml
M test/src/module/common/cryptoTest.c

Add missing asserts.

commit   : d5f451a8b99609b220e24346df636e902e9f1aee    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 May 2020 08:52:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 May 2020 08:52:15 -0400    

Click here for diff

M src/command/archive/common.c

Reduce buffer-size default to 1MiB.

commit   : ea9147e2e082ef4bcb793de31b166e65719e9d08    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 May 2020 16:58:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 May 2020 16:58:49 -0400    

Click here for diff

The prior default was determined by benchmarking the Perl code prior to the 1.0 release. In general buffer allocation was more expensive in Perl so large buffers gave the best performance. This was due to multiple buffer allocations for each filter in an IO operation.  
  
The C code allocates fixed buffers for each IO operation so the cost for buffer allocation is lower than Perl. That being the case it made sense to benchmark the C code to determine the optimal buffer default.  
  
The performance/storage tests were used to measure the performance of a variety of filters. 1GiB of data was processed by each filter 10 times and the results of the tests were averaged.  
  
While most buffer sizes gave similar performance, 1MiB appeared to perform the best overall. Of course, different architectures are likely to yield different results but this seems like a sensible default. The buffer-size option may still need to be manually configured to give optimal results.  
  
Raw test data for reference:  
  
4MB buffer (prior default)  
  
copy time 1807ms, avg time 180ms, avg throughput: 5942MB/s  
md5 time 14200ms, avg time 1420ms, avg throughput: 756MB/s  
sha1 time 11431ms, avg time 1143ms, avg throughput: 939MB/s  
sha256 time 23463ms, avg time 2346ms, avg throughput: 457MB/s  
gzip -6 time 381199ms, avg time 38119ms, avg throughput: 28MB/s  
lz4 -1 time 15484ms, avg time 1548ms, avg throughput: 693MB/s  
  
1MB buffer (new default)  
  
copy time 1760ms, avg time 176ms, avg throughput: 6100MB/s  
md5 time 13739ms, avg time 1373ms, avg throughput: 781MB/s  
sha1 time 11025ms, avg time 1102ms, avg throughput: 973MB/s  
sha256 time 22539ms, avg time 2253ms, avg throughput: 476MB/s  
gzip -6 time 372995ms, avg time 37299ms, avg throughput: 28MB/s  
lz4 -1 time 15118ms, avg time 1511ms, avg throughput: 710MB/s  
  
512K buffer  
  
copy time 1782ms, avg time 178ms, avg throughput: 6025MB/s  
md5 time 13724ms, avg time 1372ms, avg throughput: 782MB/s  
sha1 time 10959ms, avg time 1095ms, avg throughput: 979MB/s  
sha256 time 22982ms, avg time 2298ms, avg throughput: 467MB/s  
gzip -6 time 378120ms, avg time 37812ms, avg throughput: 28MB/s  
lz4 -1 time 15484ms, avg time 1548ms, avg throughput: 693MB/s  
  
256K buffer  
  
copy time 1805ms, avg time 180ms, avg throughput: 5948MB/s  
md5 time 13706ms, avg time 1370ms, avg throughput: 783MB/s  
sha1 time 11074ms, avg time 1107ms, avg throughput: 969MB/s  
sha256 time 22588ms, avg time 2258ms, avg throughput: 475MB/s  
gzip -6 time 372645ms, avg time 37264ms, avg throughput: 28MB/s  
lz4 -1 time 16346ms, avg time 1634ms, avg throughput: 656MB/s  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/config/define.auto.c
M test/src/module/command/helpTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/performance/typeTest.c

Improve storage filter performance tests.

commit   : f773d909be42ba68125df991e8fbabb30f2bbe4a    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 May 2020 14:35:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 May 2020 14:35:20 -0400    

Click here for diff

Improve the accuracy of the calculations in several areas with better integer expressions.  
  
Make the input buffer size configurable. Previously it was always 1mb, i.e. block size.  
  
Use a macro for output results to reduce code duplication.  

M test/src/module/performance/storageTest.c

Handle missing reason phrase in HTTP response.

commit   : a3d9d9a387c61d02cf66eac0c6ab84517898dc34    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 May 2020 08:20:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 May 2020 08:20:33 -0400    

Click here for diff

Reason phrases (e.g. OK) are optional in HTTP 1.1 but the space after the status code is not. When the reason phrase was missing the required space was trimmed along with the trailing CR leading to a format error.  
  
Rework the logic to preserve the space and allow empty reason phrases.  
  
Found while testing against the Backblaze S3-compatible API.  

M doc/xml/release.xml
M src/common/io/http/client.c
M test/src/module/common/ioHttpTest.c

Fix typo pp64le -> ppc64le.

commit   : cffdadad927e3860bd33247ef6b7c3037fc7d9cb    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 May 2020 19:55:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 May 2020 19:55:09 -0400    

Click here for diff

Travis-CI guessed the correct value but logged a warning.  

M .travis.yml

Use an extension to denote vendorized code.

commit   : 688ec2a8f5634694a3985ec003d118ed8c74b540    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 May 2020 19:11:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 May 2020 19:11:26 -0400    

Click here for diff

Vendorized code is copied from another project when a library is not available and a git subproject won't work. Currently all the vendorized code is copied from PostgreSQL but it makes sense to have a more general mechanism for indicating vendorized code.  
  
The .vendor extension will be used to denote vendorized code in the same way that .auto is used to denote auto-generated code.  

M src/command/backup/pageChecksum.c
M src/postgres/interface/page.c
R097 src/postgres/interface/pageChecksum.auto.c src/postgres/interface/pageChecksum.vendor.c
R097 src/postgres/interface/static.auto.h src/postgres/interface/static.vendor.h
M src/postgres/interface/version.intern.h
R099 src/postgres/interface/version.auto.h src/postgres/interface/version.vendor.h
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c

Add MD5 hash filter to performance tests.

commit   : a329afd3be16206e0397732f0de79c93686702f1    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 May 2020 19:02:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 May 2020 19:02:11 -0400    

Click here for diff

M test/src/module/performance/storageTest.c

Add code count rule for valgrind suppression missed in 6be5ea33.

commit   : 92c036b966b3f88b4914c5be24c96c22d5019e98    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 May 2020 18:09:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 May 2020 18:09:41 -0400    

Click here for diff

6be5ea33 changed valgrind suppression file naming but failed to update the code count rules.  

M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Increase buffer size for lz4 compression flush.

commit   : ac5d46dc50a662c97c2948aac9101c0df71110f3    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 May 2020 18:25:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 May 2020 18:25:31 -0400    

Click here for diff

Some lz4 versions between r131 and 1.7.5 did not return a sufficient buffer size from LZ4F_compressBound() to allow LZ4F_compressEnd() to complete reliably. While this issue was fixed in lz4 1.7.5 there are affected versions in supported distributions such as CentOS/RHEL 7.  
  
Use one of the hacks suggested in https://github.com/lz4/lz4/issues/290 to increase the buffer size enough for LZ4F_compressEnd() to complete. This means that a slightly larger buffer size is required for all versions but it seems worth it to (hopefully) to fix the issue in all lz4 versions.  

M doc/xml/release.xml
M src/common/compress/lz4/compress.c

Add missing cryptoInit() in cryptoHmacOne().

commit   : f4e6e6bd80c68dca6a33a8e23e0fa32026f52d23    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 May 2020 07:49:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 May 2020 07:49:23 -0400    

Click here for diff

If cryptoInit() had not already been called then EVP_get_digestbyname() would fail.  
  
This does not appear to be a problem currently because of call order. Also, newer versions of OpenSSL auto-initialize.  

M src/common/crypto/hash.c

Add jq to tools installed by Vagrantfile.

commit   : ea485e916bb29c9c0306b7c28148f09883835482    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 May 2020 18:45:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 May 2020 18:45:23 -0400    

Click here for diff

M test/Vagrantfile

Disable package builds for CentOS 7 which are broken upstream.

commit   : ed5149c9be9e41ba4b2ccccac3644f24ecb36bd7    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 May 2020 18:27:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 May 2020 18:27:26 -0400    

Click here for diff

M .travis.yml

Move common HTTP headers to HTTP client.

commit   : e7ad795ffba6f963457b27142fe17d4aa6bd5c01    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 May 2020 19:00:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 May 2020 19:00:48 -0400    

Click here for diff

Some headers in the S3 driver were common HTTP headers that may be used by other drivers that utilize HTTP.  
  
Also change the order of HTTP_HEADER_TRANSFER_ENCODING to be alphabetical.  

M src/common/io/http/client.c
M src/common/io/http/client.h
M src/storage/s3/storage.c

Fix incorrect whitespace.

commit   : 4cbd1f1e7efe5217540e2e69bb3baf5d55556068    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 May 2020 14:27:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 May 2020 14:27:28 -0400    

Click here for diff

M src/common/type/keyValue.h
M src/storage/s3/write.c
M test/src/common/harnessStorage.c

Make storage type more generic in the integration tests.

commit   : b5dd14e6f306b18f4a1f555ccde28ffda8e41ac8    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 May 2020 18:55:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 May 2020 18:55:20 -0400    

Click here for diff

Rather than bS3 use strStorage which can indicate more than two storage types.  
  
For the moment there are still only two storage types but this change is required before more can be added.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Add missing do...while loop to harness macro.

commit   : 9639a2c15fa3e3bb5cffc6bf6ac4eb82f85aef49    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 May 2020 13:30:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 May 2020 13:30:46 -0400    

Click here for diff

M test/src/common/harnessDebug.h

Fix incorrect command in reference documentation.

commit   : b8a5c3ac6fe5b4c321270ae256492e11b8276445    
  
author   : Magnus Hagander <[email protected]>    
date     : Tue, 12 May 2020 19:13:04 +0200    
  
committer: GitHub <[email protected]>    
date     : Tue, 12 May 2020 19:13:04 +0200    

Click here for diff

Also update process to command to be more consistent with the surrounding text.

M doc/xml/reference.xml
M src/config/define.auto.c

Add ppc64le to Travis-CI build matrix.

commit   : 33cbdb78fd3f51f758b2da948faaee97d6bcd870    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 May 2020 10:10:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 11 May 2020 10:10:22 -0400    

Click here for diff

apt.postgresql.org provides packages for ppcle64 so it's important that we support it.  
  
Rearrange jobs a bit based on current runtimes and importance. Also reduce the number of tests run for arm64 since it is slower than other architectures.  

M .travis.yml

Fix subtle timing issue in command/expire tests.

commit   : 86855e271d476e9cf04d5844a7630a979b0398d6    
  
author   : David Steele <[email protected]>    
date     : Sat, 9 May 2020 12:12:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 9 May 2020 12:12:29 -0400    

Click here for diff

cdebfb09 added relative times to backup.into but a subtle issue was introduced that would cause the tests to fail if the time acquired by cmdExpire() was exactly the same as timeNow used to format backup.info. cmdExpire() was working correctly given the inputs, but the tests did not run predictably.  
  
This was found while running the tests with --no-valgrind --no-coverage which allows them to run a lot faster, thus exposing the timing issue.  

M test/src/module/command/expireTest.c

Allow more tests to run outside of containers.

commit   : 22d260ad53c4517ece42fe64ff75e022e5054466    
  
author   : David Steele <[email protected]>    
date     : Sat, 9 May 2020 09:17:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 9 May 2020 09:17:33 -0400    

Click here for diff

These tests required sudo to achieve complete coverage.  
  
Add a new coverage exception, vm_covered, that applies to code that can only be covered in a container. When the test is run outside of a container code sections that require a container will be excluded with TEST_CONTAINER_REQUIRED and the coverage exception will be added to prevent a coverage error.  
  
This does require marking up the core code with vm_covered, which in some modules (e.g. common/io/tls/client) can be extensive. It's possible that some of these tests can be rewritten to be less dependent on sudo but no attempt was made to do that here.  
  
Only allow coverage summaries in a vm since coverage summaries outside a vm will not be complete, which was true even before this commit.  

M src/command/restore/restore.c
M src/common/io/tls/client.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M test/define.yaml
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/module/command/restoreTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/posixTest.c

Minor updates for bzip2 compression after more review.

commit   : b4fc1804a83beb2628b1094cec2962f550e40bcd    
  
author   : Stephen Frost <[email protected]>    
date     : Fri, 8 May 2020 16:27:54 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 8 May 2020 16:27:54 -0400    

Click here for diff

Update error types throw by bzip2 to be more consistent with gzip.
  

  
Update the bzip2 and gzip error default to be AssertError as that's the more common case in both, and add a 'break;' to the default clause -- we don't intend to be just falling through those case statements, even if the default is the last, we should be explicit about that.
  

  
Clean up some tabs that snuck in, rename a variable to be more clear, and add some comments.

M doc/xml/release.xml
M src/common/compress/bz2/common.c
M src/common/compress/gz/common.c
M test/src/module/common/compressTest.c

Mark variables modified in TRY block as volatile.

commit   : 14369c1c3c8231e7b09b60493f2ad5047f39053c    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 May 2020 15:36:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 May 2020 15:36:20 -0400    

Click here for diff

It's important that the values in these variables are maintained even after an exception is thrown, so they must be marked volatile.  
  
Found while testing on the ppc64le architecture.  

M src/info/info.c

Add time-based retention for full backups.

commit   : cdebfb09e01e64d6acfe650345a56d2b1805374e    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 8 May 2020 15:25:03 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 8 May 2020 15:25:03 -0400    

Click here for diff

The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days.
  

  
The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/expire/expire.c
M src/common/time.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/config/loadTest.c

Update Minio version to 2020-05-06T23-23-25Z in tests/documentation.

commit   : e873ad6da0b1c2b15a360a117f71df386aa23a0d    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 May 2020 17:26:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 May 2020 17:26:46 -0400    

Click here for diff

This release fixes the issue we submitted regarding an unquoted eTag: https://github.com/minio/minio/issues/9517  

M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm

Update Fedora container to Fedora 32.

commit   : faabf1227d05dcfbefcd0dfa57fae9a537e2fc08    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 May 2020 11:06:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 May 2020 11:06:56 -0400    

Click here for diff

This allows unit testing on gcc 10.  
  
Also fix an incorrect enum in the config/config unit test that was caught by the new compiler.  

M .travis.yml
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/src/module/config/configTest.c

Use Z_STREAM_END to detect gz compression finish.

commit   : 6646446d2a1ed6182c8388c0007e184d15522127    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 May 2020 10:22:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 May 2020 10:22:22 -0400    

Click here for diff

Checking for free space in the output buffer worked, but if the buffer was completely filled then deflate() would need to be called again, which was wasteful and a bit confusing for debugging.  
  
Instead, use Z_STREAM_END to detect that compression is done.  
  
This change was inspired by the bz2 implementation in a021c9fe since bz2 does not allow BZ2_bzCompress() to be called after BZ_STREAM_END is returned. That made it obvious that gz would prefer the same implementation, even if it is more tolerant. The documentation at https://www.zlib.net/manual.html agrees.  

M src/common/compress/gz/compress.c

Don't allow sudo to disable core dumps in test containers.

commit   : f8509ab76ca9fc3469ab8a9752223c6462213e00    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 May 2020 07:38:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 May 2020 07:38:28 -0400    

Click here for diff

Newer versions of sudo output this message to stderr when run in a container:  
  
sudo: setrlimit(RLIMIT_CORE): Operation not permitted  
  
See https://github.com/sudo-project/sudo/issues/42 for details.  
  
A simple workaround is to prevent sudo from disabling core dumps. This seems safe enough because if sudo is segfaulting then core files are the least of our worries.  

M test/lib/pgBackRestTest/Common/ContainerTest.pm

Don't allow sudo to disable core dumps in documentation.

commit   : 3a7558985505ce1a764e846aa01e943dc1e470d1    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 May 2020 07:23:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 May 2020 07:23:15 -0400    

Click here for diff

Newer versions of sudo output this message to stdout when run in a container:  
  
sudo: setrlimit(RLIMIT_CORE): Operation not permitted  
  
See https://github.com/sudo-project/sudo/issues/42 for details.  
  
A simple workaround is to prevent sudo from disabling core dumps. This seems safe enough because if sudo is segfaulting then core files are the least of our worries.  

M doc/xml/user-guide.xml

Add libz-dev to required build packages in Debian documentation.

commit   : ad784e19971d43d886abdec9ee2c326954d9d7a3    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 May 2020 07:12:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 May 2020 07:12:42 -0400    

Click here for diff

This is apparently not installed by default in Ubuntu 20.04 as it was in prior versions.  

M doc/xml/user-guide.xml

Add arm64 to Travis-CI build matrix.

commit   : 12a5d8a155aeb999a4683caaa0e2ca22925be443    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 May 2020 19:11:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 May 2020 19:11:28 -0400    

Click here for diff

apt.postgresql.org will soon be providing packages for arm64 so it's important that we support it.  
  
Testing on multiple architectures also helps expose potential issues in popular architectures. See 10a5182d for an example.  

M .travis.yml

Suppress Valgrind errors on a per-VM basis.

commit   : 6be5ea3388a0238247e2a2cf47f9efd6b1a21d74    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 May 2020 18:24:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 May 2020 18:24:48 -0400    

Click here for diff

There are a number of Valgrind errors on Ubuntu 12.04 which do not happen on newer distro versions. However, suppressions for these errors have masked legitimate issues in subsequent code.  
  
Instead, make suppressions VM specific so errors in other VMs are not masked.  

M test/lib/pgBackRestTest/Common/JobTest.pm
R100 test/src/valgrind.suppress test/src/valgrind.suppress.u12

Fix leak in TlsClient object.

commit   : 28967951abb47d82914b3b8822a6bc4b79502ac0    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 May 2020 18:17:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 May 2020 18:17:50 -0400    

Click here for diff

sckClientOpen() is the most likely part of this code to error so move it up above SSL session creation to reduce the chance of a leak.  

M src/common/io/tls/client.c

Fix leak in CipherBlock object.

commit   : e67792980249b2e35c81d26d2fa574bab52250cb    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 May 2020 18:09:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 May 2020 18:09:11 -0400    

Click here for diff

EVP_CIPHER_CTX_cleanup() was being called instead of EVP_CIPHER_CTX_free() so most of the memory was being freed but not all of it.  
  
This leak was masked by Valgrind suppressions which are only applicable to Ubuntu 12.04, which will be addressed in a future commit.  

M src/common/crypto/cipherBlock.c

Simplify retry handling in tls/http/socket clients.

commit   : 10a5182d626bd4ba55d4c6ae711555fb030121ca    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 May 2020 15:00:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 May 2020 15:00:34 -0400    

Click here for diff

Travis-CI arm64 was not happy with this pattern, perhaps because connected was being reset after a longjmp() even though it should have stayed with its originally initialized value of false. In any case, tlsClientOpen() ended up returning NULL on error rather than throwing an exception.  
  
The new pattern seems simpler and passes all tests unmodified, so even though the error was only seen in TlsClient it makes sense to propagate to the other clients.  

M src/common/io/http/client.c
M src/common/io/socket/client.c
M src/common/io/tls/client.c

Always use 127.0.0.1 on TLS tests outside of containers.

commit   : 8aede3353cac6a7ddd08f89e23343c6e8b81d3fd    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 May 2020 14:49:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 May 2020 14:49:03 -0400    

Click here for diff

Resolving localhost can vary based on the local network configuration so it is safer to just use a static IP.  
  
This was found while testing on Travis-CI arm64.  

M test/src/common/harnessTls.c

Build branches with -cit suffix on Travis-CI.

commit   : 3fe6ad5047515f51b6ef9bf783e41900eca21c24    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 May 2020 10:23:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 May 2020 10:23:42 -0400    

Click here for diff

This allows a branch to be targeted at only Travis-CI and not other CI services.  

M .travis.yml

Remove redundant Cirrus-CI branch filter.

commit   : 4c6dbe17eaea15b342f03292dd899de335ff2971    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 May 2020 09:12:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 May 2020 09:12:50 -0400    

Click here for diff

M .cirrus.yml

Add bzip2 compression support.

commit   : a021c9fe053934b9b620f20caffbbf311c457bb8    
  
author   : Stephen Frost <[email protected]>    
date     : Tue, 5 May 2020 16:49:01 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 5 May 2020 16:49:01 -0400    

Click here for diff

bzip2 is a widely available, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), while being around twice as fast at compression and six times faster at decompression.
  

  
bzip2 is currently available on all supported platforms.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/contributing.xml
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/Makefile.in
M src/build/configure.ac
A src/common/compress/bz2/common.c
A src/common/compress/bz2/common.h
A src/common/compress/bz2/compress.c
A src/common/compress/bz2/compress.h
A src/common/compress/bz2/decompress.c
A src/common/compress/bz2/decompress.h
M src/common/compress/helper.c
M src/common/compress/helper.h
M src/config/define.auto.c
M src/configure
M test/Vagrantfile
M test/container.yaml
M test/define.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/patch/debian-package.patch
M test/src/module/common/compressTest.c
M test/travis.pl

Move PostgreSQL 9.4 real integration tests to Ubuntu 18.04.

commit   : 98f30ef2225e849ecaf501ff11db3952c7ff93c7    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 May 2020 15:00:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 May 2020 15:00:13 -0400    

Click here for diff

PostgreSQL 9.4 packages for RHEL 6 were dropped from yum.p.o.  

M test/lib/pgBackRestTest/Common/VmTest.pm

Replace booleans with enums in compressType parameters.

commit   : 99405cbb1582c6dcc133ce9d722beef6e4a0cf78    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 May 2020 13:23:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 May 2020 13:23:36 -0400    

Click here for diff

This was an oversight in 438b957f which added multiple compression type support. The booleans were interpreted as none and gz which works fine for the CompressType enum until the position of gz or none changes.  

M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/info/manifestTest.c

Centralize String and Buffer constants in stringz.h.

commit   : d04c21ca83163e875ad4d9e06c57402f45979272    
  
author   : David Steele <[email protected]>    
date     : Mon, 4 May 2020 19:05:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 4 May 2020 19:05:38 -0400    

Click here for diff

It's not clear how useful single-character zero-terminated constants are or if we want propagate them through the code, but it at least makes sense to centralize the constants used by the Buffer and String objects.  

M src/common/type/buffer.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringz.h

Add Zstandard compression support.

commit   : 47aa765375844509edc1b6e406d4d020590b2e1d    
  
author   : David Steele <[email protected]>    
date     : Mon, 4 May 2020 15:25:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 4 May 2020 15:25:27 -0400    

Click here for diff

Zstandard is a fast lossless compression algorithm targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.  
  
Zstandard version >= 1.0 is required, which is generally only available on newer distributions.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/contributing.xml
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/Makefile.in
M src/build.auto.h.in
M src/build/configure.ac
M src/common/compress/helper.c
M src/common/compress/helper.h
A src/common/compress/zst/common.c
A src/common/compress/zst/common.h
A src/common/compress/zst/compress.c
A src/common/compress/zst/compress.h
A src/common/compress/zst/decompress.c
A src/common/compress/zst/decompress.h
M src/config/define.auto.c
M src/configure
M test/Vagrantfile
M test/container.yaml
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
A test/patch/debian-package.patch
M test/src/module/common/compressTest.c
M test/travis.pl

Remove Ubuntu 19.04 container definition.

commit   : 1aaaa94253c3d50b5f4adbe5521cf2cee6cadbe3    
  
author   : David Steele <[email protected]>    
date     : Mon, 4 May 2020 14:02:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 4 May 2020 14:02:25 -0400    

Click here for diff

Ubuntu 19.04 is no longer supported.  

M test/container.yaml
M test/lib/pgBackRestTest/Common/VmTest.pm

Remove PostgreSQL 9.4 for Fedora 30 dropped from yum.p.o.

commit   : 39f5f3a0b484f16c2820bd47c0a12a530bd04c30    
  
author   : David Steele <[email protected]>    
date     : Mon, 4 May 2020 13:12:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 4 May 2020 13:12:52 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/VmTest.pm

Move S3 initialization in user guide quickstart.

commit   : 64a21920e2cbe8444f78ce281d06c9dfa118ed12    
  
author   : David Steele <[email protected]>    
date     : Sun, 3 May 2020 18:42:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 3 May 2020 18:42:33 -0400    

Click here for diff

The previous location was too late to allow --var=s3-all=y to work with --require=/repo-host, which depends on /quickstart/configure-archiving.  
  
Since the section is not included in production documentation, the position is not very important to flow so just move it to where it works.  

M doc/xml/user-guide.xml

Add contributor for 816ba924 and reclassify as a bug.

commit   : ef9324992250028657f7fcc9393d79fbbd0148c3    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 May 2020 17:32:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 1 May 2020 17:32:31 -0400    

Click here for diff

M doc/xml/release.xml

Allow pg-path1 to be optional for synchronous archive-push.

commit   : 816ba9244f31bc9cc333eeed1e7e12e76e36edc3    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 May 2020 10:30:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 1 May 2020 10:30:35 -0400    

Click here for diff

If the WAL path is absolute then pg1-path should be optional but in fact it was required to load pg_control.  
  
Skip the pg_control check when pg1-path is not specified. The check against the stanza version/system-id remains to protect the repo from corruption.  

M doc/xml/release.xml
M src/command/archive/push/push.c
M test/src/module/command/archivePushTest.c

Add missing spaces between while keyword and condition.

commit   : 1d45282b9723a05d0b14e7222c42b915aa217735    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 May 2020 09:31:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 1 May 2020 09:31:50 -0400    

Click here for diff

Our convention is to have a space here but some were missed.  

M src/common/error.h
M src/common/macro.h
M src/common/memContext.h
M src/common/type/string.c
M src/common/type/stringList.c
M src/config/define.c
M test/lib/pgBackRestTest/Common/ExecuteTest.pm
M test/src/common/harnessTest.h

Remove unused struct member.

commit   : 28ab65df1049be7623ce7af99a58a6dee616ea36    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 May 2020 09:08:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 1 May 2020 09:08:37 -0400    

Click here for diff

Perhaps this was intended to verify the WAL size but was never implemented.  
  
Verifying the WAL size is probably a good idea so this member may be added back if the feature is implemented.  

M src/command/archive/push/push.c
M test/src/module/command/archivePushTest.c

Convert storagePosixNew() to storagePosixNewP().

commit   : 22ba1f02ceca93a8a0fba1f1ba93243bbcb12337    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Apr 2020 11:01:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Apr 2020 11:01:38 -0400    

Click here for diff

An upcoming feature requires new parameters for storagePosixNew() and this causes a lot of churn because almost every test creates a Posix storage object. Some refactoring in the tests might reduce this duplication but storagePosixNew() is collecting a lot of parameters so converting to storagePosixNewP() makes sense in any case.  
  
There are relatively few call sites in the core code but they still benefit from better readability after this change.  

M src/storage/helper.c
M src/storage/posix/storage.c
M src/storage/posix/storage.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/lockTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

commit   : baf8cb90684fe743e51412f305f4194196f429ea    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Apr 2020 10:47:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Apr 2020 10:47:09 -0400    

Click here for diff

There is no conflict if the path containing a file link is a parent path of a path link. The Perl code apparently had this right but the migration to C missed it.  
  
Exclude this case when checking for link conflicts.  

M doc/xml/release.xml
M src/info/manifest.c
M test/src/module/info/manifestTest.c

Enforce non-null for most string options.

commit   : ce55866714ac706e542bba7e14a9306b5de848f1    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Apr 2020 10:34:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Apr 2020 10:34:44 -0400    

Click here for diff

There have been a number of segfaults reported because a string option expected to be non-null was actually null. This is generally due to options that are expected to be set but are in fact optional.  
  
Protect against this by creating cfgOptionStrNull() to get options that can be null, while changing cfgOptionStr() to always expect non-null. There are relatively few places where nulls are expected.  
  
There is definitely a chance for breakage here as null options might currently be working in the field but will be caught by this new check. Hopefully introducing the check early in the release cycle will allow us to catch any issues.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/control/start.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/repo/get.c
M src/command/repo/ls.c
M src/command/repo/put.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/upgrade.c
M src/config/config.c
M src/config/config.h
M src/db/helper.c
M src/db/protocol.c
M src/storage/helper.c
M test/src/module/config/configTest.c
M test/src/module/config/loadTest.c
M test/src/module/protocol/protocolTest.c

Move command-line decompress check in common/compress test.

commit   : 1158e7d645331ed364542f10b2c92a1740229601    
  
author   : David Steele <[email protected]>    
date     : Wed, 29 Apr 2020 13:55:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 29 Apr 2020 13:55:05 -0400    

Click here for diff

It makes sense to do this check right after the first compression so any issues are caught early.  
  
Also, none of the current compression formats omit decompressCmd so make the test mandatory.  

M test/src/module/common/compressTest.c

Expire WAL archive only when repo-retention-archive threshold is met.

commit   : c5241e5007c4736b52a519e1eefd9a38f7971dc2    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 29 Apr 2020 08:06:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 29 Apr 2020 08:06:49 -0400    

Click here for diff

Previously when retention-archive was set (either by the user or by default), archives prior to the archive-start of the oldest remaining full backup (after backup expiration occurred) would be expired even though the retention-archive threshold had not been met. For example, if there were 1 full backup remaining after backup expiration and the retention-archive was set to 2 and retention-archive-type=full, then archives prior to the archive-start of the remaining full backup would still be removed even though retention-archive required 2 full backups remaining before archives should be expired.  
  
The thought was to keep the archive directory clean and since the full backup did not require prior archives, it was safe to delete them. However, this has caused problems for some users in the past (because they needed the WAL for other purposes) and with the new adhoc and time-based retention features, it was decided that the archives should remain until the threshold was met. The archives will eventually be removed and if having them causes space issues, the expire command and the retention-archive can always be run and adjusted.  

M doc/xml/release.xml
M src/command/expire/expire.c
M test/expect/mock-expire-001.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/src/module/command/expireTest.c

Refactor regular expression error handling per Coverity report.

commit   : 775e81a74deb4da274c1e4fc86b218e957a2173d    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Apr 2020 15:55:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Apr 2020 15:55:54 -0400    

Click here for diff

Coverity was concerned that regExpError() might return and lead to an invalid reference of "this". This was unlikely since the function should never return but Coverity didn't know that. Also, a difference in error-handling logic at the two sites could cause the issue Coverity reported if they were to get out of sync.  
  
Fix by refactoring out the core error function so that it is clear it will never return.  

M src/common/regExp.c
M test/src/module/common/regExpTest.c

Fix possible NULL reference reported by Coverity.

commit   : e421cf9dd31e0916885e1510278f918afc2318fd    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Apr 2020 15:04:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Apr 2020 15:04:34 -0400    

Click here for diff

this->input is set to NULL when the read input goes to EOF but it was possible that this->input could be used again in a subsequent loop, according to Coverity.  
  
In fact this would really only be a problem if EOF suddenly went back to false, which is not allowed. However, checking this->input is cheaper than calling ioReadEofDriver() driver on each loop so this change makes sense as an optimization and it makes Coverity happy, too.  

M src/common/io/read.c

Check that options are valid when requested.

commit   : f764953b70ee1ec839079fbda262a60d91b284c9    
  
author   : David Steele <[email protected]>    
date     : Mon, 27 Apr 2020 17:25:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 27 Apr 2020 17:25:38 -0400    

Click here for diff

If an option may not be valid for a command it should be checked with cfgOptionValid() or cfgOptionTest().  
  
It appears this rule is followed pretty strictly since the only changes required were in unit tests.  

M src/config/config.c
M test/src/module/config/configTest.c
M test/src/module/config/loadTest.c

Add --set option to the expire command.

commit   : 1c1a7104604073feff3109a74d1d8ee4a2efc86d    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 27 Apr 2020 14:00:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 27 Apr 2020 14:00:36 -0400    

Click here for diff

The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/expire/expire.c
M src/config/define.auto.c
M test/define.yaml
M test/expect/real-all-001.log
M test/expect/real-all-004.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/expireTest.c

commit   : ad33f545d165260c85cb41f39fd9b0044cc97841    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 27 Apr 2020 13:17:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 27 Apr 2020 13:17:30 -0400    

Click here for diff

This function is needed for new expire features.  

M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/backup/common.h
M test/define.yaml

The expire command now checks if a stop file is present.

commit   : 483838233fb4d36425a84dc6274cc1f8833feaa2    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 27 Apr 2020 10:57:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 27 Apr 2020 10:57:19 -0400    

Click here for diff

This check has always been missing but it was not noticed because expire is not usually run independently of backup, which does the check correctly.  

M doc/xml/release.xml
M src/command/expire/expire.c
M test/src/module/command/expireTest.c

Throw user-friendly error if expire is not run on repository host.

commit   : a5b2630d7c089b229f48e2f9cbd1e9b390fcf496    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 27 Apr 2020 10:49:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 27 Apr 2020 10:49:05 -0400    

Click here for diff

Running the expire command on a non-repository host threw an assertion because the command is not yet configured for remote execution.  

M doc/xml/release.xml
M src/command/expire/expire.c
M test/src/module/command/expireTest.c

Centralize enforcement of option constraints.

commit   : 987df62ec014c9b3f21b1ff9cd6db9a33f837a84    
  
author   : David Steele <[email protected]>    
date     : Sun, 26 Apr 2020 17:22:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 26 Apr 2020 17:22:37 -0400    

Click here for diff

Each option type enforced its own constraints but there was a lot of duplication. Centralize the enforcement to remove the duplication.  
  
Also convert the option type assert to a production error. This is unlikely to happen in production but the test is quite cheap so it can't hurt.  
  
Finally, add a NULL check. Most option types can never be NULL.  

M src/config/config.c
M test/src/module/config/configTest.c

Add reviewer.

commit   : 12069ff8f388171cf094a579ad27f5969d92b60e    
  
author   : David Steele <[email protected]>    
date     : Fri, 24 Apr 2020 08:45:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 24 Apr 2020 08:45:54 -0400    

Click here for diff

M doc/xml/release.xml

Automatically add default user as reviewer.

commit   : 0dd6629a2dba7c011e03c5f39659dc516da28456    
  
author   : David Steele <[email protected]>    
date     : Fri, 24 Apr 2020 08:40:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 24 Apr 2020 08:40:54 -0400    

Click here for diff

Contributions looked like they had no reviewer when reviewed by the default user.  

M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm

Add backup/expire running status to the info command.

commit   : e92eb709d6c60e44f406af139b8346b12fee1868    
  
author   : Stefan Fercot <[email protected]>    
date     : Fri, 24 Apr 2020 08:00:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 24 Apr 2020 08:00:00 -0400    

Click here for diff

This is implemented by checking for a backup lock on the host where info is running so there are a few limitations:  
  
* It is not currently possible to know which command is running: backup, expire, or stanza-*. The stanza commands are very unlikely to be running so it's pretty safe to guess backup/expire. Command information may be added to the lock file to improve the accuracy of the reported command.  
  
* If the info command is run on a host that is not participating in the backup, e.g. a standby, then there will be no backup lock. This seems like a minor limitation since running info on the repo or primary host is preferred.  

M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c

Restore works when PGDATA is a link.

commit   : 2e6938fad9ab4e93c472a3f86812ed82fba1df4d    
  
author   : David Steele <[email protected]>    
date     : Tue, 21 Apr 2020 17:55:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Apr 2020 17:55:36 -0400    

Click here for diff

Make the restore clean process look more like manifest build, i.e. do cleanup of each target root directory outside the main cleanup callback. This means some code duplication but removes the logic handling "dot" paths.  
  
Add tests for both restore and backup (which already worked but was not tested).  

M src/command/restore/restore.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c

Fix minor leak in TLS host verification.

commit   : 1eb26789387eff604e5902848be4ea7cb633557e    
  
author   : Roman Peshkurov <[email protected]>    
date     : Tue, 21 Apr 2020 17:20:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Apr 2020 17:20:42 -0400    

Click here for diff

sk_GENERAL_NAME_free() only freed the name stack, not the names in the stack. sk_GENERAL_NAME_pop_free() frees both.  
  
Due to aggressive connection reuse this leak was unlikely to be very noticeable.  

M src/common/io/tls/client.c

Update RHEL package location (again).

commit   : 6d11fe53f773163407cf28096f3e48f75c119460    
  
author   : David Steele <[email protected]>    
date     : Tue, 21 Apr 2020 10:51:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Apr 2020 10:51:38 -0400    

Click here for diff

This changed (again) upstream so update the file paths.  

M test/test.pl

Remove XML source for test/README.md.

commit   : a999500d2c6fe07cfc935ba40dbf2d9a192a8f0c    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 16:21:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 16:21:53 -0400    

Click here for diff

63b9f46a removed test/README.md but did not remove the XML source so it reappeared when the release documentation was built.  

M doc/manifest.xml
D doc/xml/test.xml

Begin v2.27 development.

commit   : 3e2dc058173bdd01f6d48d7715c70e3c5b17facb    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 15:58:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 15:58:06 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h