pgBackRest 2.37 commit log

v2.36: Minor Bug Fixes and Improvements

commit   : 42fd6ce4e09ee92614cfbfb6766d9c3a6ba9cc1a    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 08:59:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 08:59:14 -0400    

Click here for diff

Bug Fixes:  
  
* Allow "global" as a stanza prefix. (Reviewed by Stefan Fercot. Reported by Younes Alhroub.)  
* Fix segfault on invalid GCS key file. (Reviewed by Stephen Frost. Reported by Henrik Feldt.)  
  
Improvements:  
  
* Allow link-map option to create new links. (Reviewed by Don Seiler, Stefan Fercot, Chris Bandy. Suggested by Don Seiler.)  
* Increase max index allowed for pg/repo options to 256. (Reviewed by Cynthia Shang.)  
* Add WebIdentity authentication for AWS S3. (Reviewed by James Callahan, Reid Thompson, Benjamin Blattberg, Andrew L'Ecuyer.)  
* Report backup file validation errors in backup.info. (Contributed by Stefan Fercot. Reviewed by David Steele.)  
* Add recovery start time to online backup restore log. (Reviewed by Tom Swartz, Stefan Fercot. Suggested by Tom Swartz.)  
* Report original error and retries on local job failure. (Reviewed by Stefan Fercot.)  
* Rename page checksum error to error list in info text output. (Reviewed by Stefan Fercot.)  
* Add hints to standby replay timeout message. (Reviewed by Cynthia Shang, Stefan Fercot. Suggested by Leigh Downs.)  

M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Make analytics optional for HTML documentation.

commit   : 6abb06248c2829f2c27a7a553d373b0fdf70cfc3    
  
author   : David Steele <[email protected]>    
date     : Fri, 29 Oct 2021 11:45:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 29 Oct 2021 11:45:50 -0400    

Click here for diff

Analytics should only be added to the current HTML documentation on the website, so exclude them by default.  

M doc/lib/pgBackRestDoc/Html/DocHtmlBuilder.pm
M doc/lib/pgBackRestDoc/Html/DocHtmlPage.pm
M doc/manifest.xml
M doc/release.pl

Restore some linefeed rendering behavior from before def7d513.

commit   : 13366573261bf2562bc93ef77014f7d16b815e5b    
  
author   : David Steele <[email protected]>    
date     : Fri, 29 Oct 2021 10:35:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 29 Oct 2021 10:35:56 -0400    

Click here for diff

The new rendering behavior is correct in normal cases, but for the pre-rendered HTML blocks in the command and configuration references it causes a lot of churn. This would be OK if the new HTML was diff-able, but it is not.  
  
Go back to the old behavior of using br tags for this case to reduce churn until a more permanent solution is found.  

M doc/lib/pgBackRestDoc/Common/DocRender.pm
M doc/lib/pgBackRestDoc/Html/DocHtmlPage.pm
M doc/xml/release.xml

Use Rocky Linux for documentation builds instead of CentOS.

commit   : c32e000ab92e9e9e5495ddec7c4e347c35801570    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Oct 2021 15:15:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Oct 2021 15:15:49 -0400    

Click here for diff

Since CentOS 8 will be EOL at the end of the year it makes sense to do this now. The centos:8 image is still used in documentation.xml because changes there require manual testing, which will need to be done at a later date. The changes are not user-facing, however, and can be done at any time.  
  
Also update CentOS references to RHEL since that is what we are emulating for testing purposes.  

M .github/workflows/test.yml
M CONTRIBUTING.md
M doc/README.md
M doc/RELEASE.md
M doc/release.pl
M doc/xml/contributing.xml
M doc/xml/documentation.xml
M doc/xml/user-guide.xml
M test/ci.pl
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/test.pl

Fix typo in contributing guide.

commit   : 30c589ace7a459f3b3d09b702e314efd412e71d6    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Oct 2021 13:28:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Oct 2021 13:28:49 -0400    

Click here for diff

Not sure how this got broken but it was probably an errant search and replace.  

M CONTRIBUTING.md

Add missing assert.

commit   : 2f1a2877373c7be68d553c7f781299edaf8ff196    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Oct 2021 11:49:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Oct 2021 11:49:00 -0400    

Click here for diff

M src/config/parse.c

Minor fix for lower-casing of option summaries.

commit   : adc09ffc3bccb24c83a471c8af1f9bf68f2cf9c8    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Oct 2021 08:10:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Oct 2021 08:10:43 -0400    

Click here for diff

This works with existing cases and fixes "I/O".  

M src/command/help/help.c
M test/src/module/command/helpTest.c

Improve documentation for cmd-ssh, repo-host-cmd, pg-host-cmd options.

commit   : fa564ee1969229b5cf60d2479d8ace85325f4db3    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Oct 2021 11:08:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Oct 2021 11:08:32 -0400    

Click here for diff

Use "command" instead of "exe" and make the descriptions more consistent.  

M src/build/help/help.xml
M test/src/module/command/helpTest.c

Improve documentation for buffer-size option.

commit   : e1f6c066b3da11fd21b1155c90370c3fa2da06b7    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Oct 2021 10:52:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Oct 2021 10:52:39 -0400    

Click here for diff

M src/build/help/help.xml
M test/src/module/command/helpTest.c

Fix test descriptions in common/typeVariantTest.

commit   : 1f7c7b7dda1c736fab2673084498fc7c220b742a    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 16:56:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 16:56:44 -0400    

Click here for diff

M test/src/module/common/typeVariantTest.c

Add coverage for empty CATCH() blocks.

commit   : d74fe7a222c1e1ae0f02addbeb712f8946d3d731    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 13:53:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 13:53:44 -0400    

Click here for diff

Currently empty CATCH() blocks are always marked as covered because of the loop structure of error handling.  
  
A prototype implementation of error handling without looping has shown that these CATCH() blocks are not covered without new tests. Whether or not that prototype gets committed it is worth adding the tests.  

M test/src/module/command/backupTest.c
M test/src/module/command/repoTest.c

Add assertion for Coverity.

commit   : e2eea974c144f77448aa9d5fbb55c933b70ea5ad    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 12:09:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 12:09:41 -0400    

Click here for diff

Coverity thinks this value might be NULL but that should not be possible because of the TRY...CATCH block.  

M src/common/io/tls/client.c

Add missing paragraph tags in coding standards.

commit   : 4f10441574761c9cd4e31cdef750742e004ae669    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 08:25:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 08:25:21 -0400    

Click here for diff

M doc/xml/coding.xml
M doc/xml/release.xml

Use externed instead of extern'd in comments.

commit   : 7fb99c59c88fe11c679d6ba7835f995a969462c0    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 07:46:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Oct 2021 07:46:48 -0400    

Click here for diff

This is mostly to revert some comment changes in b11ab9f7 that will break the ppc64le patch, but at the same time keep the spelling consistent in all comments and documentation.  
  
Also revert some space changes for the same reason.  

M CODING.md
M doc/xml/coding.xml
M src/common/type/buffer.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/config/parse.c
M src/info/manifest.c
M test/lib/pgBackRestTest/Common/JobTest.pm

Adjustments for new breaking change in Azurite.

commit   : 653ffcf8d98ebfe94ae44ed54b4a295428c57850    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Oct 2021 15:42:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Oct 2021 15:42:28 -0400    

Click here for diff

Azurite released another breaking change (see fbd018cd, 096829b3, c38d6926, and Azurite issue 1039) so make adjustments as needed to documentation and tests.  
  
Also remove some dead code that hid the repo-storage-host option and was made obsolete by all these changes.  

M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/xml/user-guide.xml
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/Host/HostAzureTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm

Check return value of getsockopt().

commit   : 13d4559708819787ad05be6f37ec0badb0eccae5    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Oct 2021 15:31:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Oct 2021 15:31:39 -0400    

Click here for diff

Checking the return value is not terribly important here, but if setsockopt() fails it is likely that bind() will fail as well. May as well get it over with and this makes Coverity happy.  

M doc/xml/release.xml
M src/common/io/socket/server.c

Fix mismatched parameters in tlsClientNew() call.

commit   : 1152f7a7d64e69eed1d9e74b48a308f6c742c28a    
  
author   : Reid Thompson <[email protected]>    
date     : Mon, 25 Oct 2021 12:56:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Oct 2021 12:56:33 -0400    

Click here for diff

3879bc69 added this call and the parameters were not quite right but in way that the compiler decided they were OK. It was mostly working but TLS verification was disabled if caPath was NULL, which is not OK.  

M doc/xml/release.xml
M src/storage/s3/storage.c

Fix typos in error messages.

commit   : a1a2284c881ba6c3b9b1c316b31e0583c006f1af    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Oct 2021 09:01:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Oct 2021 09:01:22 -0400    

Click here for diff

M src/common/io/socket/common.c

Add WebIdentity authentication for AWS S3.

commit   : 3879bc69b888daa04d2ca98a2d1219cf22519ddc    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Oct 2021 18:31:55 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 22 Oct 2021 18:31:55 -0400    

Click here for diff

This allows credentials to be automatically acquired in an EKS environment.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/config/config.auto.h
M src/config/parse.auto.c
M src/storage/s3/helper.c
M src/storage/s3/storage.c
M src/storage/s3/storage.h
M test/src/module/storage/s3Test.c

Store config values as a union instead of a variant.

commit   : 51785739f44b624091246c48af6defe97c30d7a7    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Oct 2021 18:02:20 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 22 Oct 2021 18:02:20 -0400    

Click here for diff

The variants were needed to easily serialize configurations for the Perl code.
  

  
Unions are more efficient and will allow us to add new types that are not supported by variants, e.g. StringId.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/expire/expire.c
M src/command/help/help.c
M src/command/restore/restore.c
M src/config/config.c
M src/config/config.h
M src/config/config.intern.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M src/config/parse.h
M src/config/protocol.c
M src/protocol/helper.c
M test/src/module/config/parseTest.c

Fix segfault on invalid GCS key file.

commit   : 2cea005f740d640290a9948595f5933833e30e7d    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Oct 2021 17:19:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Oct 2021 17:19:16 -0400    

Click here for diff

M doc/xml/release.xml
M src/storage/gcs/storage.c

Add analytics to the HTML documentation.

commit   : cb36fec102855bf268ec5234bbb5261be98bdc61    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 17:48:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 17:48:00 -0400    

Click here for diff

M doc/lib/pgBackRestDoc/Html/DocHtmlBuilder.pm

Fix indentation.

commit   : a63e732987bc1f6f26514568dabc6c0b23df07ab    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 17:25:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 17:25:32 -0400    

Click here for diff

M doc/xml/index.xml

commit   : 78e1bd333068c4a857054490698115fa2c698e0b    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 17:10:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 17:10:00 -0400    

Click here for diff

There should be few if any users running v1 now so these links do not need to be so prominent.  

M README.md
M doc/xml/index.xml

commit   : 861df2a73cafbd49049dccdc55d5214b00dd3cec    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 17:02:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 17:02:46 -0400    

Click here for diff

M README.md
M doc/xml/index.xml

Fix incorrect host name in user guide.

commit   : 1cb8ae15de5b4276682bdd9825ca97012cd43855    
  
author   : Reid Thompson <[email protected]>    
date     : Thu, 21 Oct 2021 13:51:59 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 21 Oct 2021 13:51:59 -0400    

Click here for diff

The text indicates to populate the pg-primary IP address into the pg_hba.conf file to allow replication connections. It should indicate to populate the pg-standby IP address

M doc/xml/contributing.xml
M doc/xml/user-guide.xml

Fix typos.

commit   : b11ab9f799aa6fc32dd03e96e8a0428d5c83d9ae    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 13:31:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 13:31:22 -0400    

Click here for diff

M doc/RELEASE.md
M src/build/config/config.yaml
M src/build/config/parse.c
M src/command/archive/common.h
M src/command/archive/push/file.c
M src/command/backup/backup.c
M src/command/expire/expire.c
M src/command/restore/restore.c
M src/common/io/client.intern.h
M src/common/io/http/client.c
M src/common/io/server.intern.h
M src/common/io/socket/common.c
M src/common/io/socket/common.h
M src/common/type/buffer.h
M src/common/type/pack.c
M src/common/type/pack.h
M src/common/type/stringId.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/config/parse.c
M src/db/db.h
M src/info/manifest.c
M src/postgres/version.h
M src/protocol/server.c
M src/storage/helper.h
M src/storage/posix/storage.c
M src/storage/s3/storage.c
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/common/harnessServer.h

Fix outdated comment.

commit   : 8ad6b7330e1ee6bcbc0f06ec0562a433e7888f44    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 09:20:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Oct 2021 09:20:40 -0400    

Click here for diff

This check was moved from within the path checks at some point but the comment did not get updated.  

M src/config/parse.c

Allow S3/Azure Docker images to be specified in user guide.

commit   : fbd018cd56482efff425beb4026fe22482115138    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Oct 2021 08:22:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Oct 2021 08:22:37 -0400    

Click here for diff

It is not uncommon for the S3/Azure emulators we use to introduce breaking changes without warning. If that happens the documentation can still be built by specifying a working version of the image. In general, it is better to let the version float so we know when things break.  
  
Azurite has yet another breaking change coming up (see 096829b3, c38d6926, and Azurite issue 1039) so set azure-image at the current version until the breaking change has been released.  

M doc/xml/user-guide.xml

Add -Werror -Wfatal-errors -g flags to configure --enable-test.

commit   : 5dfdd6dd5b7b43dc3a223b9552aed4052d0db3aa    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Oct 2021 12:45:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Oct 2021 12:45:20 -0400    

Click here for diff

These flags are used for all tests but it was not possible to add them to configure before the change in 046d6643. This is especially important for adhoc tests to ensure the flags are not forgotten.  
  
Remove the flags from test make commands where they were being applied.  
  
There is no change for production builds.  

M src/build/configure.ac
M src/configure
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/test.pl

Set most compiler flags as late as possible in configure.

commit   : 046d6643373859c5e848a97e06389ed2aa553723    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Oct 2021 12:14:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Oct 2021 12:14:09 -0400    

Click here for diff

Some flags, e.g. -Wfatal-errors, will cause tests in configure to behave incorrectly so we have not been able to add them to --enable-test.  
  
Add the compiler flags as late as possible so configure checks are not affected. This will allow us to add flags that we need for testing without having to explicitly pass them to make.  

M src/build/configure.ac
M src/configure

Add br tags for HTML documentation rendering missed in def7d513.

commit   : e443e3c6c05c9d65a67dac0c8430b59239fbc1b8    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Oct 2021 09:06:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Oct 2021 09:06:06 -0400    

Click here for diff

M doc/lib/pgBackRestDoc/Html/DocHtmlPage.pm
M doc/xml/release.xml

Fix typos.

commit   : 4c2d89eb66e11017e2e73ad4171e4493c28acdad    
  
author   : Reid Thompson <[email protected]>    
date     : Mon, 18 Oct 2021 16:43:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 16:43:19 -0400    

Click here for diff

M src/build/config/config.yaml

Add missing paragraph tag in user guide.

commit   : 6cc8e45df68c299990c0ad1f40c53b9282cb46db    
  
author   : Reid Thompson <[email protected]>    
date     : Mon, 18 Oct 2021 14:42:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 14:42:49 -0400    

Click here for diff

M doc/xml/user-guide.xml

Add TLS Server.

commit   : ccc255d3e05d8ce2b6ac251d1498f71b04098a86    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 14:32:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 14:32:41 -0400    

Click here for diff

The TLS server is an alternative to using SSH for protocol connections to remote hosts.  
  
This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.  

M doc/xml/release.xml
M src/Makefile.in
M src/build/config/config.yaml
M src/build/error/error.yaml
M src/build/help/help.xml
A src/command/server/ping.c
A src/command/server/ping.h
A src/command/server/server.c
A src/command/server/server.h
M src/common/error.auto.c
M src/common/error.auto.h
A src/common/io/server.c
A src/common/io/server.h
A src/common/io/server.intern.h
M src/common/io/session.c
M src/common/io/session.h
M src/common/io/session.intern.h
M src/common/io/socket/client.c
M src/common/io/socket/client.h
M src/common/io/socket/common.c
M src/common/io/socket/common.h
A src/common/io/socket/server.c
A src/common/io/socket/server.h
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/client.h
A src/common/io/tls/common.c
A src/common/io/tls/common.h
A src/common/io/tls/server.c
A src/common/io/tls/server.h
M src/common/io/tls/session.c
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/main.c
M src/protocol/client.h
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/azure/storage.c
M src/storage/gcs/storage.c
M src/storage/s3/storage.c
M test/certificate/.gitignore
M test/certificate/README.md
D test/certificate/pgbackrest-test-alt-name.crt
A test/certificate/pgbackrest-test-client.cnf
A test/certificate/pgbackrest-test-client.crt
A test/certificate/pgbackrest-test-client.key
R057 test/certificate/pgbackrest-test-alt-name.cnf test/certificate/pgbackrest-test-server.cnf
A test/certificate/pgbackrest-test-server.crt
R100 test/certificate/pgbackrest-test.key test/certificate/pgbackrest-test-server.key
D test/certificate/pgbackrest-test.cnf
D test/certificate/pgbackrest-test.crt
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Common/HostTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessProtocol.c
M test/src/common/harnessServer.c
M test/src/common/harnessServer.h
M test/src/module/command/remoteTest.c
A test/src/module/command/serverTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/config/loadTest.c
M test/src/module/db/dbTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/s3Test.c
M test/src/test.c

Write command configuration overrides explicitly.

commit   : 09fb9393f14b47effebaecc449a97ad07ef4c752    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 14:02:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 14:02:05 -0400    

Click here for diff

If not written explicitly then it is impossible to distinguish the override from a NULL, which indicates no override.  

M src/build/help/render.c

Add missing static keywords in test modules.

commit   : 90f7f11a9f71152185219bbb57bf1de001e3a91b    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 12:22:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 12:22:48 -0400    

Click here for diff

M test/src/module/build/commonTest.c
M test/src/module/build/configTest.c
M test/src/module/build/errorTest.c
M test/src/module/build/helpTest.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/assertOffTest.c
M test/src/module/common/assertOnTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/debugOffTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/encodeTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/execTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/forkTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/logTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/stackTraceTest.c
M test/src/module/common/statTest.c
M test/src/module/common/timeTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeMcvTest.c
M test/src/module/common/typeObjectTest.c
M test/src/module/common/typePackTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/common/typecTest.c
M test/src/module/common/userTest.c
M test/src/module/common/waitTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Allow error buffer to be resized for testing.

commit   : 4570c7e27528400373ece8dc7bd348baf3ff064e    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 11:32:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 11:32:53 -0400    

Click here for diff

Some tests can generate very large error messages for diffs and they often get cut off before the end.  
  
Also fix a test so it does not create too large a buffer on the stack.  

M src/common/error.c
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/module/common/errorTest.c

Increase some storage test timeouts.

commit   : 838ee3bd08c739e3dcf611e9bddfaa6c8acbb2aa    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 11:05:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Oct 2021 11:05:53 -0400    

Click here for diff

32-bit Debian 9 is sometimes timing out on these tests so increase the timeouts to make the tests more reliable.  

M test/src/module/storage/gcsTest.c
M test/src/module/storage/s3Test.c

Convert configuration optional rules to pack format.

commit   : 6b9e19d423d99d3063c4bff3d3533b1e5081e4cb    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Oct 2021 12:35:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Oct 2021 12:35:47 -0400    

Click here for diff

The previous format was custom for configuration parsing and was not as expressive as the pack format. An immediate benefit is that commands with the same optional rules are merged.  
  
Defaults are now represented correctly (not multiplied), which simplifies the option default functions used by help.  

M src/build/config/parse.c
M src/build/config/parse.h
M src/build/config/render.c
M src/command/help/help.c
M src/config/config.c
M src/config/config.h
M src/config/config.intern.h
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M test/src/module/build/configTest.c
M test/src/module/command/helpTest.c
M test/src/module/config/parseTest.c

Update 32-bit test container to Debian 9.

commit   : 360cff94e4e9e1ab5a690a1f5c38eb278158a892    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Oct 2021 12:33:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Oct 2021 12:33:31 -0400    

Click here for diff

Also rebalance PostgreSQL version integration tests.  

M .github/workflows/test.yml
M src/common/io/tls/client.c
M test/container.yaml
M test/lib/pgBackRestTest/Common/VmTest.pm
D test/src/valgrind.suppress.u16

Remove allow range from pg-socket-path option.

commit   : 0e84c19a9fde0480b30078f5d3b419267b2f7673    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Oct 2021 17:50:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Oct 2021 17:50:54 -0400    

Click here for diff

The allow range was never processed because the string type does not allow ranges, but it is wasteful to have it in the parse rules.  
  
It would be good if auto-generation errored on useless allow ranges, but that will need wait since it does not impact production.  

M src/build/config/config.yaml
M src/config/parse.auto.c

Add const buffer functions to Pack type.

commit   : 144469b9772bad14466fcafc65edef58c5366755    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Oct 2021 15:50:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Oct 2021 15:50:55 -0400    

Click here for diff

These allow packs to be created without allocating a buffer in the case that the buffer already exists or the data is in a global constant.  
  
Also fix a rendering issue in hrnPackReadToStr().  

M src/common/type/pack.c
M src/common/type/pack.h
M test/src/common/harnessPack.c
M test/src/module/common/typePackTest.c

Rename SSH connection control parameters in integration tests.

commit   : 66bfd1327e56f0f2de99fc6009431f3ee06ad6b8    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 19:48:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 19:48:41 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm

Update RHEL package URL.

commit   : 447b24309d02938d04e036ec7814e75982210eb4    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 19:43:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 19:43:40 -0400    

Click here for diff

M test/test.pl

Rename PostgreSQL pid file constants and tests.

commit   : 01b20724daf4c5cb25d6f636fb90456759773d22    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 19:36:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 19:36:59 -0400    

Click here for diff

M doc/lib/pgBackRestDoc/Common/DocRender.pm
M src/build/help/help.xml
M src/command/backup/backup.c
M src/command/restore/restore.c
M src/command/stanza/delete.c
M src/info/manifest.c
M src/postgres/interface.c
M src/postgres/interface.h
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/info/manifestTest.c

Rename manifest file primary flag in tests.

commit   : 570162040864b8c56b236aa66d8f5c8d610b754b    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 19:02:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 19:02:58 -0400    

Click here for diff

M src/info/manifest.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/src/module/command/backupTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/verifyTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/manifestTest.c

Update Vagrantfile to Ubuntu 20.04.

commit   : a44f9e373b47354a09bec0eaf2f3bf9e261c6941    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 13:21:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 13:21:04 -0400    

Click here for diff

M test/Vagrantfile

Do not show output of PostgreSQL upgrade status check in user guide.

commit   : b16e827d69364408ea687a8e4b8894f7e889792e    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 13:20:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 13:20:11 -0400    

Click here for diff

On some platforms the output may contain UTF-8 characters that the latex code is not prepared to handle.  
  
Showing the command is much more important than showing the output, so no big loss.  

M doc/xml/user-guide.xml

Update comments referring to the PostgreSQL primary.

commit   : 5e84645ac030544b572036c08d055885d96d8905    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 12:16:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 12:16:47 -0400    

Click here for diff

M src/command/check/check.c
M src/command/stanza/common.c
M test/lib/pgBackRestTest/Env/Manifest.pm

commit   : 430efff98a5b8dcf7c048f383abc12d9c0e5bbf0    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 12:01:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 12:01:53 -0400    

Click here for diff

M .github/pull_request_template.md
M CONTRIBUTING.md
M README.md
M doc/RELEASE.md
M doc/xml/contributing.xml
M doc/xml/index.xml

Update contributing.xml with rendering changes from def7d513.

commit   : 1212668d5eff51756b0719b5296f7640e8096605    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 11:43:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 11:43:14 -0400    

Click here for diff

Also update help.xml path missed in f4e1babf.  

M CONTRIBUTING.md
M doc/xml/contributing.xml

Add libc6-dbg required by updated valgrind to Vagrantfile/Dockerfile.

commit   : 90c73183ea5de6a63a23a9047ae2debb3f59b940    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 09:37:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 09:37:03 -0400    

Click here for diff

M test/Dockerfile
M test/Vagrantfile

Add debug options to code generation make in test.pl.

commit   : c2d4552b7328489d703dc03defa769b1ccb8f739    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 08:51:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Oct 2021 08:51:58 -0400    

Click here for diff

M test/test.pl

Remove command overrides for output options.

commit   : bd91ebca759d2d6cfc2f7aa660366f5f7f09994a    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Oct 2021 16:16:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Oct 2021 16:16:05 -0400    

Click here for diff

The overrides are not needed since both commands require the same default and allow list.  

M src/build/config/config.yaml
M src/config/parse.auto.c

Remove command overrides for restore-only options.

commit   : e8e346bc8738815b07bd80a37ace862fdab3dc1d    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Oct 2021 08:53:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Oct 2021 08:53:12 -0400    

Click here for diff

The overrides are not needed since these options are only valid for one command.  

M src/build/config/config.yaml
M src/config/parse.auto.c

Invert required in set option to simplify generated rules.

commit   : 576b04763477877d4f2a61ad692703e20471901b    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Oct 2021 16:25:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 11 Oct 2021 16:25:36 -0400    

Click here for diff

M src/build/config/config.yaml
M src/config/parse.auto.c

Fix indentation.

commit   : 980b777a4a0cc07200ccb06cc55b89181101b266    
  
author   : David Steele <[email protected]>    
date     : Sat, 9 Oct 2021 12:39:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 9 Oct 2021 12:39:54 -0400    

Click here for diff

M src/common/type/pack.h

Add assert in pckReadNext() to prevent reading into a field.

commit   : cc7f2eea900d0a7a429ffdc2cd45ead88c0298a9    
  
author   : David Steele <[email protected]>    
date     : Sat, 9 Oct 2021 12:37:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 9 Oct 2021 12:37:25 -0400    

Click here for diff

M src/common/type/pack.c

Increase tolerance for 0ms sleep in common/time test.

commit   : 610bfd736ef091a8298e5602d41aba86f10189bb    
  
author   : David Steele <[email protected]>    
date     : Sat, 9 Oct 2021 12:34:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 9 Oct 2021 12:34:45 -0400    

Click here for diff

M test/src/module/common/timeTest.c

Read tag size in pckReadTagNext().

commit   : 7ab8dcbe6e007831e8a7f9f26e08f083c1026388    
  
author   : David Steele <[email protected]>    
date     : Sat, 9 Oct 2021 12:15:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 9 Oct 2021 12:15:19 -0400    

Click here for diff

Rather than reading the size everywhere it is needed, get it when the tag is read, if it exists.  
  
This simplifies logic around consuming the data when not needed. There are more use cases for this coming up.  

M src/common/type/pack.c

Rename strNewN() to strNewZN().

commit   : ed68792e765411a994d8ac79e4d047bbafc25582    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 19:57:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 19:57:28 -0400    

Click here for diff

Make the function name consistent with other functions that accept zero-terminated strings, e.g. strNewZ() and strCatZN().  

M src/command/archive/common.c
M src/command/control/stop.c
M src/command/help/help.c
M src/common/ini.c
M src/common/io/read.c
M src/common/io/tls/client.c
M src/common/lock.c
M src/common/regExp.c
M src/common/type/json.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/variant.h
M src/config/parse.c
M src/storage/posix/storage.c
M src/storage/storage.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/logTest.c
M test/src/module/common/typeStringTest.c

More efficient memory allocation for Strings and String Variants.

commit   : b7e17d80ea02d70e56327828310c578af51795b5    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 19:43:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 19:43:28 -0400    

Click here for diff

The vast majority of Strings are never modified so for most cases allocate memory for the string with the object. This results in one allocation in most cases instead of two. Use strNew() if strCat*() functions are needed.  
  
Update varNewStr() in the same way since String Variants can never be modified. This results in one allocation in all cases instead of three. Also update varNewStrZ() to use STR() instead of strNewZ() to save two more allocations.  

M src/build/common/render.h
M src/build/config/render.c
M src/build/error/render.c
M src/build/help/render.c
M src/command/archive/get/get.c
M src/command/archive/push/file.c
M src/command/backup/common.c
M src/command/command.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/restore/restore.c
M src/command/verify/verify.c
M src/common/exit.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/http/request.c
M src/common/type/json.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/variant.c
M src/config/load.c
M src/config/parse.c
M src/db/db.c
M src/info/info.c
M src/info/manifest.c
M src/postgres/client.c
M src/protocol/server.c
M src/storage/gcs/storage.c
M src/storage/helper.c
M src/storage/s3/storage.c
M test/src/common/harnessLog.c
M test/src/common/harnessStorage.c
M test/src/module/command/verifyTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/performance/typeTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/s3Test.c

Use constant string for user/group in performance/type test.

commit   : 208641ac7fd22f676a55e7305b3e69df574f36f8    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 18:50:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 18:50:56 -0400    

Click here for diff

It is not safe to return strings created with STRDEF() from a function.  

M test/src/module/performance/typeTest.c

More efficient generation of diff/incr backup label.

commit   : 74d3131830646c9f71ffa9847729cae40e3aa866    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 14:58:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 14:58:11 -0400    

Click here for diff

M src/command/backup/common.c

Allow "global" as a stanza prefix.

commit   : 498902e885c9d7a44648ccc01b20454094d5b742    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 12:18:24 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 7 Oct 2021 12:18:24 -0400    

Click here for diff

A stanza name like global_stanza was not allowed because the code was not selective enough about how a global section should be formatted.
  

  
Update the config parser to correctly recognize global sections.

M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c

Improve instructions for rebuilding pgbackrest during stress testing.

commit   : 338102861fd0ea4d2773b010dee34a39a96ad702    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 11:01:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 Oct 2021 11:01:48 -0400    

Click here for diff

M doc/xml/user-guide.xml

Add configurable storage helpers to create repository storage.

commit   : fb3f6928c9aef499938e195fbb612c1940a2dc19    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Oct 2021 19:27:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 Oct 2021 19:27:04 -0400    

Click here for diff

Remove the hardcoded storage helpers from storageRepoGet() except for the the built-in Posix helper and the special remote helper.  
  
The goal is to make storage driver development a bit easier by isolating as much of the code as possible into the driver module. This also makes coverage reporting much simpler for additional drivers since they do not need to provide coverage for storage/helper.  
  
Consolidate the CIFS tests into the Posix tests since CIFS is just a special case of the Posix.  
  
Test all storage features in the Posix test so that other storage driver tests do not need to provide coverage for storage/storage.  
  
Remove some dead code in the storage/s3 test.  

M src/Makefile.in
M src/main.c
A src/storage/azure/helper.c
A src/storage/azure/helper.h
A src/storage/cifs/helper.c
A src/storage/cifs/helper.h
A src/storage/gcs/helper.c
A src/storage/gcs/helper.h
M src/storage/helper.c
M src/storage/helper.h
A src/storage/s3/helper.c
A src/storage/s3/helper.h
M src/storage/storage.intern.h
M test/define.yaml
M test/src/common/harnessStorage.c
M test/src/module/storage/azureTest.c
D test/src/module/storage/cifsTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Refactor S3 storage driver for additional auth methods.

commit   : cfd823355af2ac99f30c5e1393a121c6dbf622b7    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Oct 2021 12:38:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 Oct 2021 12:38:56 -0400    

Click here for diff

Currently only two auth methods are supported and a lot of refactoring is required to add a third one.  
  
Do the refactoring now to reduce noise in the commit that adds the third auth method.  

M src/storage/s3/storage.c

commit   : 68c5f3eaf18fc9bd10fde15a323890bcdcbf4534    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Oct 2021 17:59:05 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 5 Oct 2021 17:59:05 -0400    

Click here for diff

Currently link-map only allows links that exist in the backup manifest to be remapped to a new destination.
  

  
Allow link-map to create a new link as long as a valid path/file from the backup is referenced.

M doc/xml/release.xml
M src/command/restore/restore.c
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/command/restoreTest.c

Add state to ProtocolClient.

commit   : f2aeb30fc706c04d6200cefeeb2645229a31ff69    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Oct 2021 14:06:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Oct 2021 14:06:59 -0400    

Click here for diff

This is currently only useful for debugging, but in the future the state may be used for resetting the protocol when something goes wrong.  

M src/protocol/client.c
M src/protocol/client.h
M src/protocol/parallel.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M test/src/module/protocol/protocolTest.c

Add missing asserts and move temp mem context block.

commit   : 2c65fed80f47124283d4f8be92f987dc55237f48    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Oct 2021 12:29:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Oct 2021 12:29:16 -0400    

Click here for diff

M src/protocol/client.c

Report original error and retries on local job failure.

commit   : 6af827cbb1e78cd4c5d649ed4cb24c49a7204b8f    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Oct 2021 09:00:16 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 5 Oct 2021 09:00:16 -0400    

Click here for diff

The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync.
  

  
Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.

M doc/xml/release.xml
M src/protocol/server.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/verifyTest.c
M test/src/module/protocol/protocolTest.c

Report backup file validation errors in backup.info.

commit   : 34f78734325743b4e34cb39224852e6debf49750    
  
author   : Stefan Fercot <[email protected]>    
date     : Mon, 4 Oct 2021 19:45:53 +0200    
  
committer: GitHub <[email protected]>    
date     : Mon, 4 Oct 2021 19:45:53 +0200    

Click here for diff

Currently errors found during the backup are only available in text output when specifying --set.
  

  
Add a flag to backup.info that is available in both the text and json output when --set is not specified. This at least provides the basic info that an error was found in the cluster during the backup, though details are still only available as described above.

M doc/xml/release.xml
M src/command/info/info.c
M src/info/infoBackup.c
M src/info/infoBackup.h
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/infoTest.c
M test/src/module/info/infoBackupTest.c

Add arm64 testing on Cirrus CI.

commit   : 57c62315465972f6b85558020198134e34cf2ee0    
  
author   : David Steele <[email protected]>    
date     : Sat, 2 Oct 2021 17:27:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 2 Oct 2021 17:27:33 -0400    

Click here for diff

These tests run in a container without permissions to mount tempfs, so add an option to ci.pl to not create tempfs. Also add some packages not in the base image.  

M .cirrus.yml
M test/ci.pl

Sort WAL segment names when reporting duplicates.

commit   : f1ed8f0e5112d1a74d86168e67632be55eddb416    
  
author   : David Steele <[email protected]>    
date     : Sat, 2 Oct 2021 16:29:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 2 Oct 2021 16:29:31 -0400    

Click here for diff

Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output.  
  
Found on arm64.  

M src/command/archive/get/get.c
M test/src/module/command/archiveGetTest.c

Use strncpy() to limit characters copied to optionName.

commit   : 71047a9d6d1eea71b3fbd430983541a54049cc69    
  
author   : David Steele <[email protected]>    
date     : Sat, 2 Oct 2021 16:17:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 2 Oct 2021 16:17:33 -0400    

Click here for diff

Valgrind complained about uninitialized values on arm64 when comparing the reset prefix, probably because "reset" ended up being larger than the option name: Conditional jump or move depends on uninitialised value(s) at cfgParseOption (parse.c:568).  
  
Coverity complained because it could not verify the size of the string to be copied into optionName, probably because it does not understand the purpose of strSize(): You might overrun the 65-character fixed-size string "optionName" by copying the return value of "strZ" without checking the length.  
  
Use strncpy() even though we have already checked the size and make sure the string is terminated. Keep the size check because searching for truncated option names is not a good idea.  
  
This is not a production bug since the code has not been released yet.  

M doc/xml/release.xml
M src/config/parse.c

Use temp mem context when calling command handlers.

commit   : b792a14cd7dbdfb61362700ffc5fc01997db890c    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 Oct 2021 18:23:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 1 Oct 2021 18:23:03 -0400    

Click here for diff

It is safer and more efficient to free memory after each handler completes.  
  
The db command handlers use the server context so update them to use the top context.  

M src/db/protocol.c
M src/protocol/server.c

Add jobRetry parameter to HRN_CFG_LOAD().

commit   : ae40ed6ec9cf77e577518b128518a5763767f589    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 Oct 2021 17:15:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 1 Oct 2021 17:15:36 -0400    

Click here for diff

Allow the default of 0 to be overridden to test retry behavior for commands.  

M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/common/harnessProtocol.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/verifyTest.c

Allow stack trace to be specified for errorInternalThrow().

commit   : 136d309dd4bc1ada9f3d775f036b62292fda390b    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 Oct 2021 15:29:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 1 Oct 2021 15:29:31 -0400    

Click here for diff

This allows the stack trace to be set when an error is received by the protocol, rather than appending it to the message. Now these errors will look no different than any other error and the stack trace will be reported in the same way.  
  
One immediate benefit is that test.pl --vm-out --log-level-test=debug will work for tests that check expect log results. Previously, the test would error at the first check because the stack trace included in the message would not match the expected log output.  

M src/common/error.c
M src/common/error.h
M src/protocol/client.c
M test/define.yaml
M test/src/module/common/errorTest.c
M test/src/module/protocol/protocolTest.c

Update file mode in info/manifest test to 0600.

commit   : 62f6fbe2a9ecd9bc48611ecfaf27a05b9f36a87d    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 Oct 2021 10:15:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 1 Oct 2021 10:15:34 -0400    

Click here for diff

0400 is not a very realistic mode. It may have become the default due to copy-pasting.  

M test/src/module/info/manifestTest.c

Remove repository format 6 notes.

commit   : 0690cb25a077735780b2fe24343c946f8a4efbc6    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 17:55:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 17:55:38 -0400    

Click here for diff

The notes have been moved to a Github project.  

D doc/FORMAT.md

Move archive-header-check option to the archive reference section.

commit   : 376362475e3b3b13b70313d9002d4ff8b25b4b40    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 16:15:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 16:15:45 -0400    

Click here for diff

M src/build/config/config.yaml
M src/build/help/help.xml

commit   : cf1a57518fe3230886509f59fcec9c9a81e6513c    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 14:29:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 14:29:49 -0400    

Click here for diff

This will allow new links to be added in a future commit. The current implementation is driven by the links that already exist in the manifest, which would make the new use case more complex to implement.  
  
Also, add a more helpful error when a tablespace link is specified.  

M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Refactor restoreManifestMap() tests in the command/restore unit.

commit   : d89a67776cfbb7b3047dbe297cc0c768e5c670e8    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 13:39:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 13:39:29 -0400    

Click here for diff

Add test titles, new tests, and rearrange.  
  
Also manifestTargetFindDefault(), which will soon be used by core code in a refactoring commit.  

M src/info/manifest.h
M test/src/module/command/restoreTest.c

Add note to comment for int64 typedef.

commit   : 7a53ba7c7f38bff2c9ef99c6ab58d22c59dd2290    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 13:28:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 13:28:14 -0400    

Click here for diff

M src/postgres/interface/version.vendor.h

Finalize catalog number for PostgreSQL 14 release.

commit   : 815377cc6009c800b6a5fdc1fe98ddfceaaae824    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 13:27:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Sep 2021 13:27:14 -0400    

Click here for diff

M src/postgres/interface/version.vendor.h
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Fix comment typos.

commit   : baf186bfb05ca683714a59485900892f5e7e8a1b    
  
author   : Stefan Fercot <[email protected]>    
date     : Wed, 29 Sep 2021 12:03:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 29 Sep 2021 12:03:01 -0400    

Click here for diff

M src/command/backup/backup.c
M src/command/restore/restore.c
M test/src/module/command/infoTest.c

Add recovery start time to online backup restore log.

commit   : 9e79f0e64b661e944ce2b3897c366feea1544ac2    
  
author   : David Steele <[email protected]>    
date     : Wed, 29 Sep 2021 10:31:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 29 Sep 2021 10:31:51 -0400    

Click here for diff

This helps give an idea of how much recovery needs to be done to reach the end of the WAL stream and is easier to read than the backup label.  

M doc/xml/release.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Rename page checksum error to error list in info text output.

commit   : 9346895f5b61627b50d431f09447e08d8a50caa8    
  
author   : David Steele <[email protected]>    
date     : Wed, 29 Sep 2021 09:58:47 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 29 Sep 2021 09:58:47 -0400    

Click here for diff

"error list" makes it clearer that other errors may be reported. For example, if checksum-page is true in the manifest but no checksum-page-error list is provided then the error is in alignment, i.e. the file size is not a multiple of the page size, with allowances made for a valid-looking partial page at the end of the file.
  

  
It is still not possible to differentiate between alignment and page checksum errors in the output but this will be addressed in a future commit.

M doc/xml/release.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c

Add hints to standby replay timeout message.

commit   : b7ef12a76f219881d0b24c715592af96fe5c9b8f    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Sep 2021 15:55:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Sep 2021 15:55:13 -0400    

Click here for diff

M doc/xml/release.xml
M src/db/db.c
M test/src/module/db/dbTest.c

Add repo-azure-uri-style option.

commit   : 096829b3b257444162417662612e80d8cb2ac6ec    
  
author   : David Steele <[email protected]>    
date     : Mon, 27 Sep 2021 09:01:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 27 Sep 2021 09:01:53 -0400    

Click here for diff

Azurite introduced a breaking change in 8f63964e to use automatically host-style URIs when the endpoint appears to be a multipart hostname.  
  
This option allows the user to configure which style URI will be used, but changing the endpoint might cause breakage if Azurite decides to use a different style. Future changes to Azurite may also cause breakage.  

M doc/xml/user-guide.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/config/config.auto.h
M src/config/parse.auto.c
M src/storage/azure/storage.c
M src/storage/azure/storage.h
M src/storage/helper.c
M test/lib/pgBackRestTest/Env/Host/HostAzureTest.pm
M test/src/module/command/helpTest.c
M test/src/module/storage/azureTest.c

Convert page checksum filter result to a pack.

commit   : c8ea17c68f8fa72f2bf3b979539be3f709448493    
  
author   : David Steele <[email protected]>    
date     : Fri, 24 Sep 2021 17:40:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 24 Sep 2021 17:40:31 -0400    

Click here for diff

The pack is both more compact and more efficient than a variant.  
  
Also aggregate the page error info in the main process rather than in the filter to allow additional LSN filtering, to be added in a future commit.  

M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/pageChecksum.c
M src/command/backup/protocol.c
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/type/pack.h
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/common/typePackTest.c
M test/src/module/storage/remoteTest.c

Centralize and optimize tag stack management.

commit   : ac1f6db4a25520b1ee957b66925dde2e1ef156ab    
  
author   : David Steele <[email protected]>    
date     : Thu, 23 Sep 2021 14:06:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 23 Sep 2021 14:06:00 -0400    

Click here for diff

The push and pop code was duplicated in four places, so centralize the code into pckTagStackPop() and pckTagStackPush().  
  
Also create a default bottom item for the stack to avoid allocating a list if there will only ever be the default container, which is very common. This avoids the extra time and memory to allocate a list.  

M src/common/type/pack.c
M test/src/module/common/typePackTest.c

Add Pack pseudo-type.

commit   : 15e7ff10d3d6fe3570335a5abec5ff683c07e2e6    
  
author   : David Steele <[email protected]>    
date     : Thu, 23 Sep 2021 08:31:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 23 Sep 2021 08:31:32 -0400    

Click here for diff

Rather than working directly with Buffer types, define a new Pack pseudo-type that represents a Buffer containing a pack. This makes it clearer that a pack is being stored and allows stronger typing.  

M src/build/help/render.c
M src/command/backup/pageChecksum.c
M src/command/backup/pageChecksum.h
M src/common/compress/bz2/compress.c
M src/common/compress/gz/compress.c
M src/common/compress/helper.c
M src/common/compress/helper.h
M src/common/compress/lz4/compress.c
M src/common/compress/zst/compress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/cipherBlock.h
M src/common/crypto/hash.c
M src/common/crypto/hash.h
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/size.c
M src/common/io/write.intern.h
M src/common/type/pack.c
M src/common/type/pack.h
M src/info/info.c
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.c
M src/protocol/server.c
M src/protocol/server.h
M src/storage/gcs/write.c
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/write.c
M test/src/common/harnessInfo.c
M test/src/common/harnessPack.c
M test/src/common/harnessPack.h
M test/src/module/build/helpTest.c
M test/src/module/command/backupTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/typePackTest.c
M test/src/module/storage/remoteTest.c

Rename pckReadNew()/pckWriteNew() to pckReadNewIo()/pckWriteNewIo().

commit   : 131ac0ab5e98500569d4ff6985d31c55a9ef53b9    
  
author   : David Steele <[email protected]>    
date     : Wed, 22 Sep 2021 11:18:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 22 Sep 2021 11:18:12 -0400    

Click here for diff

These names more accurately describe the purpose of the constructors.  

M src/command/help/help.c
M src/common/type/pack.c
M src/common/type/pack.h
M src/protocol/client.c
M src/protocol/command.c
M src/protocol/server.c
M test/src/module/common/typePackTest.c

Convert filter param/result to Pack type.

commit   : 0e76ccb5b7b2089f8d0300ab0086c454aebbbbbf    
  
author   : David Steele <[email protected]>    
date     : Wed, 22 Sep 2021 10:48:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 22 Sep 2021 10:48:21 -0400    

Click here for diff

The Pack type is more compact and flexible than the Variant type. The Pack type also allows binary data to be stored, which is useful for transferring the passphrase in the CipherBlock filter.  
  
The primary purpose is to allow more (and more complex) result data to be returned efficiently from the PageChecksum filter. For now the PageChecksum filter still returns the original Variant. Converting the result data will be the subject of a future commit.  
  
Also convert filter types to StringId.  

M src/command/archive/push/file.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/pageChecksum.c
M src/command/backup/pageChecksum.h
M src/command/restore/file.c
M src/command/verify/file.c
M src/command/verify/verify.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/compress.h
M src/common/compress/bz2/decompress.c
M src/common/compress/bz2/decompress.h
M src/common/compress/gz/compress.c
M src/common/compress/gz/compress.h
M src/common/compress/gz/decompress.c
M src/common/compress/gz/decompress.h
M src/common/compress/helper.c
M src/common/compress/helper.h
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/compress.h
M src/common/compress/lz4/decompress.c
M src/common/compress/lz4/decompress.h
M src/common/compress/zst/compress.c
M src/common/compress/zst/compress.h
M src/common/compress/zst/decompress.c
M src/common/compress/zst/decompress.h
M src/common/crypto/cipherBlock.c
M src/common/crypto/cipherBlock.h
M src/common/crypto/hash.c
M src/common/crypto/hash.h
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/sink.c
M src/common/io/filter/sink.h
M src/common/io/filter/size.c
M src/common/io/filter/size.h
M src/common/type/pack.c
M src/common/type/pack.h
M src/info/info.c
M src/storage/gcs/write.c
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/write.c
M test/define.yaml
M test/src/common/harnessInfo.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/ioTest.c
M test/src/module/performance/storageTest.c
M test/src/module/storage/remoteTest.c

Limit valgrind error output to the first error.

commit   : 802373cb9df28384529fe5a7bd102bfe5c8f3911    
  
author   : David Steele <[email protected]>    
date     : Tue, 21 Sep 2021 10:16:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Sep 2021 10:16:16 -0400    

Click here for diff

Generally the first error is the only important error. The rest simply lead to a lot of scrolling.  

M test/lib/pgBackRestTest/Common/JobTest.pm

Copy data page before verifying checksum.

commit   : 473afce57bc7646c53bae4a6300b41b11b5b0357    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Sep 2021 11:03:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Sep 2021 11:03:50 -0400    

Click here for diff

Using UNCONSTIFY() is potentially dangerous since the buffer is modified while calculating the checksum, even though the page is reverted to the original state. Instead make a copy to ensure that the original data is never modified.  
  
This requires the logic to be shuffled a bit since the copy cannot be made until we are sure the page is complete.  

M src/command/backup/pageChecksum.c

Automatically include all PostgreSQL version interface files.

commit   : 0efb8adb9452cb8bc67ebaa8b6dca0d4a69c1682    
  
author   : David Steele <[email protected]>    
date     : Sun, 19 Sep 2021 20:38:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 19 Sep 2021 20:38:51 -0400    

Click here for diff

M src/Makefile.in

Specify size for helpData array.

commit   : 95d814cf81b84d162f40717346c3ad0cb642f724    
  
author   : David Steele <[email protected]>    
date     : Sun, 19 Sep 2021 20:32:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 19 Sep 2021 20:32:27 -0400    

Click here for diff

M src/build/help/render.c

Skip comments when rendering help output.

commit   : 912a498b0bcc988bd5ebaaa22198956d752251bd    
  
author   : David Steele <[email protected]>    
date     : Sat, 11 Sep 2021 16:07:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 11 Sep 2021 16:07:59 -0400    

Click here for diff

Comments should not appear in the help. They are simply notes on implementation.  

M src/build/help/render.c
M test/src/module/build/helpTest.c

Revert Azurite version for testing to 3.14.0.

commit   : c38d6926d6c9aa01b895a28e66fc0aa6965350a3    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Sep 2021 08:48:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Sep 2021 08:48:45 -0400    

Click here for diff

3.14.2 is causing breakage in the documentation. There is no obvious cause so for now just revert to the last working version.  

M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Env/Host/HostAzureTest.pm

Migrate command-line help generation to C.

commit   : f4e1babf6b4ce7087ace8221cac7cadb51488f0e    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Sep 2021 18:16:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 8 Sep 2021 18:16:06 -0400    

Click here for diff

Command-line help is now generated at build time so it does not need to be committed. This reduces churn on commits that add configuration and/or update the help.  
  
Since churn is no longer an issue, help.auto.c is bzip2 compressed to save space in the binary.  
  
The Perl config parser (Data.pm) has been moved to doc/lib since the Perl build path is no longer required.  
  
Likewise doc/xml/reference.xml has been moved to src/build/help/help.xml since it is required at build time.  

M CONTRIBUTING.md
D build/lib/pgBackRestBuild/Build.pm
D build/lib/pgBackRestBuild/Build/Common.pm
D build/lib/pgBackRestBuild/Config/BuildHelp.pm
M doc/doc.pl
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/lib/pgBackRestDoc/Common/DocManifest.pm
M doc/lib/pgBackRestDoc/Common/DocRender.pm
R099 build/lib/pgBackRestBuild/Config/Data.pm doc/lib/pgBackRestDoc/Custom/DocConfigData.pm
M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
M doc/manifest.xml
M doc/xml/contributing.xml
M src/.gitignore
M src/Makefile.in
M src/build/config/parse.c
M src/build/config/parse.h
R100 doc/xml/reference.xml src/build/help/help.xml
A src/build/help/main.c
A src/build/help/parse.c
A src/build/help/parse.h
A src/build/help/render.c
A src/build/help/render.h
D src/command/help/help.auto.c
M src/command/help/help.c
M src/command/help/help.h
M src/main.c
M test/code-count/file-type.yaml
M test/define.yaml
A test/src/module/build/helpTest.c
M test/src/module/command/helpTest.c
M test/test.pl

Eliminate linefeed formatting from documentation.

commit   : def7d513cdd2d4579acf6e8c675a3d6f7da4f655    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Sep 2021 17:35:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 8 Sep 2021 17:35:45 -0400    

Click here for diff

Linefeeds were originally used in the place of <p> tags to denote a paragraph. While much of the linefeed usage has been replaced over time, there were many places where it was still being used, especially in reference.xml. This made it difficult to get consistent formatting across different output types. In particular there were formatting issues in the command-line help because it is harder to audit than HTML or PDF.  
  
Replace linefeed formatting with proper <p> tags to make formatting more consistent.  
  
Remove double spaces in all text where <p> tags were added since it does not add churn.  
  
Update all <ul>/<ol>/<li> tags to the more general <list>/<list-item> tags.  
  
Add a few missing periods.  

M doc/lib/pgBackRestDoc/Common/Doc.pm
M doc/lib/pgBackRestDoc/Common/DocRender.pm
M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
M doc/lib/pgBackRestDoc/Html/DocHtmlBuilder.pm
M doc/xml/dtd/doc.dtd
M doc/xml/index.xml
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c

Add initializer braces required by older versions of gcc.

commit   : 245b0c048ae62eb174b534d156b7a00b1de19a53    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Sep 2021 13:32:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 8 Sep 2021 13:32:13 -0400    

Click here for diff

M src/build/common/yaml.c

Remove invalid comment block.

commit   : 38eeac48d617ec5d5f14a1cb612c9a8b721326d8    
  
author   : David Steele <[email protected]>    
date     : Fri, 3 Sep 2021 10:39:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 3 Sep 2021 10:39:30 -0400    

Click here for diff

M src/common/io/tls/session.h

Build newer valgrind and move CA cert install in test containers.

commit   : 1afea449e2f54e69f9f8f0d772a294fcf09e0d83    
  
author   : David Steele <[email protected]>    
date     : Thu, 2 Sep 2021 13:26:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 2 Sep 2021 13:26:21 -0400    

Click here for diff

The newer version of valgrind helps with some arm64 issues that have been fixed since the architecture has become more popular. Also add the valgrind builds to the Vagrantfile and Dockerfile.  
  
Move the CA cert install from the base container to the test container. This means the CA cert can be changed without rebuilding all the base containers.  

M test/Dockerfile
M test/Vagrantfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Allow additional memory to be allocated with a mem context.

commit   : 475b57c89b58e9709a3eafd08fe9949b467a99cd    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Sep 2021 11:10:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Sep 2021 11:10:35 -0400    

Click here for diff

The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer.  
  
Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size.  
  
Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall.  
  
Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/build/common/yaml.c
M src/command/backup/pageChecksum.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/compress/zst/compress.c
M src/common/compress/zst/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/exec.c
M src/common/exec.h
M src/common/ini.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/client.c
M src/common/io/client.h
M src/common/io/fdRead.c
M src/common/io/fdWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/sink.c
M src/common/io/filter/size.c
M src/common/io/http/client.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/common/io/http/response.c
M src/common/io/http/response.h
M src/common/io/http/session.c
M src/common/io/http/url.c
M src/common/io/http/url.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/session.c
M src/common/io/session.h
M src/common/io/socket/client.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/session.c
M src/common/io/write.c
M src/common/io/write.h
M src/common/memContext.c
M src/common/memContext.h
M src/common/regExp.c
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/keyValue.c
M src/common/type/keyValue.h
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/mcv.c
M src/common/type/object.c
M src/common/type/object.h
M src/common/type/pack.c
M src/common/type/xml.c
M src/common/type/xml.h
M src/common/wait.c
M src/common/wait.h
M src/config/parse.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M src/postgres/client.c
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M src/protocol/server.c
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/cifs/storage.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/read.h
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/write.h
M test/src/common/harnessProtocol.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/execTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeObjectTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

Increase max index allowed for pg/repo options to 256.

commit   : 02b06aa4957cf3fe64ceb7a44cc1893bff12fe99    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 Aug 2021 12:09:50 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 31 Aug 2021 12:09:50 -0400    

Click here for diff

The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained.
  

  
Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly.
  

  
Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX.
  

  
Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior.
  

  
cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/config/parse.c
M src/build/config/parse.h
M src/build/config/render.c
M src/command/help/help.c
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/build/configTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/storage/remoteTest.c

Add required header for Illumos.

commit   : 8a4063c2b69919d1bba305d986568f529139eaa3    
  
author   : John W. Terrell <[email protected]>    
date     : Fri, 27 Aug 2021 03:41:56 -0700    
  
committer: GitHub <[email protected]>    
date     : Fri, 27 Aug 2021 03:41:56 -0700    

Click here for diff

M src/common/io/fd.c

Do not use certs in container for testing.

commit   : 5d857e196a6eeb2e1d137f2faf9001cccc066cfd    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 Aug 2021 07:33:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Aug 2021 07:33:31 -0400    

Click here for diff

The certs are available in test/certificate so it makes more sense to use them there. In addition the container does not need to be rebuilt unless the CA cert changes.  

M test/src/common/harnessServer.c

Add missing header.

commit   : 7b4d69211e64855f9ca0db0e4b2950a6492afa59    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 16:09:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 16:09:14 -0400    

Click here for diff

M src/common/io/tls/client.h

Add const to memContextFreeing()/memContextName() parameters.

commit   : beafdbfc1873902f4b17fbbc5f88e47d6344d113    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 08:37:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 08:37:57 -0400    

Click here for diff

These functions do not modify the mem context so they can use const pointers.  

M src/common/memContext.c
M src/common/memContext.h

Remove useless if in memContextFree().

commit   : 726d04f13a2217b78e2cbef48ef4c889f15b5b68    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 08:31:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 08:31:50 -0400    

Click here for diff

If the upper bound of the loop is 0 then nothing will be done, making the if condition extraneous.  

M src/common/memContext.c

Use contextParentIdx instead of a loop when moving a context.

commit   : d72d4415a7c79296e3160aede9df59cd5d959694    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 08:27:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 08:27:07 -0400    

Click here for diff

contextParentIdx was introduced in 90709dfd to improve the performance of mem context frees. memContextMove() did not get the message, however, and continued to use a loop to find the mem context in the old parent.  
  
Use contextParentIdx to find the mem context in the old parent to avoid a loop.  

M src/common/memContext.c
M test/src/module/common/memContextTest.c

Remove useless memFree() in regExpNew().

commit   : 8a9887f27cb46728e85801e075474928207e012f    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 07:58:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 07:58:59 -0400    

Click here for diff

The MEM_CONTEXT_NEW*() block will automatically free memory if there is an error.  
  
This code is pretty old so this may not have been true when it was written.  

M src/common/regExp.c

Update config.guess and config.sub to latest versions.

commit   : 5ffb7cc77127a60de98e2d9080f27ed28bf95b4f    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 07:14:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 07:14:22 -0400    

Click here for diff

M doc/RELEASE.md
M src/build/config.guess
M src/build/config.sub

Begin v2.36 development.

commit   : aee0e7bac746c5c68baee106d8d4809cf6d8edfd    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 07:03:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 07:03:40 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h