pgBackRest 2.40 commit log

v2.39: Verify and File Bundling

commit   : 901e829f6dd05fdc15d899bc7f1108883002b1d5    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 May 2022 08:46:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 May 2022 08:46:24 -0400    

Click here for diff

Bug Fixes:  
  
* Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.)  
* Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson, Geir Råness. Reported by Geir Råness.)  
  
Features:  
  
* Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.)  
* Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.)  
* PostgreSQL 15 support. (Reviewed by Stefan Fercot.)  
* Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.)  
* Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.)  
  
Improvements:  
  
* Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.)  
* Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.)  
* Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.)  
* Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.)  
* Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)  
* Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.)  
* Add user:group to lock permission error. (Reviewed by Reid Thompson.)  
  
Documentation Bug Fixes:  
  
* Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.)  
* Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.)  
* Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.)  
  
Documentation Improvements:  
  
* Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.)  
* Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.)  
* Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)  

M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Fix issues in improved path validation for repo-* commands.

commit   : 19dd015d589ed3efd31a2a238249a150fb3748c1    
  
author   : David Steele <[email protected]>    
date     : Fri, 13 May 2022 09:41:53 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 13 May 2022 09:41:53 -0400    

Click here for diff

If the user requested the exact repo path then strSub() would be passed an invalid start value leading to an assertion:
  

  
$ pgbackrest --stanza=test repo-ls /var/lib/pgbackrest
  
ASSERT: [025]: start <= this->pub.size (on dev builds)
  
ASSERT: [025]: string size must be <= 1073741824 bytes (on prod builds)
  

  
Fix this by checking if the requested path exactly equals the repo path and returning an empty relative path in this case.
  

  
Another issue was that invalid subpaths were not detected if they started with the repo path. For example, /var/lib/pgbackrestsub would not generate an error if the repo path was /var/lib/pgbackrest. Fix this by explictly checking for a / between the repo path and the subpath. This also requires special handling when the repo path is /.
  

  
This is not a live bug since the issues were found in an unreleased feature introduced in 5ae84d5.

M doc/xml/release.xml
M src/command/repo/common.c
M test/src/module/command/repoTest.c

Reduce encrypted WAL segments sizes in command tests.

commit   : 024500782e9fde91cf236db70a584beb52586d38    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 May 2022 17:02:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 May 2022 17:02:08 -0400    

Click here for diff

The encrypted archive-push and repo tests were running very slowly on 32-bit with Valgrind enabled. This appears to be an issue with a newer version of Valgrind, but it has been going on long enough that bisecting does not seem to be worthwhile.  
  
Reduce the size of the encrypted test segments where possible to improve overall test performance.  

M test/src/module/command/archivePushTest.c
M test/src/module/command/repoTest.c

Move 32-bit CI testing from Debian 9 to Ubuntu 18.04.

commit   : 7d9b2e267c3f68463000cfcfec6018cc12203f23    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 May 2022 11:57:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 May 2022 11:57:12 -0400    

Click here for diff

Debian 9 will be EOL before our next release, so move 32-bit testing to Ubuntu 18.04, which is older than Debian 10.  

M .github/workflows/test.yml
M test/container.yaml
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/src/module/common/ioTlsTest.c

Remove excessive const usage in build module.

commit   : 24f138b0672ed55aeef1673b816b803487bca932    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 May 2022 16:20:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 May 2022 16:20:41 -0400    

Click here for diff

This was an experiment that attempted to create immutable structs (at least without casting). It turned out to be a bit burdensome and required unsafe-looking casting in some cases.  

M src/build/config/parse.c
M src/build/config/parse.h
M src/build/error/parse.h

Move some PostgreSQL integration tests to rh7.

commit   : 3d8ee552fe85aae6f2f00cd2a9436fb6b34513e1    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 May 2022 15:59:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 May 2022 15:59:37 -0400    

Click here for diff

This makes the u20 test run faster. Since u20 is currently the longest running test, the time to complete all tests is also improved.  

M test/lib/pgBackRestTest/Common/VmTest.pm

Add option type descriptions.

commit   : a913113fda5886ffd2ff136cdd52a3f469eca026    
  
author   : Reid Thompson <[email protected]>    
date     : Wed, 11 May 2022 15:23:41 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 11 May 2022 15:23:41 -0400    

Click here for diff

This cuts down on repetition of the size descriptions and adds basic descriptions for the other option types.

M doc/xml/release.xml
M src/build/help/help.xml
M test/src/module/command/helpTest.c

Add br tag for documentation.

commit   : 5fbea6da811ee93dcd966b8f7d84d6da2e65c6c5    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 May 2022 10:39:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 May 2022 10:39:31 -0400    

Click here for diff

This tag allows for a simple linefeed in a p tag instead of being forced to start a new paragraph.  

M doc/lib/pgBackRestDoc/Common/DocRender.pm
M doc/xml/dtd/doc.dtd

Fix "that that" typos.

commit   : 8ee85bc605da4aff5069bd1b965bbb585d9fe851    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 May 2022 08:46:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 May 2022 08:46:23 -0400    

Click here for diff

M doc/xml/release.xml
M src/common/log.h
M test/lib/pgBackRestTest/Common/Storage.pm

Fix typos in help.

commit   : c4f7edef2b6a9b541e94793c1dabbff7db854a9b    
  
author   : Reid Thompson <[email protected]>    
date     : Wed, 11 May 2022 08:42:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 May 2022 08:42:46 -0400    

Click here for diff

M src/build/help/help.xml

Add FAQ about backup types and restore speed.

commit   : 50d409a812be577f2c296f3d6096cf97fab049ff    
  
author   : David Christensen <[email protected]>    
date     : Tue, 10 May 2022 13:17:05 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 10 May 2022 13:17:05 -0500    

Click here for diff

Based on several questions/misunderstandings, provide clarification about the backup type only affecting the backup action, and not the restore.

M doc/xml/faq.xml
M doc/xml/release.xml

Remove integration expect log testing.

commit   : de816a0f5793747a774b98af627776112cc7cb2d    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 May 2022 13:18:26 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 10 May 2022 13:18:26 -0400    

Click here for diff

Integration expect log testing was originally used as a rough-and-ready way to make sure that certain code paths were being executed before the unit tests existed. Now that we have 100% unit test coverage (with expect log testing) the value of the integration expect tests seems minimal at best.
  

  
But they do cause numerous issues:
  

  
- Maintenance of the expect code and replacements that are required to keep logs reproducible.
  
- Even a trivial change can cause massive churn in the expect logs, e.g. d9088b2. These changes should be minutely audited but since the expect logs have little value now it is seldom worth the effort.
  
- The OS version used to do expect testing (RHEL7) can only be used to test one version of PostgreSQL. This makes it hard to balance the PostgreSQL version testing between OS versions.
  
- When a commit affects expect logs it is not clear (especially for new developers) how to regenerate them and our contributing guide is silent on the issue.
  

  
The goal is to migrate the integration tests to C and expect testing is not part of that plan. It seems best to get rid of them now.

M doc/xml/release.xml
M test/code-count/file-type.yaml
D test/expect/mock-all-001.log
D test/expect/mock-all-002.log
D test/expect/mock-archive-001.log
D test/expect/mock-archive-002.log
D test/expect/mock-archive-stop-001.log
D test/expect/mock-archive-stop-002.log
D test/expect/mock-expire-001.log
D test/expect/mock-expire-002.log
D test/expect/mock-stanza-001.log
D test/expect/mock-stanza-002.log
D test/expect/real-all-001.log
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/ExecuteTest.pm
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
D test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/test.pl

Remove obsolete test in common/memContext.

commit   : 3a403944729e03c66de83e6750c38cab97743333    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 May 2022 11:18:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 May 2022 11:18:12 -0400    

Click here for diff

Once upon a time the allocation array was allocated up front so this test was required for the top context, which did not allocate up front.  
  
Now allocations are done on demand so this case is covered for every context that does not allocate memory.  

M test/src/module/common/memContextTest.c

Fix typo.

commit   : 6b98b3534e15a8d277547eb2932c085bc7515eec    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 10 May 2022 06:52:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 May 2022 06:52:56 -0400    

Click here for diff

M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Document required base branch for pull requests.

commit   : cc5b0614894e1b5ef78f1ceba3bc8e4760a40a24    
  
author   : David Christensen <[email protected]>    
date     : Mon, 9 May 2022 17:07:11 -0500    
  
committer: GitHub <[email protected]>    
date     : Mon, 9 May 2022 17:07:11 -0500    

Click here for diff

Be explicit when submitting a PR about which branch to use as the base.

M CONTRIBUTING.md
M doc/xml/contributing.xml
M doc/xml/release.xml

Split 32-bit CI tests.

commit   : b4c1ca7b807a906d42d540307da3603052c4a674    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 May 2022 14:19:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 May 2022 14:19:05 -0400    

Click here for diff

This helps rebalance some of the tests that are running long, i.e. d9 and u20.  
  
I would be better to move more PostgreSQL versions to d9, but the base VM does not contain more versions. New minor versions will be out later in the week so that seems a better time to be rebuilding containers.  

M .github/workflows/test.yml
M test/lib/pgBackRestTest/Common/VmTest.pm

Add limited CI for ppc64le/s390x using emulation.

commit   : 39dddbb6bc7d296fe25a7210468b2f51ea6b1816    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 May 2022 12:48:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 May 2022 12:48:19 -0400    

Click here for diff

The emulation is so slow that running all the unit tests would be too expensive, but this at least shows that the build works and some of the more complex tests run. In particular, it is good to test on one big-endian architecture to be sure that checksums are correct.  
  
Update checksums in the tests where they had gotten out of date since the last time we were testing on s390x. Also use a different test in command/archivePushTest to show the name of the file when a checksum does not match to aid in debugging.  
  
The command/archive-push test was updated but not included because there is also a permissions issue, which looks to be the same as what we see on MacOS/FreedBSD. Hopefully we'll be able to fix all of those at the same time.  

M .github/workflows/test.yml
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c

Simplify messaging around supported versions in the documentation.

commit   : eefa0b161a6a02b7843e18455443a4bce41cd4b9    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 May 2022 11:59:08 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 9 May 2022 11:59:08 -0400    

Click here for diff

The version ranges given in the user guides caused confusion. For example, because the user guide for RHEL specified PostgreSQL 9.6-11, users questioned whether pgBackRest worked for PostgreSQL 12 on RHEL.
  

  
Remove these ranges and add more explanatory text to the introduction to try and make it clearer how the user guides work and which versions are covered (basically all of them).

M doc/xml/release.xml
M doc/xml/user-guide-index.xml
M doc/xml/user-guide.xml

Use variable instead of function to track FINALLY() state.

commit   : ef4c4ab8525f81c9ed22c7e82ed12096e81703c5    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 May 2022 10:39:43 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 9 May 2022 10:39:43 -0400    

Click here for diff

The function worked fine, but Coverity was unable to determine that the finally block was run, which led to false positives about unfreed memory.
  

  
Using a boolean in the block makes it clear to Coverity that the finally block will always be run no matter what else happens.
  

  
We'll depend on the compiler to optimize away the boolean if it is not used in a finally block. The cost of the boolean is fairly low in comparison to everything else being done in these macros, so it does not seem worth having a separate block even if the compiler is not able to eliminate the boolean.
  

  
This reverts most of 9a271e9 that fixed a bug caused by c5b5b58, which was also attempting to help Coverity understand FINALLY() blocks.

M doc/xml/release.xml
M src/common/error.c
M src/common/error.h
M test/src/module/common/errorTest.c

Remove unnecessary TRY() block in common/regExp module.

commit   : e8c40a24df14fcf281de672c19d0a55dc7cb50f6    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 May 2022 09:56:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 May 2022 09:56:19 -0400    

Click here for diff

This code was written before MEM_CONTEXT_TEMP*() was available, which is a better solution.  

M doc/xml/release.xml
M src/common/regExp.c

Remove legacy Travis-CI configuration.

commit   : 4d8c36715ddd67c11d1ab166dcd0b4850631d83d    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 19:44:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 19:44:46 -0400    

Click here for diff

Travis-CI is now strictly a paid service. Multiple attempts to use their "free" service have failed due to lack of community credit and general issues with their plugin.  
  
Remove the configuration so it does not appear we are testing on Travis-CI.  

D .travis.yml

Add hint when unable to find the primary cluster.

commit   : 46b7b7287446d0bb5ed0963fd01c51fb2b08893f    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 18:23:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 18:23:36 -0400    

Click here for diff

If all available clusters are in recovery, pgBackRest will not be able to find a primary for the backup.  

M src/db/helper.c
M test/src/module/db/dbTest.c

Remove useless test in config/parse unit test.

commit   : 53bfdbc01ece396c802022fea2216d3b9d7beaa1    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 16:02:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 16:02:44 -0400    

Click here for diff

Since the packSize field is 7 bits, it could never fail the check for > 127.  
  
The compiler will catch any packs that are larger than 7 bits and then the pack size will need to be adjusted. For now just adjust the comment to reflect what the test does and give a clearer indication of what to do when a pack grows too large.  

M test/src/module/config/parseTest.c

Fix indentation.

commit   : 77311a9af7870e79d7a3c9ecadc268c2364458c8    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 15:38:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 15:38:03 -0400    

Click here for diff

gcc11 complains about this indentation being misleading.  

M test/src/test.c

Update Docker test image for Debian 9.

commit   : efbcd975c4c5b4c29c937154e2b1e221aa3ea156    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 13:56:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 13:56:58 -0400    

Click here for diff

A change invalidated the current image which has been causing the d9 test to run longer.  

M test/container.yaml

Add zNewFmt().

commit   : 68a410779ad92a91f6e8bfe504a9acd7fecd06f2    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 12:32:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 12:32:49 -0400    

Click here for diff

This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.  

M src/Makefile.in
M src/build/config/render.c
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/control/start.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/verify/verify.c
M src/common/io/tls/common.c
M src/common/io/tls/server.c
M src/common/type/buffer.c
M src/common/type/string.c
M src/common/type/string.h
A src/common/type/stringZ.c
M src/common/type/stringZ.h
M src/storage/gcs/write.c
M test/define.yaml
M test/src/common/harnessPq.h
M test/src/common/harnessServer.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typePackTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/config/parseTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Clean up dividers in the documentation.

commit   : 475e7c692d7e54b9417de74cb566c2ccc7158997    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 12:11:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 12:11:04 -0400    

Click here for diff

Dividers were used in some files, but not others, and some had section names (which are hard to maintain) and others did not.  
  
Try to make this more consistent by putting a divider on front of every section, variable block, and wherever else seems appropriate.  

M doc/xml/coding.xml
M doc/xml/contributing.xml
M doc/xml/documentation.xml
M doc/xml/faq.xml
M doc/xml/index.xml
M doc/xml/metric.xml
M doc/xml/user-guide-index.xml
M doc/xml/user-guide.xml

Remove key dividers in help.xml.

commit   : 356bc27bf200762148afb7b495b5a54efeea0159    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 11:41:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 11:41:28 -0400    

Click here for diff

The idea was to make this file easier to browse and edit, but in fact it is much easier to just search for the command/option needed.  
  
The dividers were never applied consistently and at some point we decided to get rid of the comments because they were hard to keep updated. The result was a mix of styles which did nobody any favors.  

M src/build/help/help.xml

Add verify output and verbose options.

commit   : 65d22e43257d46ff6523ffbdb3af59e30deeaea5    
  
author   : Reid Thompson <[email protected]>    
date     : Fri, 6 May 2022 11:11:36 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 6 May 2022 11:11:36 -0400    

Click here for diff

These options allow the user to control how the verify results will be output to the console and log.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/verify/verify.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/define.yaml
M test/src/module/command/helpTest.c
M test/src/module/command/verifyTest.c

Backup file bundling documentation.

commit   : f405fc6ae281eebf4d61abb39963a9830fe2fa2f    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 10:21:20 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 6 May 2022 10:21:20 -0400    

Click here for diff

Make the feature user visible and add documentation to the user guide.

M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/config/config.yaml

Fix comment wrapping.

commit   : 4cc0d46d606382e6b00493e73ee401ecfe5b9f3e    
  
author   : Reid Thompson <[email protected]>    
date     : Fri, 6 May 2022 09:34:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 09:34:39 -0400    

Click here for diff

M src/command/verify/verify.c

Use uint8_t for optionResolveOrder.

commit   : e70c71049ec1ff3bf1f3e6a1987c82d9b95bce82    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 May 2022 07:49:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 May 2022 07:49:23 -0400    

Click here for diff

This saves a bit of space and should not affect processing speed.  
  
On MacOS (clang) this unexpectedly reduces the size of the binary by 16kiB but on Linux (gcc) there are no savings at all.  

M src/build/config/render.c
M src/config/parse.auto.c
M test/src/module/build/configTest.c

Replace strNewFmt() with TEST_ERROR_FMT() in command/archive-push module.

commit   : 808f7bf11c367a5be2130ea573c9447d8ea2dda8    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 May 2022 20:14:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 May 2022 20:14:13 -0400    

Click here for diff

This test was likely written before TEST_ERROR_FMT() existed.  

M test/src/module/command/archivePushTest.c

Use TEST_ERROR_FMT() rather than strNewFmt() in common/lock module.

commit   : efe0a39a75ed40f5568b1f5f566fee84d76fe094    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 May 2022 20:01:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 May 2022 20:01:02 -0400    

Click here for diff

These tests were likely written before TEST_ERROR_FMT() existed.  

M test/src/module/common/lockTest.c

Convert strNewFmt() to THROW_FMT() in config/parse module.

commit   : 5089a26633c965c4d9293d3de527a02b42d9b88d    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 May 2022 18:35:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 May 2022 18:35:00 -0400    

Click here for diff

It's not clear why strNewFmt() was used here, but there is no need for it.  

M src/config/parse.c

Remove COLON_STR and separator parameter from cfgParseCommandRoleName().

commit   : 876f3bbd1cd46af7f7d3dc2e9d454b77d51fff80    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 May 2022 18:15:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 May 2022 18:15:05 -0400    

Click here for diff

The separator parameter in cfgParseCommandRoleName() was useless since it was always set to : and COLON_STR did not provide any clarity its the single other usage.  

M src/common/type/string.c
M src/common/type/string.h
M src/config/config.c
M src/config/exec.c
M src/config/parse.c
M src/config/parse.h
M test/src/common/harnessConfig.c

Remove most _Z constants.

commit   : 7ae5478d9840d8b64a1211466e49d2835853ee55    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 May 2022 12:09:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 May 2022 12:09:21 -0400    

Click here for diff

Most of the time these were not making the code any clearer.  
  
For cases where they were used to construct Strings and Buffers, replace with constants.  
  
Also cleanup unused Buffers and Strings.  

M src/common/io/http/query.c
M src/common/io/http/request.c
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringZ.h
M src/protocol/helper.c

Remove extraneous linefeed when writing a lock file.

commit   : a6b1adb5fd533b4b9aca29646c8acf647d872959    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 May 2022 11:15:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 May 2022 11:15:14 -0400    

Click here for diff

Linefeeds are no longer part of the lock file format.  

M doc/xml/release.xml
M src/common/lock.c

Add ClockError for unexpected clock skew and timezone changes.

commit   : 5f8c9cd66a1b39a47acaddfbc3cf443ae8057519    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 May 2022 10:19:11 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 5 May 2022 10:19:11 -0400    

Click here for diff

A distinct result code should help debugging of clock skew and timezone issues.

M doc/xml/release.xml
M src/build/error/error.yaml
M src/command/backup/backup.c
M src/common/error.auto.c
M src/common/error.auto.h
M test/src/module/command/backupTest.c

Strip extensions from history manifest before showing in error message.

commit   : b6bfd9f99d74b1f5e631cf6462b68342589d2b80    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 May 2022 09:20:49 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 5 May 2022 09:20:49 -0400    

Click here for diff

In cases where clock skew or timezone issues are preventing backup label generation the user could see an error like this:
  

  
new backup label '20220504-152308F' is not later than latest backup label '20220504-222042F_20220504-222141I.manifest.gz'
  

  
This will happen if the most recent label is drawn from the history. It is cleaner (and probably less confusing) to strip off the extensions so the user sees:
  

  
new backup label '20220504-152308F' is not later than latest backup label '20220504-222042F_20220504-222141I'

M doc/xml/release.xml
M src/command/backup/backup.c
M src/info/manifest.h
M test/src/module/command/backupTest.c

Prevent memContextFree() from freeing memory needed by callbacks.

commit   : ef672c74adf871f17f6e14f5f0d6ab70e8fa3d52    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 May 2022 14:53:05 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 4 May 2022 14:53:05 -0400    

Click here for diff

The order of callbacks and frees meant that memory needed during a callback (for logging in all known cases) might end up being freed before a callback needed it.
  

  
Requiring callbacks and logging to check the validity of their allocations is pretty risky and it is not clear that all possible cases have been accounted for.
  

  
Instead recursively execute all the callbacks first and then come back and recursively free the context. This is safer and it removes the need to check if a context is freeing so a simple active flag (in debug builds) will do. The caller no longer needs this information at all so remove memContextFreeing() and objMemContextFreeing().

M doc/xml/release.xml
M src/common/io/tls/client.c
M src/common/io/tls/session.c
M src/common/memContext.c
M src/common/memContext.h
M src/common/type/object.h
M test/src/module/common/memContextTest.c
M test/src/module/common/typeObjectTest.c

Show backup percent complete in info output.

commit   : d9088b2e2b16a89d29b967187a547976f96061fc    
  
author   : Reid Thompson <[email protected]>    
date     : Wed, 4 May 2022 12:52:05 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 4 May 2022 12:52:05 -0400    

Click here for diff

In the JSON output the percent complete is storage as an integer of the percent complete * 100. So, before display it should be converted to double and divided by 100, or split using integer mod and div.
  

  
Note that percent complete will only be displayed on the host where the backup was executed. Remote hosts will show a backup/expire running with no percent complete.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/info/info.c
M src/common/lock.c
M src/common/lock.h
M src/main.c
M test/expect/mock-all-001.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/backupTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/lockTest.c

PostgreSQL 15 support.

commit   : 20782c88bc8d0842d547bd5f4fecdb292e69d1f7    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 May 2022 11:55:59 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 4 May 2022 11:55:59 -0400    

Click here for diff

PostgreSQL 15 drops support for exclusive backup and renames the start/stop backup commands.
  

  
This is based on the pgdg-testing repo since beta1 has not been released yet, but it seems unlikely that breaking changes will be made at this point. beta1 should be tagged just before our next release so we'll retest before the release.

M doc/xml/release.xml
M src/db/db.c
M src/info/manifest.c
M src/postgres/interface.c
A src/postgres/interface/v150.c
M src/postgres/interface/version.h
M src/postgres/interface/version.vendor.h
M src/postgres/version.h
M test/container.yaml
M test/define.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessPostgres.c
A test/src/common/harnessPostgres/harness150.c
M test/src/common/harnessPq.h
M test/src/module/db/dbTest.c

Add PRs to thread locking on Github actions.

commit   : 8e849ae85d6ad0ecbd1f34649be4a664712fd40c    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 May 2022 10:28:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 4 May 2022 10:28:39 -0400    

Click here for diff

It also makes sense to lock old PRs. They can be manually unlocked if they are needed for some reason.  
  
Also add output logging to make it easier to determine if thread locking is completing.  

M .github/workflows/lock-thread.yml

Move issue locking to Github actions.

commit   : 09b387fccddb658ae93acc6ec8dc490155c4fdea    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 May 2022 09:24:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 4 May 2022 09:24:35 -0400    

Click here for diff

The old plugin has been defunct for some time so there are currently a lot of unlocked issues.  
  
Running this once per week seems sufficient for now. Worst case it can be run manually if it gets behind.  

D .github/lock.yml
A .github/workflows/lock-thread.yml

Remove dependency on pg_database.datlastsysoid.

commit   : 692fe496bdb5fa6dcffeb9f85b6188ceb1df707a    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 May 2022 08:22:45 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 4 May 2022 08:22:45 -0400    

Click here for diff

This column has been removed in PostgreSQL 15. Rather than add a lot of special handling, it seems better just to update all versions to not depend on this column.
  

  
Add centralized functions to identify the type of database (i.e. system or user) by name and use FirstNormalObjectId when a name is not available.
  

  
The new query in the db module will still return the prior result for PostgreSQL <= 15, which will be stored in the manifest. This is important to preserve behavior when downgrading pgBackRest. There are no concerns here for PostgreSQL 15 since older versions of pgBackRest won't be able to restore backups for PostgreSQL 15 anyway.

M doc/xml/release.xml
M src/command/info/info.c
M src/command/restore/restore.c
M src/db/db.c
M src/info/manifest.h
M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/interface/static.vendor.h
M test/define.yaml
M test/src/common/harnessPq.h
M test/src/module/command/infoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/interfaceTest.c

Remove extra linefeed.

commit   : 302e0c09217a2c5c11145e84976b54677f7a544e    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 May 2022 16:53:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 May 2022 16:53:29 -0400    

Click here for diff

M src/postgres/interface/static.vendor.h

Fix error thrown from FINALLY() causing an infinite loop.

commit   : 9a271e925c6239fbd3577f9827c34940e7e43ae4    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 May 2022 14:34:05 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 3 May 2022 14:34:05 -0400    

Click here for diff

Any error thrown resets execution to the last setjmp(), which means that parts of the try block need to make sure they don't get run again. FINALLY() was not doing this so if it threw an error it would end up back in the FINALLY() block, where the error would likely be thrown again, causing an infinite loop.
  

  
Fix this by tracking the state of FINALLY() and only running it once. This requires cleaning the error stack like CATCH*() and clearing the error like TRY_END() depending on the order of execution.

M doc/xml/release.xml
M src/common/error.c
M src/common/error.h
M test/src/module/common/errorTest.c

Fix obsolete variable naming.

commit   : b89c568b5fefd1ea0e8319402bab0f4fba6bfa2e    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 May 2022 10:50:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 May 2022 10:50:48 -0400    

Click here for diff

M test/src/module/common/memContextTest.c

Error on all lock failures except another process holding the lock.

commit   : 962990869454b15db058f4037dbc0b1b611fc95e    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 May 2022 10:13:32 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 3 May 2022 10:13:32 -0400    

Click here for diff

The archive-get/archive-push commands would not error for, .e.g permissions errors, when attempting to get a lock before launching the async process. Since the async process was not launched there would be no error status file and the user would get a generic failure message. Also, there would be no async log.
  

  
Refactor lockAcquireFile() to throw an error when failOnNoLock = false unless the file is locked by another process. This seems to be the original intent of this parameter and there may have been a mistake when porting from Perl. In any case it looks wrong enough to be considered a bug.

M doc/xml/release.xml
M src/common/lock.c
M test/src/module/common/lockTest.c

Exclude mem context name from production builds.

commit   : eb435becb3e4d3dd8ea01b0e8899654a268d3867    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 May 2022 15:17:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 May 2022 15:17:34 -0400    

Click here for diff

The mem context name is used to produce clearer debug errors but it has no purpose in production builds.  
  
Also remove memContextName() and access the struct directly since the name is only used within the common/memContext module.  
  
Note that a few errors that were thrown in production builds (and required the name) are now only thrown in debug builds. In practice we have not seen these errors in production builds due to extensive coverage so it does not seem worth modifying the error to work without the context name.  
  
This saves some memory, which is worthwhile, but the goal is to refactor Strings and Variants to have their own mem contexts and this change will prevent them from using more memory than they are now, along with other changes that will be coming later.  

M src/common/memContext.c
M src/common/memContext.h
M test/src/module/common/memContextTest.c

Add user:group to lock permission error.

commit   : 0055fa40fee2636fed88699f52e9e67cd2803f71    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 May 2022 09:45:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 May 2022 09:45:57 -0400    

Click here for diff

This will help debug permissions errors when the lock file cannot be created.  

M doc/xml/release.xml
M src/common/lock.c
M test/src/module/common/lockTest.c

Add hint to check the log on archive-get/archive-push async error.

commit   : 03c71aa606bc0dc89303b3907e7fcbd83ace79d6    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 May 2022 08:49:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 May 2022 08:49:13 -0400    

Click here for diff

If this error is thrown rather than a specific error returned from the async process, it means the async process is unable to write the status files for some reason and the only way to get the error is out of the async log.  
  
This hint includes the exact async log path and name to make finding errors easier.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/config/load.c
M src/config/load.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c

Fix leaks in common/json, common/keyValue, and common/variantList.

commit   : 126fc99c77998cc65094d92e955347100aa42531    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 14:10:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 14:10:53 -0400    

Click here for diff

This doesn't solve the problem of the variant code making far too many copies, but it at least plugs the leaks.  
  
jsonReadVarRecurse() could leak KeyValue and VariantList.  
  
kvDup() leaked object allocations into the calling context.  
  
kvDefault() gets a more efficient return structure.  
  
kvGetList() leaked a Variant into the calling context.  
  
varLstNewStrLst() leaked object allocations into the calling context. Update varLstDup() to reflect changes made in varLstNewStrLst().  

M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/variantList.c

Fix leaked String in cfgParseSize().

commit   : 96166539cf7df010f9077cd624f24fd77e5a4feb    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 13:33:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 13:33:03 -0400    

Click here for diff

M src/config/common.c

Improvements to test harness memory debugging.

commit   : 4872a3f121e71f79a29dd63acad34b7c3537c775    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 12:33:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 12:33:39 -0400    

Click here for diff

Only set -DDEBUG_MEM for the modules currently being tested rather than globally.  
  
Also run tests in a temp mem context. Running in the top context can confuse memory accounting when a new context is created in the top context.  

M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/test.c

Fix leaks in common/io unit test.

commit   : 90f939b36f8963272cdfe8d0ed5a50606990ff5e    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 12:31:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 12:31:59 -0400    

Click here for diff

These leaks make it harder to detect leaks in the core code, so fix them.  

M test/src/module/common/ioTest.c

Fix leaked String and Variant in harnessPqScriptRun().

commit   : 8047e97e31fb133551ad69c7fd170a8a96d188d6    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 12:17:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 12:17:33 -0400    

Click here for diff

M test/src/common/harnessPq.c

Fix comment typo.

commit   : ceb303f9e2f47d9818919dfd0127e558f2d83495    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 11:46:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 11:46:55 -0400    

Click here for diff

M src/common/stackTrace.h

Fix leaks in the storage/s3 module.

commit   : c463993b4cc96a92c61af298bc95d88129068902    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 11:41:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 11:41:16 -0400    

Click here for diff

storageS3Helper() leaked a few Strings which ended up in a long-lived context.  
  
storageS3AuthAuto() and storageS3AuthWebId() were cleaned up by their callers but since they are not called often a temp mem context seems better.  
  
storageS3Request() leaked an HttpRequest.  
  
storageS3Info() leaked an HttpResponse.  
  
storageS3PathRemoveInternal() leaked a variety of objects. Fix by freeing some of them and adding a temp mem context.  
  
storageS3Remove() leaked an HttpResponse object.  
  
storageWriteS3Part() leaked an HttpResponse object.  

M src/storage/s3/helper.c
M src/storage/s3/storage.c
M src/storage/s3/write.c

Fix leaks in the storage/remote module.

commit   : 4750bc94dd8279b166233463ff32f2f51fb67e18    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 11:20:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 11:20:31 -0400    

Click here for diff

storageRemoteFilterGroup() leaked a number of objects. Use a temp mem context to prevent that.  
  
storageRemoteProtocolInfoListCallback() leaked a PackWrite.  
  
storageWriteRemoteFreeResource() leaked a PackWrite.  

M src/storage/remote/protocol.c
M src/storage/remote/write.c

Fix leaks in the storage/posix module.

commit   : c123a6af9f5aee4baea1d9ab202e03df6de319e3    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 11:19:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 11:19:01 -0400    

Click here for diff

storagePosixPathCreate() leaked a String.  
  
storagePosixPathRemoveCallback() leaked a String.  

M src/storage/posix/storage.c

Fix leaks in the storage/gcs module.

commit   : d89bc6f2d241ae864cb9dcae2f5c6fc3a0e655ee    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 10:53:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 10:53:11 -0400    

Click here for diff

storageGcsAuthToken() memory was being cleaned up by the calling context, but seems better to keep this tidy and add a temp mem context.  
  
storageGcsRequest() leaked an HttpRequest.  
  
storageGcsInfo() leaked a number of objects. Use a temp mem context to prevent that.  
  
storageGcsPathRemoveCallback() leaked an HttpResponse.  
  
storageGcsRemove() leaked an HttpReponse.  
  
storageWriteGcsVerify() leaked a number of objects. Use a temp mem context to prevent that.  
  
storageWriteGcsBlock) leaked an HttpReponse.  

M src/storage/gcs/storage.c
M src/storage/gcs/write.c

Reuse Strings in iniLoad().

commit   : 083c93eaa3d54d4594048b37542276fc6f6665fb    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 10:11:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 10:11:15 -0400    

Click here for diff

Reuse the section/key/value Strings by truncating them instead of creating a new one every time.  
  
Also add an error for empty sections. This function is only used for loading info files (not config files), which should never contain an empty section.  

M src/common/ini.c
M test/src/module/common/iniTest.c

Add cvtZSubNTo*() functions.

commit   : bc46d4e37b8c324d30543147eaeb8058184d3543    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 09:50:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 09:50:23 -0400    

Click here for diff

These functions allow conversion from substrings without needing to create a String or a temporary buffer.  
  
httpDateToTime() no longer requires a temp mem context. Also improve handling of month search to avoid an allocation.  
  
httpUriDecode() no longer requires a temp mem context.  
  
jsonReadStr() no longer requires a temp mem context.  
  
pgLsnFromWalSegment() no longer requires a temp mem context.  
  
pgVersionFromStr() no longer requires a temp mem context. Also do a bit of refactoring.  
  
storageGcsCvtTime() no longer leaks six Strings per call.  
  
storageS3CvtTime() no longer leaks six Strings per call.  

M src/command/expire/expire.c
M src/command/restore/restore.c
M src/common/io/http/common.c
M src/common/io/http/response.c
M src/common/type/convert.c
M src/common/type/convert.h
M src/common/type/json.c
M src/postgres/interface.c
M src/storage/gcs/storage.c
M src/storage/s3/storage.c
M test/src/module/command/backupTest.c
M test/src/module/common/typeConvertTest.c

Fix leak in varLstNewStrLst().

commit   : 6e18235be8b56eaedd940d7382e09e52ba22ffe9    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 06:52:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 06:52:07 -0400    

Click here for diff

The duplicated list elements were leaked into the calling context.  

M src/common/type/variantList.c

Remove unused header.

commit   : aeb7568a0ffeba72f254e46d2aa5d7bcca7c9a19    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 06:48:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Apr 2022 06:48:02 -0400    

Click here for diff

M src/common/type/list.c

Fix incorrect test param.

commit   : 5285b9aa7bdda8c160adde2b7d64e217fd2be123    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Apr 2022 20:18:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Apr 2022 20:18:18 -0400    

Click here for diff

M src/storage/gcs/storage.c

Use OBJ_NEW*() macros in SocketServer object.

commit   : eb65a5674dfa84d6b516490ef8796d44934bb0ef    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Apr 2022 11:40:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Apr 2022 11:40:29 -0400    

Click here for diff

This was missed in ccc255d3 when the TLS server was introduced, probably because work on that commit preceded when the macros were introduced in 475b57c8. It would have been easy to miss in a merge.  

M src/common/io/socket/server.c

Fix leaks in the storage/azure module.

commit   : 6f2654a5eb0e2cb63a4ff3af187f991817189dca    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 12:53:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 12:53:55 -0400    

Click here for diff

storageAzureHelper() leaked a few Strings which ended up in a long-lived context.  
  
storageAzureNew() failed to make a copy of the endpoint. This worked because storageAzureHelper() leaked the endpoint into the long-lived parent context.  
  
storageAzureRequest() leaked an HttpRequest.  
  
storageAzureInfo() leaked an HttpResponse.  
  
storageAzurePathRemoveCallback() leaked an HttpResponse.  
  
storageAzureRemove() leaked an HttpResponse.  

M src/storage/azure/helper.c
M src/storage/azure/storage.c
M src/storage/azure/write.c

Refactor and fix leak in storageRepoPathExpression().

commit   : be120c746c403b878b6bc7df83b4ffbf6e513f72    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 12:27:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 12:27:51 -0400    

Click here for diff

storageRepoPathExpression() could leak a StringList. Also refactor to remove unneeded assignments and create the String one time.  

M src/storage/helper.c

Refactor functions in postgres/interface module and fix leak.

commit   : cca6df872a7861de18f7e69e1df5183acd504681    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 12:09:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 12:09:44 -0400    

Click here for diff

pgLsnFromWalSegment() leaked two Strings.  
  
Refactor pgLsnRangeToWalSegmentList() to create the StringList in the calling context rather than moving it later.  

M src/postgres/interface.c

Fix leaks in protocol module.

commit   : a56fa0eb45871dff8b6362d9298290a52c7e6cd9    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:59:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:59:21 -0400    

Click here for diff

These leaks were not a big deal individually and there are generally few protocol objects created, but the leaks ended up in mem contexts that persist for most of execution. It makes sense to keep long-lived contexts as tidy as possible.  

M src/protocol/helper.c

Fix leaks in info module.

commit   : 78e912a9320770122808384ae202b89fa6236e89    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:20:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:20:51 -0400    

Click here for diff

*LoadFileCallback() all leaked between retries.  
  
infoPgArchiveId() leaked a String.  

M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c

Fix leaks in db module.

commit   : 36b0a9fa58f4f248fdabf170235f9f624d828d56    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:07:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:07:05 -0400    

Click here for diff

dbFreeResource() leaked ProtocolCommand.  
  
dbTimeMSec() leaked PackRead.  

M src/db/db.c

Fix leak in configOptionRemote().

commit   : 78b90e5ad80f87e0c1167701ac40504dc888d0dd    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:03:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:03:06 -0400    

Click here for diff

A Pack and String were leaked.  

M src/config/protocol.c

Fix spacing.

commit   : 9a6df398393c116c73dbbf459073271d84c45bec    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:00:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 11:00:20 -0400    

Click here for diff

M src/config/parse.c

Fix leak in cfgCommandJobRetry().

commit   : 8efb4e1e7f8d806f4688c54052f2325c21f470a9    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:58:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:58:12 -0400    

Click here for diff

A Variant was leaked for each retry record.  
  
Also remove an extra linefeed.  

M src/config/config.c

Fix leak in pckReadStrLst().

commit   : a8fed52ecb653229298682280b179fdf92438cdb    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:53:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:53:13 -0400    

Click here for diff

A String would be leaked on each loop iteration.  

M src/common/type/pack.c

Refactor httpRequestNew().

commit   : d7e45f12a54ea186e66824b9127478eaa1355d32    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:49:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:49:04 -0400    

Click here for diff

Move httpRequestProcess() outside of the object mem context. In case it ever returns a value we don't want that to end up in the object context.  

M src/common/io/http/request.c

Fix leak in httpHeaderPutRange().

commit   : cd1cf337ff8011f01657ab1baca31927d5d2343d    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:45:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:45:30 -0400    

Click here for diff

The range string was leaked.  

M src/common/io/http/header.c

Fix leak in ioFilterGroupResultAllSet().

commit   : 07e280199723eef35563ec71ff0c20687d642d9b    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:39:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:39:28 -0400    

Click here for diff

The PackRead containing the filter results was leaked.  

M src/common/io/filter/group.c

Add missing FUNCTION_TEST*() macros.

commit   : 3310decf8e4b09293e3f6b7de7e0212323ad0c81    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:36:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:36:27 -0400    

Click here for diff

M src/common/crypto/hash.c

Fix leak in cipherBlockNew().

commit   : 7eed9730aa8af91999e542394d05ef53c1cc7e36    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:34:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:34:10 -0400    

Click here for diff

The string used to look up the cipher type was leaked.  

M src/common/crypto/cipherBlock.c

Improved memory management in the common/exit module.

commit   : 6eed4125e6297bc6e80c9c11f839ba65979293ee    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:29:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:29:48 -0400    

Click here for diff

Refactor so that error detail is only logged in one place. This reduces calls to exitErrorDetail() and LOG_INTERNAL_FMT().  
  
Fix minor leaks in exitErrorDetail() and exitSafe().  

M src/common/exit.c

Mem allocation cleanup in command/verify module.

commit   : fa6c68cb029e675d9f4b3e4fa48862b6644ecc78    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:18:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:18:24 -0400    

Click here for diff

Move the temp mem context out of verifyJobCallback() into verifyBackup() and verifyArchive(). This makes it clearer that verifyJobCallback() allocates no memory and reduces mem usage when both verifyBackup() and verifyArchive() are called.  
  
Update verifyErrorMsg() to return zero-terminated strings to save on allocations. The output of this function is used when formatting strings so this is also simpler. Do a similar thing in verifyRender().  
  
Also fix a minor leak in verifyInfoFile().  

M src/command/verify/verify.c

Fix object allocations in incorrect mem context in execOpen().

commit   : 3f7c8bc9230acaf2dfb0a013303559b54a54c26f    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:15:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 10:15:47 -0400    

Click here for diff

Object variables were begin allocated in the calling context rather than the object context.  
  
This is not a live bug because Exec objects are currently created and opened in a long-lived context.  

M src/common/exec.c
M test/src/module/db/dbTest.c

Fix comment formatting.

commit   : 4dbe76a5bb6e5aae779b2538ac9ce54782f46b8f    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 26 Apr 2022 07:33:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Apr 2022 07:33:57 -0400    

Click here for diff

M src/command/backup/backup.c

Combine functions in the command/stanza module into one function.

commit   : 41f9d69edcf671fbda21cb6d87dbca08e48e9976    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 15:38:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 15:38:49 -0400    

Click here for diff

It is not clear why these were split out, but it probably had something to do with testing before storageList() could return NULL for an empty directory.  
  
Also remove the tests that depended on a boolean return, which are no longer needed for coverage.  

M src/command/stanza/delete.c
M test/src/module/command/stanzaTest.c

Add temporary mem contexts and fix a leak in the command/restore module.

commit   : 55a828f999f6fc8a975ea5c2ea9710e2109a5c0a    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 15:03:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 15:03:37 -0400    

Click here for diff

restoreRecoveryConf() and restoreRecoveryWriteConf() do enough work to deserve their own memory contexts.  
  
restoreFilePgPath() was leaking a String every time it was called, which could be a lot.  
  
Also fix a spacing issue.  

M src/command/restore/restore.c

Add temporary mem contexts in the command/help module.

commit   : 4e7414d48fb3cca45ca66b3eb853a08b6daf2833    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 14:49:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 14:49:08 -0400    

Click here for diff

These were not really leaks since memory was being freed by the calling function, but these functions do enough work to deserve their own memory contexts.  

M src/command/help/help.c

Add temporary mem context to removeExpiredBackup().

commit   : fd295f002b720dc1acc5d9523bd442f372394d15    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 14:19:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 14:19:10 -0400    

Click here for diff

This was not really a leak since memory was being freed by the calling function, but this function does enough work to deserve its own memory context.  
  
Also fixed a doubled semicolon.  

M src/command/expire/expire.c

Improve result declaration in backupRegExp().

commit   : 9314be36b161da8128cc4149372f41b89087e844    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 14:14:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 14:14:29 -0400    

Click here for diff

No need to set this to NULL when the initial value can be used instead.  

M src/command/backup/common.c

Fix memory leaks in archivePushDrop().

commit   : 5b5dbe0e7e4d3b3f412ea869487c3018c7a8d67c    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 13:29:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 13:29:43 -0400    

Click here for diff

A string was leaked for every file processed.  
  
Since the list can be quite large it also makes sense to reset the context occasionally.  

M src/command/archive/push/push.c

Fix formatting in common/string module.

commit   : c3b08f71ce03e2b4103f12e1d7c5c05701f02ef3    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 13:26:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 13:26:09 -0400    

Click here for diff

M src/common/type/string.c

Fix memory leaks in archivePushFile().

commit   : 40ef64f2be60fb947cf68aae2bb7d510ab36c8ec    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 12:59:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 12:59:46 -0400    

Click here for diff

The errorList is only used when throwing an error and the joined list is not needed after the error is thrown, so put both in the temp mem context.  

M src/command/archive/push/file.c

Remove useless context switches in archiveGetFile()/archivePushFile().

commit   : 774db650868c86c791f2791d02bee129a7669be5    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 12:55:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 12:55:51 -0400    

Click here for diff

These context switches do nothing since the list is already in the prior context.  

M src/command/archive/get/file.c
M src/command/archive/push/file.c

Add strLstAddSub*() and strLstAddSubZ*() functions.

commit   : 582c3dab4cff6883150c9c5de7f7b41d12982da9    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 12:32:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 12:32:33 -0400    

Click here for diff

These help with readability and remove a cause of leaks.  

M src/command/archive/push/push.c
M src/command/help/help.c
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/info/manifest.c
M test/src/module/common/typeStringTest.c

Use strLstAddZ() instead of strLstAdd() where possible.

commit   : ff45f463cf4d4521df5fd69b48bd553d862f0dc1    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 11:58:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 11:58:30 -0400    

Click here for diff

Using STRDEF() to convert the zero-terminated string to a String has no performance advantage but generates more code.  

M src/command/backup/backup.c
M src/command/restore/restore.c
M test/src/module/command/repoTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c

Add strLstNewFmt().

commit   : 7900660d3a12573db8a68bc7df46d7552dae54d1    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 11:47:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 11:47:43 -0400    

Click here for diff

Simplifies adding a formatted string to a list and removes a common cause of leaks.  

M src/command/archive/get/file.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/command.c
M src/command/restore/restore.c
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/config/exec.c
M src/config/parse.c
M src/postgres/interface.c
M src/protocol/helper.c
M test/src/common/harnessConfig.c
M test/src/common/harnessStorage.c
M test/src/module/command/repoTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/config/parseTest.c

Fix memory leak in archiveAsyncErrorClear().

commit   : 3475514b61fec52a5bd9e7fa73d38d5be612a9ad    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 10:57:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 10:57:36 -0400    

Click here for diff

A string was leaked on each call.  

M src/command/archive/common.c

Fix outdated comment.

commit   : 23d645b5e7f1069296f1a51165412049e1dccbdc    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 10:52:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 10:52:13 -0400    

Click here for diff

M src/db/protocol.c

Remove THIS_MEM_CONTEXT() macro.

commit   : 699f15dd2b227ad13ba313fa549c32938bd6cf2b    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 09:24:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 09:24:00 -0400    

Click here for diff

objMemContext(this) performs the same task and is easier to read.  
  
Most instances of this macro were removed by 6e7be3c0.  

M src/common/type/object.h
M src/storage/posix/read.c
M src/storage/posix/write.c
M src/storage/remote/write.c

Add MEM_CONTEXT_OBJ_*() macros.

commit   : 6e7be3c0525782cb58e0eb5046f1385e4eb8b1df    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 09:12:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 09:12:25 -0400    

Click here for diff

These provide a standard way to switch to an object mem context.  
  
Update the two different patterns that were used before to the new macros.  

M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/ini.c
M src/common/io/filter/group.c
M src/common/io/http/response.c
M src/common/type/buffer.c
M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/object.h
M src/common/type/pack.c
M src/protocol/command.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/storage/azure/read.c
M src/storage/azure/write.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/remote/read.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c

Improve JSON handling.

commit   : 45c3f4d53c7e5198d4d49109398326fecd282bec    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 09:06:26 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 25 Apr 2022 09:06:26 -0400    

Click here for diff

Previously read/writing JSON required parsing/render via a variant, which add many more memory allocations and loops.
  

  
Instead allow JSON to be read/written serially to improve performance and simplify the code. This also allows us to get rid of many String and Variant constant which are no longer required.
  

  
The goal is to be able to read/write very large (e.g. gigabyte manifest) JSON structures, which would not be practical with the current code.
  

  
Note that external JSON (GCS, S3, etc) is still handled using variants. Converting these will require more consideration about key ordering since it cannot be guaranteed as in our own formats.

M doc/xml/release.xml
M src/command/repo/create.c
M src/command/repo/ls.c
M src/common/ini.c
M src/common/ini.h
M src/common/stat.c
M src/common/type/json.c
M src/common/type/json.h
M src/info/info.c
M src/info/info.h
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/infoPg.h
M src/info/manifest.c
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/server.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/s3/storage.c
M test/define.yaml
M test/src/common/harnessInfo.c
M test/src/common/harnessInfo.h
M test/src/common/harnessServer.c
M test/src/module/command/backupTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/performance/typeTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/gcsTest.c

Add ASSERT_PARAM() macro.

commit   : 58f24568f5bf578e8ad0f5e4866bf2ede8b45d46    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 08:25:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Apr 2022 08:25:48 -0400    

Click here for diff

Allows adding a parameter to a function that is used only for assertions.  

M src/common/assert.h

Require type for FUNCTION_TEST_RETURN*() macros.

commit   : 1e2b545ba47d66760cf4c6df65c6325bb6404bb2    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Apr 2022 19:19:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Apr 2022 19:19:46 -0400    

Click here for diff

This allows code to run after the return type has been generated in the case where it is an expression.  
  
No new functionality here yet, but this will be used by a future commit that audits memory usage.  

M src/build/common/yaml.c
M src/command/archive/common.c
M src/command/archive/get/get.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/check/common.c
M src/command/command.c
M src/command/control/common.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/restore/restore.c
M src/command/stanza/common.c
M src/command/stanza/delete.c
M src/command/verify/verify.c
M src/common/compress/bz2/common.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/helper.c
M src/common/compress/lz4/common.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/compress/zst/common.c
M src/common/compress/zst/compress.c
M src/common/compress/zst/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/common.c
M src/common/debug.h
M src/common/encode.c
M src/common/exec.c
M src/common/exit.c
M src/common/ini.c
M src/common/io/fd.c
M src/common/io/fdRead.c
M src/common/io/fdWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/group.c
M src/common/io/http/common.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/http/response.c
M src/common/io/http/session.c
M src/common/io/http/url.c
M src/common/io/io.c
M src/common/io/read.c
M src/common/io/session.c
M src/common/io/socket/client.c
M src/common/io/socket/server.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/common.c
M src/common/io/tls/server.c
M src/common/io/tls/session.c
M src/common/lock.c
M src/common/log.c
M src/common/memContext.c
M src/common/regExp.c
M src/common/stat.c
M src/common/time.c
M src/common/type/buffer.c
M src/common/type/convert.c
M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/list.c
M src/common/type/pack.c
M src/common/type/string.c
M src/common/type/stringId.c
M src/common/type/stringList.c
M src/common/type/variant.c
M src/common/type/variantList.c
M src/common/type/xml.c
M src/common/user.c
M src/config/common.c
M src/config/config.c
M src/config/parse.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M src/postgres/client.c
M src/postgres/interface.c
M src/protocol/command.c
M src/protocol/helper.c
M src/storage/azure/read.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/helper.c
M src/storage/posix/read.c
M src/storage/posix/write.c
M src/storage/remote/read.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/storage.c
M test/src/module/protocol/protocolTest.c

Add SIZE_OF_STRUCT_MEMBER() macro.

commit   : 91bbe810592c8b57079230d0bf15720d74c705fa    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Apr 2022 09:14:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Apr 2022 09:14:12 -0400    

Click here for diff

Gets the size of a struct member, which requires a bit of additional syntax.  

M src/common/macro.h

Fix instances where STRDEF() was used instead of STR().

commit   : a2eee156b5a85237d9b9ba7896b237ec4f216da0    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 18:23:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 18:23:17 -0400    

Click here for diff

In practice this didn't cause problems because the string buffer was still valid and strSize() was not being called.  

M src/common/type/string.c
M test/src/common/harnessLog.c

Add STR_SIZE() macro.

commit   : 627921c72ab53a88519902c17450cedfa9dbda7a    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 08:35:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 08:35:31 -0400    

Click here for diff

This allows efficiently creating strings where the length is already known and simplifies the STR() and STRDEF() macros.  

M src/common/type/string.h

Use STRDEF() instead of STR() in command/help module.

commit   : 1ce613a24d4de1c219d2b7d844cf7e88649887fc    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 08:34:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 08:34:03 -0400    

Click here for diff

STRDEF() is more efficient since this is a constant string.  

M src/command/help/help.c

Add ASSERT_DECLARE() macro.

commit   : 4daddebaca6b4b1861778bc2b09d265302c05ed1    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 08:07:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 08:07:22 -0400    

Click here for diff

Declare variables that will be used by later assertions with the goal of making them easier to read and maintain.  
  
This is particularly useful for variables that are used more than once and require a lot of syntax to extract.  

M src/common/assert.h
M src/config/config.c

Allow *RETURN*() macros to accept struct initializers.

commit   : e18b70bf555de24ba38d7c51f9d0ffbe8a02a355    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 07:45:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Apr 2022 07:45:59 -0400    

Click here for diff

Struct initializers look like multiple parameters in a macro so use __VA_ARGS__ to reconstruct them.  

M src/common/debug.h
M test/src/common/harnessDebug.h

Fix ordering of backup-lsn-stop field in command/restore unit test.

commit   : ea4d73f375558c237fce696f20c19d1efef00bc1    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:56:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:56:26 -0400    

Click here for diff

All fields should be alphabetical. Currently the read code is tolerant of this, but that will not always be the case.  
  
Fields are always written alphabetically so this is just a test issue introduced by d8d41321.  

M test/src/module/command/restoreTest.c

Add JSON error when value does not parse in Ini object.

commit   : cb7a5f1ef3bbfe0a26e3cd718d9635280f2e9b83    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:49:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:49:23 -0400    

Click here for diff

If the JSON value fails to parse it is helpful to have the error message, at least for debugging.  

M src/common/ini.c
M test/src/module/common/iniTest.c

Handle missing archive start/stop in info/info backup unit test.

commit   : da6b4abc58674902196fe549586c16094dc37e38    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:41:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:41:28 -0400    

Click here for diff

This is not a very realistic case since archive start/stop are always written, but it appears in many other unit tests so it should also be tested here.  

M test/src/module/info/infoBackupTest.c

Add size to info/manifest unit test.

commit   : d897bf1ec2c74e46c56c31c6ba2ca910394ce0e5    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:36:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:36:33 -0400    

Click here for diff

This prevents the check from being order dependent.  

M test/src/module/info/manifestTest.c

Remove unnecessary mem contexts in the Manifest object.

commit   : d6f466be2bd1b68d45f109f0332783b784838ab7    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:31:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 19:31:35 -0400    

Click here for diff

The correct context is set by the various *Add() functions so these are not needed and cause leaks, though the leaks will only be noticeable in cases where there are a lot of page checksum errors.  

M src/info/manifest.c

Add FUNCTION_TEST_NO_RETURN() macro.

commit   : da9f2618528e9bd4b5538ff81cc359172a36af7b    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 14:09:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 14:09:49 -0400    

Click here for diff

This is required for the (currently) single place where a function with test FUNCTION_TEST*() macros does not return.  
  
This allows return to be added to the FUNCTION_TEST_RETURN_VOID() macro, which means return no longer needs to be added when returning from a function early.  

M src/command/backup/backup.c
M src/command/repo/ls.c
M src/command/restore/restore.c
M src/common/crypto/common.c
M src/common/debug.h
M src/info/manifest.c
M src/storage/storage.c

Refactor PgClient to return results in Pack format.

commit   : c304fafd456f67d11b0e9e7a8f399732716cf216    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Apr 2022 08:36:53 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 20 Apr 2022 08:36:53 -0400    

Click here for diff

Packs support stronger typing than JSON and are more efficient. For the small result sets that we deal with efficiency is probably not very important, but this removes another place where we are using JSON instead of Pack.
  

  
Push checking for result struct (e.g. single row) down into PgClient since it has easy access to this information rather than needing to parse the result set to find out.
  

  
Refactor all code downstream that depends on PgClient results.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/db/db.c
M src/db/db.h
M src/db/protocol.c
M src/info/manifest.c
M src/info/manifest.h
M src/postgres/client.c
M src/postgres/client.h
M test/src/module/db/dbTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/clientTest.c

Remove extra linefeed.

commit   : e699402f99f70819bd922eb6150fbe1b837eca0d    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Apr 2022 18:07:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Apr 2022 18:07:11 -0400    

Click here for diff

M src/build/config/render.c

Refactor remote storage protocol to use Packs instead of JSON.

commit   : b7fccaf9949f15ea1a00abd434e765c9c4d3e2f1    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Apr 2022 14:08:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Apr 2022 14:08:53 -0400    

Click here for diff

Packs are more efficient and strongly typed so they make more sense for the protocol.  

M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c

Use specific integer types in postgres/client and db unit tests.

commit   : cfd6c7ceb490d8c2766fd91a4719bdf9f6e4f110    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Apr 2022 12:14:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Apr 2022 12:14:22 -0400    

Click here for diff

This will work better once we are able to transmit the results with stronger typing.  
  
Also remove int2 which was not being used.  

M src/postgres/client.c
M test/src/common/harnessPq.h
M test/src/module/db/dbTest.c
M test/src/module/postgres/clientTest.c

Update postgres/client unit test to conform to current patterns.

commit   : 9751ddc4f80928de4b42bc56c54731afa4ff16aa    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Apr 2022 11:53:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Apr 2022 11:53:31 -0400    

Click here for diff

This includes adding test titles and using constants for query and error values that repeat.  

M test/src/module/postgres/clientTest.c

Update postgres/client unit test for changes in libpq.

commit   : bc5f6fac34fefd0595eee87f250dc9abef06d191    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Apr 2022 10:47:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Apr 2022 10:47:44 -0400    

Click here for diff

There have been some behavioral changes in libpq which require changes to the test.  
  
Also update the instructions since it is now a bit easier to run against a real cluster.  

M test/src/module/postgres/clientTest.c

Fix URL for apt.p.o archives.

commit   : 214ee9eb0e8398b1871a53441f4b926eb194d765    
  
author   : David Steele <[email protected]>    
date     : Sun, 17 Apr 2022 09:41:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 17 Apr 2022 09:41:22 -0400    

Click here for diff

A new archive repo was created in March of 2020: https://www.df7cb.de/blog/2020/apt-archive.postgresql.org.html  

M doc/xml/faq.xml

Return stats as a JSON string rather than a KeyValue object.

commit   : d103dd6238dffb5ddc3d5366c674278a67426f25    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 20:34:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 20:34:42 -0400    

Click here for diff

There is no need to process the stats so a KeyValue is overkill.  
  
Also remove the performance tests that check the stat totals since this is covered in the unit tests.  

M src/command/command.c
M src/common/stat.c
M src/common/stat.h
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/statTest.c
M test/src/module/performance/typeTest.c

Add test for protocol greeting when a field is missing.

commit   : e1ce731f8ae5d1aea009bc369fd157cddb44e059    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 19:37:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 19:37:03 -0400    

Click here for diff

A missing field and a NULL field are not exactly the same so it seems best to test both.  
  
Because of the way KeyValue objects work the error is the same, but that will not always be true.  

M test/src/module/protocol/protocolTest.c

Update comment in pack module to cover a more common use case.

commit   : 8a29d56f3cbf37166f3722ac7ffe18841e9e57f3    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 19:18:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 19:18:00 -0400    

Click here for diff

The KeyValue object is actively being removed so this is no longer the best example.  
  
Instead use an example that should outlive the KeyValue object.  

M src/common/type/pack.h

Fix reported error line number when ini key length is zero.

commit   : aeecd07ad87fafb34087f0df9d7c7413f749194c    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 18:29:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 18:29:54 -0400    

Click here for diff

The line number was one less than it should have been, which could cause some confusion.  
  
Since this only affected ini files with JSON values, which are always written programmatically, there is almost zero chance this has ever been a problem in the field.  

M src/common/ini.c
M test/src/module/common/iniTest.c

Update FreeBSD and MacOS images for Cirrus CI.

commit   : 47f8e11889ab8c05e1e548c07e6a338b69745c86    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 08:13:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Apr 2022 08:13:39 -0400    

Click here for diff

This fixes fe1ac210. Apparently FreeBSD 12.2 went EOL and the image was immediately broken.  
  
Also add FreeBSD 13.0 and update MacOS to Monterey.  

M .cirrus.yml

Add static keywords.

commit   : bcbac8a067e84a48e0849c5336f21044fec89df5    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Apr 2022 17:54:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 11 Apr 2022 17:54:18 -0400    

Click here for diff

This is more efficient than pushing these variables onto the stack.  

M src/common/type/stringId.c

Throw error when unable to read lock process.

commit   : fa40bcdc5cf9e6fc6f46236df258d3fa536ccdc7    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Apr 2022 14:08:16 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 11 Apr 2022 14:08:16 -0400    

Click here for diff

Previously the process id was skipped if it did not exist. Instead, throw an error and handle the errors in downstream code.
  

  
This was probably ignored at some point to provide backward-compatibility, but that is no longer required, if it ever was.

M doc/xml/release.xml
M src/command/control/stop.c
M src/common/lock.c
M test/src/module/command/controlTest.c

Disable FreeBSD builds on Cirrus CI.

commit   : fe1ac210bba9439d9a4baa4adbadd1413c61e2f5    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Apr 2022 13:17:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 11 Apr 2022 13:17:54 -0400    

Click here for diff

This build has started breaking with the following error:  
  
cd .. && perl ${CIRRUS_WORKING_DIR}/test/test.pl --no-gen --make-cmd=gmake --vm=none --vm-max=2 --no-coverage --no-valgrind --module=command --test=backup  
2022-04-11 17:11:53.034 P00   INFO: test begin on amd64 - log level info  
2022-04-11 17:11:53.107 P00   INFO: configure build  
ld-elf.so.1: /usr/local/lib/perl5/5.32/mach/CORE/libperl.so.5.32: Undefined symbol "strerror_l@FBSD_1.6"  
  
Disable the build to unstick the pipeline until this can be fixed.  

M .cirrus.yml

Fix whitespace.

commit   : 15021a0e976c1a8eadebf642dfff293df0fed54a    
  
author   : David Steele <[email protected]>    
date     : Sat, 9 Apr 2022 18:29:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 9 Apr 2022 18:29:57 -0400    

Click here for diff

M src/command/repo/create.c

Add lockRead*() functions for reading locks from another process.

commit   : 79b204166394295e40e7ceddc359368c5770db26    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Apr 2022 15:55:41 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 8 Apr 2022 15:55:41 -0400    

Click here for diff

Sometimes we need to read a lock from another process. This was done two different ways and in the case of cmdStop() was definitely hacky.
  

  
Centralize the logic to make it easier to read the locks for another process. This will also make it easier to add new lock data.

M doc/xml/release.xml
M src/command/control/stop.c
M src/command/info/info.c
M src/common/lock.c
M src/common/lock.h
M test/define.yaml
M test/src/module/common/lockTest.c

Suppress existing WAL warning when archive-mode-check is disabled.

commit   : aad7171940afdb221a03825bee6eb2f1827289f1    
  
author   : Reid Thompson <[email protected]>    
date     : Fri, 8 Apr 2022 15:00:20 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 8 Apr 2022 15:00:20 -0400    

Click here for diff

When archive-mode-check is disabled and archive-push is running from multiple hosts, it is very likely that the file will already exist with the same checksum, so disable the warning.
  

  
However, if the checksums do not match, an error will still be thrown.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/command/archive/push/file.c
M src/command/archive/push/file.h
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M src/config/parse.auto.c
M test/src/module/command/archivePushTest.c

Handle NULL path in TEST_STORAGE_LIST when remove is specified.

commit   : 4f543a4d67e0c18c8eccdd864a32003d45df7426    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Apr 2022 11:07:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Apr 2022 11:07:26 -0400    

Click here for diff

Using the path variable directly resulted in a path with (null) in it, which caused the remove to fail.  
  
The pathFull variable already exists for this purpose so use it.  

M test/src/common/harnessStorage.c

Add LENGTH_OF() macro.

commit   : 571dceefec5c64da2b2d79dd7e471fe0262b916a    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Apr 2022 19:00:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 Apr 2022 19:00:15 -0400    

Click here for diff

Determining the length of arrays that could be calculated at compile time was a bit piecemeal, with special macros used sometimes and with the math done directly other times.  
  
This macro makes the task easier, uses less space, and automatically adjusts when the type changes.  

M src/build/config/render.c
M src/command/local/local.c
M src/command/remote/remote.c
M src/common/compress/helper.c
M src/common/error.c
M src/common/io/http/common.c
M src/common/macro.h
M src/common/stackTrace.c
M src/common/type/convert.c
M src/common/type/pack.c
M src/common/type/variant.c
M src/main.c
M src/postgres/interface.c
M src/protocol/client.c
M src/protocol/server.h
M test/src/common/harnessPostgres.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/errorTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c
M test/src/test.c

Replace strCatFmt() with strCat()/strCatZ() where appropriate.

commit   : 8be11d32e43bc889c4889b28f88fe20a0c4cfd60    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Apr 2022 11:44:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 Apr 2022 11:44:45 -0400    

Click here for diff

Most of these looked like copy/paste from a prior required strCatFmt() call.  
  
There is no issue here since strCatFmt() works the same in these cases, but using strCat()/strCatZ() is more efficient.  

M src/build/config/render.c
M src/build/help/render.c
M src/command/command.c
M src/command/info/info.c
M src/command/restore/restore.c
M src/config/load.c
M src/db/db.c
M test/src/module/storage/s3Test.c

Add default for boolean options with unresolved dependencies.

commit   : cff147a7d257e270df8fee184fdc3684347be330    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Apr 2022 14:45:51 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 6 Apr 2022 14:45:51 -0400    

Click here for diff

If a boolean option had an unresolved dependency then the value would be NULL, which meant the dependency would need to be checked in the code to avoid an error. For example, cfgOptionBool(cfgOptOnline) needed to be checked before it was safe to call cfgOptionBool(cfgOptArchiveCheck).
  

  
Allow a default for boolean options when they are unresolved to simplify the code. This makes using the options easier and less prone to error. Not all boolean options get a dependency default in this commit, but more may be added in the future.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/config/parse.c
M src/build/config/parse.h
M src/build/config/render.c
M src/command/backup/backup.c
M src/config/parse.auto.c
M src/config/parse.c
M test/src/module/build/configTest.c
M test/src/module/config/parseTest.c

Move cfgParseOptionalFilterDepend() and add comment block.

commit   : acc9f3b72a481f64f5362d862959f300eb2aa63b    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Apr 2022 10:04:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 Apr 2022 10:04:08 -0400    

Click here for diff

M src/config/parse.c

Set option-archive-copy flag in backup.manifest to false when offline.

commit   : 5dba0d6e9b9129c3cb7f3a4bc46da57544f36e73    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Apr 2022 18:42:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Apr 2022 18:42:19 -0400    

Click here for diff

In offline mode the pg_wal directory is copied, but that is not the same as archive-copy, which copies the exact set of WAL required from the archive.  
  
This flag is purely for informational purposes so there is no live bug here, but the prior behavior was certainly misleading.  

M src/command/backup/backup.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Show Docker output when building containers if --log-level=detail.

commit   : 54b4187527287bf3dd8ae158329f0feb44eebed8    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Apr 2022 13:14:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Apr 2022 13:14:42 -0400    

Click here for diff

This helps with debugging and monitoring container builds.  

M test/lib/pgBackRestTest/Common/ContainerTest.pm

Auto-select backup for restore command --type=lsn.

commit   : d8d4132118edd7f93a5b221478c85c2d213c39dd    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 5 Apr 2022 11:59:12 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 5 Apr 2022 11:59:12 -0400    

Click here for diff

For PITR with --type=lsn, attempt to auto-select the appropriate backup set based on the --target LSN provided. Pick the most recent backup where backup-lsn-stop is less than or equal to the provided LSN.

M doc/xml/release.xml
M src/build/help/help.xml
M src/command/restore/restore.c
M test/src/module/command/helpTest.c
M test/src/module/command/restoreTest.c

Refactor target type checking for clarity in restore module.

commit   : 08d9e269c6f03e7797584e90c53eb0743b787f60    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Apr 2022 09:14:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Apr 2022 09:14:56 -0400    

Click here for diff

This avoids using targetTime != 0 as an indicator that a time target was selected.  

M src/command/restore/restore.c

Fix typo in db-timeout help.

commit   : f5fdab2989140630a62a4ef0d4c7fc35d789f967    
  
author   : Stefan Fercot <[email protected]>    
date     : Thu, 31 Mar 2022 16:18:21 +0200    
  
committer: Stefan Fercot <[email protected]>    
date     : Thu, 31 Mar 2022 16:18:21 +0200    

Click here for diff

M src/build/help/help.xml

Fix tls-server-auth example and add clarifications.

commit   : c222ce1a5f6f813188b7e87a3838d6dce656e0ff    
  
author   : David Steele <[email protected]>    
date     : Fri, 25 Mar 2022 08:52:41 -0600    
  
committer: David Steele <[email protected]>    
date     : Fri, 25 Mar 2022 08:52:41 -0600    

Click here for diff

M doc/xml/release.xml
M src/build/help/help.xml

Fix comment typo.

commit   : d4954fcbf49790475d26c90e951ac76660cef1e9    
  
author   : David Steele <[email protected]>    
date     : Fri, 25 Mar 2022 07:52:16 -0600    
  
committer: David Steele <[email protected]>    
date     : Fri, 25 Mar 2022 07:52:16 -0600    

Click here for diff

M src/protocol/helper.c

Cleanup output to stderr in unit tests.

commit   : f60ec5055a633308422797902e9487085c862850    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 18:43:43 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 18:43:43 -0600    

Click here for diff

The unit tests were ignoring stderr but nothing being output there was important. Now a test will fail if there is anything on stderr.  
  
This makes it easier to work with -fsanitize, which outputs to stderr.  

M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessTest.c
M test/src/module/command/serverTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/errorTest.c
M test/src/module/config/loadTest.c

Align checksum page error list in manifest file packs.

commit   : 50ee4b19fec0b6bc888e4ff6603166365ce88233    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 17:55:38 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 17:55:38 -0600    

Click here for diff

This was left unaligned on purpose to save space but the sanitizer did not like it. Since this field is seldom used go ahead and align it to make the sanitizer happy.  
  
Also add some macros to make working with alignment easier.  
  
Found with -fsanitize=undefined.  

M src/common/macro.h
M src/info/manifest.c

Fix incorrect reference to stanza-update in the user guide.

commit   : 3dd796045160c4d4dbf70ec2800ad483867629f1    
  
author   : Abubakar Mohammed <[email protected]>    
date     : Thu, 24 Mar 2022 15:59:41 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 15:59:41 -0600    

Click here for diff

This should be stanza-upgrade. Also fix in the git history cache since the comment was copied from the user guide.  

M doc/resource/git-history.cache
M doc/xml/release.xml
M doc/xml/user-guide.xml

Check that sha1 checksum is not empty in manifestFileUpdate().

commit   : 14016a86e7d37030ae934a004cad70888b539b33    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 13:13:35 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 13:13:35 -0600    

Click here for diff

The manifest test module was setting a blank value here and causing a stack overflow because memcpy() is used instead of strcpy().  
  
This was really just a test issue but add an assert just in case the same were to happen in production code.  
  
Also update a bogus checksum in the integration tests to the correct length to avoid running afoul of the assert.  
  
Found with -fsanitize=address.  

M src/info/manifest.c
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/info/manifestTest.c

Fix incorrect struct type in list initialization.

commit   : 4e5ac11517022f445f4ec078cc30bd2ce3aa6d15    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 12:56:26 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 12:56:26 -0600    

Click here for diff

This looks like a copy-paste error.  
  
The code is only run during development so this is not a live issue.  
  
Found with -fsanitize=address.  

M src/build/config/parse.c

Use strNewZ() in cases where STRDEF() assignment goes out of scope.

commit   : 75b26319aea4493d8b9624a61795f8128d50c3b5    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 12:26:09 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 12:26:09 -0600    

Click here for diff

If a variable assigned with STRDEF() is referenced out of scope of the STRDEF() assignment then the value is undefined.  
  
Luckily most of the instances are in tests but there is one in the core code. It is not clear if this is a live bug or not but it certainly needs to be fixed.  
  
Found with -fsanitize=address.  

M src/command/backup/backup.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/info/manifestTest.c
M test/src/module/protocol/protocolTest.c

Prevent signed integer overflow in cfgParseSize().

commit   : edf6c70baa97ad9c9045c01df4f587880a78cdc6    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 11:00:51 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 11:00:51 -0600    

Click here for diff

If the value and multiplier were large enough then the return value could overflow unpredictably.  
  
Check the value to make sure it will not overflow with the current multiplier.  
  
It would be better to present an "out of range" error to the user rather than "is not valid" but it doesn't seem worth the effort since the error is extremely unlikely.  
  
Found with -fsanitize=undefined.  

M src/config/common.c
M test/src/module/config/parseTest.c

Do not pass NULL to memcpy() in Buffer/String objects.

commit   : ccbe2a1f70b3ec0a328c7f5d0f5053ef3de1b543    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 09:32:18 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 09:32:18 -0600    

Click here for diff

glibc and others seem tolerant of this but the behavior is undefined.  
  
Found with -fsanitize=undefined.  

M src/common/type/buffer.c
M src/common/type/string.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeStringTest.c

Do not pass NULL to bsearch()/qsort() in List object.

commit   : 98792b1b0cd084ff90343e65b64526dfede396e7    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 09:22:05 -0600    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Mar 2022 09:22:05 -0600    

Click here for diff

glibc and others seem tolerant of this but the behavior is undefined.  
  
Found with -fsanitize=undefined.  

M src/common/type/list.c
M test/src/module/common/typeListTest.c

Use bufUsed() instead of struct member in Buffer object.

commit   : 333ef84606cb1597e09a324c81759a2a222e3bf4    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 Mar 2022 21:47:56 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 23 Mar 2022 21:47:56 -0600    

Click here for diff

bufUsed() is an inline function so it is just as efficient and should also be easier to read and maintain.  

M src/common/type/buffer.c

Allow files that become zero-length after the backup manifest is built.

commit   : 424008d293661dfb1a167cfec2697def48057447    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 Mar 2022 10:41:36 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 23 Mar 2022 10:41:36 -0600    

Click here for diff

It is possible that a file will be be truncated to zero-length after the backup manifest has been built. We could build logic into backupFile() to handle this case but it is hard to test well because of the race condition so tests would need to written directly against backupFile() and backupJobResult(). It hardly seems worth all that effort for a condition that occurs rarely, if ever.  
  
Instead just remove the manifest check and add tests to restore to make sure it handles bundled zero-length files correctly. Logging will show that the file was bundled so if it happens a lot (which seems very unlikely) then we can think about an alternate implementation.  

M doc/xml/release.xml
M src/info/manifest.c
M test/src/module/command/restoreTest.c

commit   : fe9fd2ff2d778b86c0df159aabe7644f18f6479f    
  
author   : David Steele <[email protected]>    
date     : Tue, 22 Mar 2022 09:02:33 -0600    
  
committer: David Steele <[email protected]>    
date     : Tue, 22 Mar 2022 09:02:33 -0600    

Click here for diff

Hardlinking will not work with bundles because files are not stored individually.  

M doc/xml/release.xml
M src/build/config/config.yaml
M src/command/backup/backup.c
M src/config/parse.auto.c

commit   : 7afaac0a3d0e43e5598d1abf548559dbe6329363    
  
author   : David Steele <[email protected]>    
date     : Tue, 22 Mar 2022 08:35:34 -0600    
  
committer: GitHub <[email protected]>    
date     : Tue, 22 Mar 2022 08:35:34 -0600    

Click here for diff

This rule was added because there were not sufficient tests to demonstrate that the repo-hardlink option could be changed in a backup set.
  

  
Remove the restriction and add/update tests to show that it works.
  

  
This is necessary now because bundling requires that hardlinking be disabled. Rather than add code complexity, it seems better just to address this limitation.

M doc/xml/release.xml
M src/command/backup/backup.c
M test/expect/mock-all-001.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/command/backupTest.c

Improve path validation for repo-* commands.

commit   : 5ae84d5e474e95dcc00c174aa5111815a5a70ba5    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 22 Mar 2022 09:50:26 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 22 Mar 2022 09:50:26 -0400    

Click here for diff

Check for invalid path in repo-* commands. Perform path validation and throw an error when appropriate. Path may not contain '//'. Strip trailing '/' from path. Absolute path must fall under repo path.

M doc/xml/release.xml
M src/Makefile.in
A src/command/repo/common.c
A src/command/repo/common.h
M src/command/repo/get.c
M src/command/repo/ls.c
M src/command/repo/put.c
M src/command/repo/rm.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/module/command/repoTest.c

Add AWS IMDSv2 support.

commit   : 21cef09dfd03c313531dc66ba9556164ae2835ab    
  
author   : nunopi <[email protected]>    
date     : Wed, 16 Mar 2022 18:02:29 +0100    
  
committer: GitHub <[email protected]>    
date     : Wed, 16 Mar 2022 18:02:29 +0100    

Click here for diff

IMDSv2 provides additional security to prevent instance metadata from being read by an attacker.
  

  
All AWS instances should provide IMDSv2 but still fail back to IMDSv1 if the IMDSv2 token request fails. This is in case there are any services outside AWS that are emulating IMDSv1 but have not implemented IMDSv2.

M doc/xml/release.xml
M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Remove extraneous double spaces in code and comments.

commit   : 2c96327e654ab634ee4077a39d6d20a7eae1065f    
  
author   : David Steele <[email protected]>    
date     : Tue, 15 Mar 2022 17:55:48 -0600    
  
committer: David Steele <[email protected]>    
date     : Tue, 15 Mar 2022 17:55:48 -0600    

Click here for diff

M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/lib/pgBackRestDoc/Common/Log.pm
M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
M src/command/archive/get/get.c
M src/command/backup/backup.c
M src/command/info/info.c
M src/command/restore/file.c
M src/command/restore/restore.c
M src/command/server/server.c
M src/common/type/pack.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/config/parse.c
M src/protocol/helper.c
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/Io/Process.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Env/ArchiveInfo.pm
M test/lib/pgBackRestTest/Env/BackupInfo.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/test.pl

Rename bundle-* options to repo-bundle-*.

commit   : 3f66f42ef9605a83c356e4c6eb3ef47bcb12c30d    
  
author   : David Steele <[email protected]>    
date     : Mon, 14 Mar 2022 17:49:52 -0600    
  
committer: GitHub <[email protected]>    
date     : Mon, 14 Mar 2022 17:49:52 -0600    

Click here for diff

It seems best for these to be repo options so they can be configured per repo, rather than globally.
  

  
All clarify usage for repo-bundle-size and repo-bundle-limit.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/backup/backup.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/module/command/backupTest.c

Improve error message for invalid repo-azure-key.

commit   : 7c9208ba856a71833dc7d8f1cf8cb2b4bb4b7414    
  
author   : Reid Thompson <[email protected]>    
date     : Fri, 11 Mar 2022 11:10:02 -0500    
  
committer: GitHub <[email protected]>    
date     : Fri, 11 Mar 2022 11:10:02 -0500    

Click here for diff

Check that repo-azure-key is valid base64 when repo-azure-key-type = shared.

M doc/xml/release.xml
M src/storage/azure/helper.c
M test/src/module/storage/azureTest.c

Add bundle logging to backup command.

commit   : 0054677147121967889ead7feab0959fb19e39a6    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Mar 2022 15:34:15 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Mar 2022 15:34:15 -0600    

Click here for diff

This was added to the restore command so add it to the backup command as well.  

M doc/xml/release.xml
M src/command/backup/backup.c
M test/src/module/command/backupTest.c

Optimize restore command for file bundling.

commit   : dca6da86bfe84a0cda004df1140d125db1630c90    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Mar 2022 15:03:28 -0600    
  
committer: GitHub <[email protected]>    
date     : Wed, 9 Mar 2022 15:03:28 -0600    

Click here for diff

Since files are stored sequentially in a bundle, it is often possible to restore multiple files with a single read. Previously, each restored file required a separate read. Reducing the number of reads is particularly beneficial for object stores, but performance should benefit on any file system.
  

  
Currently if there is a gap then a new read is required. In the future we might set a limit for how large a gap we'll skip without starting a new read.

M doc/xml/release.xml
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/command/restore/restore.c
M src/info/manifest.c
M test/src/module/command/restoreTest.c

Improve stop command to honor stanza option.

commit   : f7ab002aa74dd2ca277123f7500018ae68a04c74    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 8 Mar 2022 13:18:23 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 8 Mar 2022 13:18:23 -0500    

Click here for diff

Improve the stop command, when force and stanza options are specified, to terminate only processes holding lock files for the given stanza. Prior to these changes, termination of all processes holding lock files regardless of stanza occurred.

M doc/xml/release.xml
M src/command/control/stop.c
M src/common/lock.c
M src/common/lock.h
M test/define.yaml
M test/src/module/command/controlTest.c

Add limit parameter to ioCopyP().

commit   : 514137040e60e8fff0e7f42f43b493113abae5ff    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Mar 2022 08:23:31 -0600    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Mar 2022 08:23:31 -0600    

Click here for diff

Allows the number of bytes copied to be limited.  

M doc/xml/release.xml
M src/command/backup/file.c
M src/command/repo/get.c
M src/command/repo/put.c
M src/common/io/io.c
M src/common/io/io.h
M src/common/lock.c
M src/storage/storage.c
M test/src/module/common/ioTest.c
M test/src/module/performance/storageTest.c

Fix example for repo-gcs-key-type option in configuration reference.

commit   : 166039c0da1875581a08d41ab8a44adf95fa6db1    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Mar 2022 08:09:58 -0600    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Mar 2022 08:09:58 -0600    

Click here for diff

This looks like a copy-paste error from another option.  

M doc/xml/release.xml
M src/build/help/help.xml

Fix comment typo.

commit   : 91b718bd4f89157c7401e3904bb171387b2e8a3e    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 14:58:43 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 14:58:43 -0600    

Click here for diff

M src/command/backup/file.c

Increase precision of percent complete logging for backup and restore.

commit   : 330e19900ee3415d920680cf6e6856bf5e8e9a69    
  
author   : Reid Thompson <[email protected]>    
date     : Sun, 6 Mar 2022 14:01:24 -0500    
  
committer: GitHub <[email protected]>    
date     : Sun, 6 Mar 2022 14:01:24 -0500    

Click here for diff

For very large backups only getting an update per percent may not be often enough.
  

  
Add hundredths to the percent complete logging to provide more timely information.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/restore/restore.c
M test/expect/mock-all-001.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c

Fix incorrect comment.

commit   : be731f825432c738571f4c055f1b9d42d3fa18cf    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 12:03:57 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 12:03:57 -0600    

Click here for diff

This appears to have been blindly copied from the archive-get command.  

M src/command/backup/file.h

Replace percentage and size with a constant in restore test logs.

commit   : 8f23b46b4be9a0ce55e0c2d13be0edfd5bdc5349    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 11:57:20 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 11:57:20 -0600    

Click here for diff

Checking percentage and size in every test can cause quite a bit of churn when changes are made.  
  
Follow the example of the backup tests and replace percentage and size after the few tests to reduce churn.  

M test/src/module/command/restoreTest.c

Remove redundant restoreFile() test and improve coverage.

commit   : 4d2fef1c37d357965f0ffb9af0d7489f3be1371b    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 11:48:22 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 11:48:22 -0600    

Click here for diff

These tests were written before the restore command was fully migrated to C so many of them have become redundant.  
  
In the cases were they still provide coverage, add tests to synthetic restores to replace them. In general, these higher level tests provide better coverage than poking at the restoreFile() function directly.  

M test/src/module/command/restoreTest.c

Update config.guess and config.sub to latest versions.

commit   : 009d33aa8ebbcd00a85392e1424e8f078952e2f5    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 11:17:01 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 11:17:01 -0600    

Click here for diff

M src/build/config.guess
M src/build/config.sub

Begin v2.39 development.

commit   : c242b966e0352505ee8389fcebcfbde48b2a55d4    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 11:14:01 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 11:14:01 -0600    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h