pgBackRest 2.35 commit log

v2.34: PostgreSQL 14 Support

commit   : d10a99d73b772f221807024abbf61fdc5ca3a800    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 06:51:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 06:51:08 -0400    

Click here for diff

Bug Fixes:  
  
* Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.)  
* Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.)  
* Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete.)  
* Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.)  
  
Features:  
  
* Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.)  
* Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.)  
* Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)  
* Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)  
  
Improvements:  
  
* Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.)  
* Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.)  
* Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.)  
  
Documentation Improvements:  
  
* Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.)  
* Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.)  
* Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)  

M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Add HRN_STORAGE_PATH_REMOVE() macro.

commit   : c9a8ff27f2468dd769bfe8829b16e68d64357efb    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 2 Jun 2021 11:46:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Jun 2021 11:46:54 -0400    

Click here for diff

Update command/archive-push test to show how it is used.  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/archivePushTest.c

Add storage/remote test for unknown user/group name.

commit   : a97fc6e708b67fcf32aec85d312fd3ff5e7d93aa    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Jun 2021 17:10:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Jun 2021 17:10:15 -0400    

Click here for diff

M test/src/module/storage/remoteTest.c

Add missing parameter log debug.

commit   : d35315aaf99be7492b7712cda7922f54ccdd924f    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Jun 2021 15:33:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Jun 2021 15:33:07 -0400    

Click here for diff

M src/common/io/fdWrite.c

Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro.

commit   : 8250990afb06e0bfb74d2f9ece19338127060b94    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Jun 2021 09:03:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Jun 2021 09:03:44 -0400    

Click here for diff

HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases.  
  
It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/common/harnessProtocol.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/exitTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Add HRN_STORAGE_PATH_CREATE() macro.

commit   : c1277677a3e595a169f3bec7e1da31e8431f2f71    
  
author   : David Steele <[email protected]>    
date     : Fri, 28 May 2021 15:23:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 May 2021 15:23:18 -0400    

Click here for diff

Update command/archive-get test to show how it is used.  
  
Also move one path create that was much earlier than it needed to be.  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/archiveGetTest.c

Improve storage harness/test macros.

commit   : c8427682eafd8d38403818464711d63086dafc35    
  
author   : David Steele <[email protected]>    
date     : Fri, 28 May 2021 14:39:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 May 2021 14:39:43 -0400    

Click here for diff

Make the macros more consistent in format and make sure that each macro outputs a line number before doing any work so when errors happen it is clear where they happened.  
  
Add noRecurse option to TEST_STORAGE_LIST().  
  
Add comment option to all storage macros.  

M test/src/common/harnessInfo.h
M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/common/harnessTest.c
M test/src/common/harnessTest.intern.h

Use L to denote line numbers in test output.

commit   : 0a43be018347292a2d5e7ec68bbde595604bd104    
  
author   : David Steele <[email protected]>    
date     : Fri, 28 May 2021 13:08:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 May 2021 13:08:41 -0400    

Click here for diff

The lower-case l was too easy to confuse with a 1.  

M test/src/common/harnessTest.c

Remove line parameter from hrnTestResultBegin().

commit   : cfe8e5191039b63915f459f9f2cead7aab6b594d    
  
author   : David Steele <[email protected]>    
date     : Fri, 28 May 2021 08:50:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 May 2021 08:50:28 -0400    

Click here for diff

Instead store the line number in hrnTestLogPrefix() so it doesn't need to be passed to hrnTestResultBegin().  
  
Also add missing linefeed in hrnStorageList().  

M test/src/common/harnessStorage.c
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h

Remove unused padding parameter from hrnTestLogPrefix().

commit   : b3ac9e8010e225542fe4cb584e42a53adcba9d22    
  
author   : David Steele <[email protected]>    
date     : Fri, 28 May 2021 08:03:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 May 2021 08:03:22 -0400    

Click here for diff

Also add fflush() so log prefix is displayed immediately.  

M test/src/common/harnessConfig.c
M test/src/common/harnessStorage.c
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h

Replace storageTest with storagePg*() in command/archive-get test.

commit   : 52217f12445dd05cb828ed2f8cd19be2ad8759fd    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 May 2021 16:50:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 May 2021 16:50:18 -0400    

Click here for diff

All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted.  
  
Also remove some headers which are no longer needed.  

M test/src/module/command/archiveGetTest.c

Change archive expiration logging from detail to info level.

commit   : aceb956815a87f0c19ea171de25d2c3842a6d806    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 27 May 2021 08:03:44 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 27 May 2021 08:03:44 -0400    

Click here for diff

To assist with issue debugging, removal of files during archive expiration has been changed from detail level logging to info level.

M doc/xml/release.xml
M src/command/expire/expire.c
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/src/module/command/expireTest.c

Add repo-retention-history option to expire backup history.

commit   : 5b332b22b508d896abb64fa6abe26c78c83a29bc    
  
author   : Stefan Fercot <[email protected]>    
date     : Wed, 26 May 2021 19:07:18 +0200    
  
committer: GitHub <[email protected]>    
date     : Wed, 26 May 2021 19:07:18 +0200    

Click here for diff

The default is to keep all backup history to match the current behavior. In minimal configuration (0 days), unexpired backups are always kept in history.
  

  
When a full backup manifest expires, all dependent differential/incremental manifests expire as well.

M doc/xml/reference.xml
M doc/xml/release.xml
M src/build/config/config.yaml
M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/backup/common.h
M src/command/expire/expire.c
M src/command/help/help.auto.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/src/module/command/expireTest.c

Show diff when test.pl --gen-check detects unexpected autogeneration.

commit   : a69a6c2f5579a55624763592ce9d8d7e92dc55f0    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 May 2021 12:41:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 May 2021 12:41:15 -0400    

Click here for diff

This will help track down autogeneration issues if they can't be reproduced locally.  

M test/test.pl

Refactor storage/remote unit test using the protocol remote shim.

commit   : ba351e9c5ce475831de2aa22cc2b9afa8714a21d    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 May 2021 12:38:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 May 2021 12:38:23 -0400    

Click here for diff

Using the local process shim improves coverage and simplifies the tests.  

M doc/xml/release.xml
M test/define.yaml
M test/src/module/storage/remoteTest.c

Add remote process shim.

commit   : 58369c02df39ac9916d5de42d2df9ad509c33b6f    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 May 2021 18:16:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 May 2021 18:16:59 -0400    

Click here for diff

Run the remote process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the remote process rather than needing to test the remote protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test and the test does not need to run in a container.  

M test/define.yaml
M test/src/common/harnessProtocol.c
M test/src/common/harnessProtocol.h

Factor remote process exec out of protocolRemoteGet().

commit   : 441c000b5c3d8a4dfe67616d2a281c58184ef3ea    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 May 2021 18:09:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 May 2021 18:09:29 -0400    

Click here for diff

This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing.  
  
The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.  

M doc/xml/release.xml
M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Set buffer-size in the configuration test harness.

commit   : e1bc1b3e5321c585ed4e2162c28a5f881b1b9342    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 May 2021 17:48:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 May 2021 17:48:42 -0400    

Click here for diff

Most tests can set buffer-size as needed, but local/remote process shims are forked, so the best way to set buffer-size is via the configuration.  

M test/src/common/harnessConfig.c

Move protocol test module before config module.

commit   : cd88f8232950f1d6939c0642fa784c2d8a2110d6    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 May 2021 11:08:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 May 2021 11:08:51 -0400    

Click here for diff

The protocol module should be tested before modules that have a dependency on it.  

M test/define.yaml

Add local process shim to archive-get/archive-push unit tests.

commit   : 6732806d89ded2969fbf31d85c51e35c2d6f8345    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 May 2021 11:06:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 May 2021 11:06:05 -0400    

Click here for diff

Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.  

M doc/xml/release.xml
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c

Protocol shim improvements.

commit   : d55b9471a89163ed362c64960316c292e51cdd8e    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 May 2021 11:00:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 May 2021 11:00:24 -0400    

Click here for diff

Add executable to parameter list to avoid first option being lost. The backup, restore, and verify tests worked OK with their first option being defaulted because it ended up being job-retry which worked fine as the default.  
  
Add hrnProtocolLocalShimUninstall() allow the shim to be uninstalled.  
  
Log shim at debug level to make it obvious in the logs when a shim is in use.  

M doc/xml/release.xml
M test/src/common/harnessProtocol.c
M test/src/common/harnessProtocol.h

Fix shims with more than one function.

commit   : 55f52955a5da07e4ebee312c493759fa9dd3f566    
  
author   : David Steele <[email protected]>    
date     : Mon, 24 May 2021 17:45:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 24 May 2021 17:45:31 -0400    

Click here for diff

Each subsequent function would double every line in the C module.  
  
Move the loop to detect shimmed functions down to prevent this issues.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm

Add PostgreSQL 14 support.

commit   : 2452c4d5a4c230b8e3c33d3bddf69e56dfbec284    
  
author   : David Steele <[email protected]>    
date     : Mon, 24 May 2021 17:17:03 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 24 May 2021 17:17:03 -0400    

Click here for diff

There are no code changes from PostgreSQL 13 so simply add the new version.
  

  
Add CATALOG_VERSION_NO_MAX to allow the catalog version to "float" during the PostgreSQL beta/rc period so new pgBackRest versions are not required when the catalog version changes.
  

  
Update the integration tests to handle new PostgreSQL startup messages.

M doc/xml/release.xml
M src/Makefile.in
M src/postgres/interface.c
A src/postgres/interface/v140.c
M src/postgres/interface/version.h
M src/postgres/interface/version.intern.h
M src/postgres/interface/version.vendor.h
M src/postgres/version.h
M test/container.yaml
M test/define.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessPostgres.c
A test/src/common/harnessPostgres/harness140.c

commit   : eba013b49b850e53d59795ac6358fcb3381048a3    
  
author   : David Steele <[email protected]>    
date     : Mon, 24 May 2021 16:29:36 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 24 May 2021 16:29:36 -0400    

Click here for diff

manifestLinkCheck() was pretty inefficient so large numbers of links caused it to use a lot of memory and eventually crash. This is a more efficient implementation which runs O(nlogn) and uses far less memory.
  

  
Checking for duplicate file links has been added, which represents a change in behavior, but hopefully a good one.

M doc/xml/release.xml
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/info/manifestTest.c

Consolidate RHEL/CentOS user guide into a single document.

commit   : ccac75e7de8754b39bb691ff8830a2839ea7a285    
  
author   : David Steele <[email protected]>    
date     : Mon, 24 May 2021 16:03:15 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 24 May 2021 16:03:15 -0400    

Click here for diff

The user guide was split primarily to provide documentation for the stop-auto option in PostgreSQL <= 9.5. Now that 9.5 is EOL there does not seem to be a good reason to generate an extra user guide. The stop-auto function is still documented in the reference.
  

  
Leave the stop-auto documentation in the user guide in case we want to manually generate documentation for older versions.
  

  
Also rename centos to rhel for most identifiers since that is the core platform we are building for, similar to how we label 'debian' builds even though we generally use Ubuntu. With CentOS set to become an upstream for RHEL later this year, we'll likely need to pick a new test distribution, perhaps Rocky Linux if that gets off the ground.

M CONTRIBUTING.md
M doc/README.md
M doc/RELEASE.md
M doc/release.pl
M doc/xml/contributing.xml
M doc/xml/documentation.xml
M doc/xml/release.xml
M doc/xml/user-guide-index.xml
M doc/xml/user-guide.xml

Minor fixes for command/archive-get unit test.

commit   : f6303152f5f4b9ee538f4378797c9bf8f81bb3c8    
  
author   : David Steele <[email protected]>    
date     : Mon, 24 May 2021 12:40:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 24 May 2021 12:40:27 -0400    

Click here for diff

Test and remove WAL segment 000000010000000100000002 in the test where it is created rather than as a byproduct of a much later test.  
  
Remove incorrect local role from test. It worked, but was not the correct command role to be using when calling cmdArchiveGet().  
  
Move some harness headers down to the correct section.  

M test/src/module/command/archiveGetTest.c

Replace system() in tests with HRN_SYSTEM*().

commit   : e5e0b2119634847f624c824929f5e4af24da9c02    
  
author   : David Steele <[email protected]>    
date     : Sat, 22 May 2021 15:07:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 22 May 2021 15:07:18 -0400    

Click here for diff

M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/repoTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Rename TEST_SYSTEM*() to HRN_SYSTEM*().

commit   : bd40156c223837e3130ef9041ab7baf44570b362    
  
author   : David Steele <[email protected]>    
date     : Sat, 22 May 2021 14:22:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 22 May 2021 14:22:51 -0400    

Click here for diff

These calls are not tests, rather they setup data for tests.  

M test/src/common/harnessTest.h
M test/src/module/command/backupTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/compressTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Replace hrnLogResult() with TEST_RESULT_LOG/_FMT().

commit   : 73885f8c2e3a1ac234cce85684e81db961ad6d47    
  
author   : David Steele <[email protected]>    
date     : Sat, 22 May 2021 14:09:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 22 May 2021 14:09:45 -0400    

Click here for diff

The macros provide more information when there is an error and may be updated in the future without changing the test code.  

M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/exitTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/protocol/protocolTest.c

Replace TEST_ERROR_FMT() with TEST_ERROR() where possible.

commit   : 6baad5cdd1f73771e6f87d788769ba658c52ff80    
  
author   : David Steele <[email protected]>    
date     : Sat, 22 May 2021 11:59:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 22 May 2021 11:59:43 -0400    

Click here for diff

Some calls did not need TEST_ERROR_FMT() at all and others could be converted by replacing parameters with available defines, e.g. TEST_PATH.  

M test/src/module/command/archiveGetTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/info/manifestTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Remove comment formatting from TEST_*() macros.

commit   : a4f057bb70ca79c8b57ab5418f1b0cc65d3ae521    
  
author   : David Steele <[email protected]>    
date     : Sat, 22 May 2021 11:28:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 22 May 2021 11:28:56 -0400    

Click here for diff

Comment formatting was not used much but it incurred a heavy cost in each macro to process possible formatting.  
  
Remove formatted comments where they did not contain valuable information and replace with strZ(strNewFmt()) otherwise.  

M test/src/common/harnessInfo.h
M test/src/common/harnessStorage.h
M test/src/common/harnessTest.h
M test/src/module/common/logTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typePackTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/config/parseTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/protocol/protocolTest.c

Add defines for many test*() getter functions.

commit   : b270253a69cd44682e94e9cbfcf68f2dce4058a4    
  
author   : David Steele <[email protected]>    
date     : Sat, 22 May 2021 09:30:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 22 May 2021 09:30:54 -0400    

Click here for diff

A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests.  
  
Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE.  
  
Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements.  
  
In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.  

M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessConfig.c
M test/src/common/harnessLog.c
M test/src/common/harnessServer.c
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/logTest.c
M test/src/module/common/userTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c
M test/src/test.c

Rename strNew() to strNewZ() and add parameter-less strNew().

commit   : aed3d468a1557474eba4a24854732fc09581b265    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 May 2021 17:36:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 May 2021 17:36:43 -0400    

Click here for diff

Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not.  
  
In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/command/archive/common.c
M src/command/archive/get/get.c
M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/command.c
M src/command/expire/expire.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/restore/restore.c
M src/command/verify/verify.c
M src/common/exit.c
M src/common/ini.c
M src/common/io/http/common.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/lock.c
M src/common/type/buffer.c
M src/common/type/json.c
M src/common/type/pack.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringId.c
M src/common/type/stringList.c
M src/common/type/variant.c
M src/common/type/xml.c
M src/common/user.c
M src/config/config.c
M src/config/parse.c
M src/info/manifest.c
M src/postgres/client.c
M src/storage/azure/storage.c
M src/storage/gcs/storage.c
M src/storage/helper.c
M src/storage/s3/storage.c
M src/storage/storage.c
M test/src/common/harnessLog.c
M test/src/common/harnessPack.c
M test/src/common/harnessPq.c
M test/src/common/harnessServer.c
M test/src/common/harnessStorage.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/execTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/logTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Update contributing documentation.

commit   : 68faf1482a42ea14f5b00a91e19bd5aedec275b8    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 21 May 2021 13:21:25 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 21 May 2021 13:21:25 -0400    

Click here for diff

Add more examples of how to write code, add configuration options, test, etc.

M CONTRIBUTING.md
M README.md
M doc/xml/contributing.xml
M doc/xml/index.xml
M doc/xml/release.xml

Add log shim.

commit   : 15b8b9207da40df3df92ee7464a546c335a5a4d9    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 May 2021 12:51:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 May 2021 12:51:32 -0400    

Click here for diff

This allows DEBUG_UNIT and DEBUG_UNIT_EXTERN to be removed since static log variables can now be exposed by functions in the harness.  

M doc/xml/release.xml
M src/common/debug.h
M src/common/log.c
M test/define.yaml
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/module/common/debugOffTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c

Add local process shim.

commit   : ef63750e0bc8292caefe8e421a31e985026f72ee    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 May 2021 12:45:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 May 2021 12:45:00 -0400    

Click here for diff

Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test.  
  
The backup, restore, and verify command tests have been updated to use the new shim for coverage.  

M doc/xml/release.xml
M test/define.yaml
A test/src/common/harnessProtocol.c
A test/src/common/harnessProtocol.h
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/verifyTest.c

Add shim feature for unit tests.

commit   : cab7a97ab6308d72b509ef2e02c559ac691f436b    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 May 2021 18:47:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 May 2021 18:47:31 -0400    

Click here for diff

A shim allows a test harness to access static functions and variables in a C module, and also allows functions to be shimmed (i.e. overridden) for the purposes of testing.  
  
For instance, coverage testing works when a process that is normally exec'd is run as a forked child process instead.  

M test/define.yaml
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm

Factor local process exec out of protocolLocalGet().

commit   : e31df55c8dc3ce8c5cd071b0a66d83b3c7c3c4ed    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 May 2021 18:35:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 May 2021 18:35:30 -0400    

Click here for diff

This allows protocolLocalExec() to be shimmed, which means the local can be run as a child of the test process, simplifying coverage testing.  

M src/protocol/helper.c

Replace getopt_long() with custom implementation.

commit   : 45a4e801ed52f60f27c73eacaf63ff4b483ab352    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 May 2021 16:02:31 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 20 May 2021 16:02:31 -0400    

Click here for diff

getopt_long() requires an exhaustive list of all possible options that may be found on the command line. Because of the way options are indexed (e.g. repo1-4, pg1-8) optionList[] has 827 entries and we have kept it small by curtailing the maximum indexes very severely. Another issue is that getopt_long() scans the array sequentially so parsing gets slower as the index maximums increase.
  

  
Replace getopt_long() with a custom implementation that behaves the same but allows options to be parsed with a function instead of using optionList[]. This commit leaves the list in place in order to focus on the getopt_long() replacement, but cfgParseOption() could be replaced with a more efficient implementation that removes the need for optionList[].
  

  
This implementation also fixes an issue where invalid options were misreported in the error message if they only had one dash, e.g. -config. This seems to have been some kind of problem in getopt_long(), but no investigation was done since the new implementation fixes it.
  

  
Tests were added at 0825428, 2b8d2da, 34dd663, and 384f247 to check that previously untested getopt_long() behavior doesn't change.

M doc/xml/release.xml
M src/command/help/help.c
M src/config/parse.c
M src/config/parse.h
M src/config/protocol.c
M test/src/module/config/parseTest.c

Rename default command role to main.

commit   : 831ee81466c28ca563d5f5f5468681aca6fe3bea    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 May 2021 14:39:47 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 20 May 2021 14:39:47 -0400    

Click here for diff

Main makes more sense because we refer to the main process in the code, not the default process. The word default is pretty overloaded anyway.

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/xml/release.xml
M src/build/config/config.yaml
M src/command/help/help.c
M src/command/restore/restore.c
M src/config/config.c
M src/config/config.h
M src/config/load.c
M src/config/parse.auto.c
M src/config/parse.c
M src/main.c
M test/src/common/harnessConfig.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/commandTest.c
M test/src/module/common/exitTest.c
M test/src/module/config/execTest.c
M test/src/module/config/parseTest.c

Add recursion to TEST_STORAGE_LIST() macro.

commit   : 8453d89a92f6b7846bc8eeead285a1abd8fdfb1b    
  
author   : David Steele <[email protected]>    
date     : Wed, 19 May 2021 14:18:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 19 May 2021 14:18:43 -0400    

Click here for diff

This makes the macro useful when subpaths are present.  
  
Identify types other than files (path, link, etc.) with a single appended character for easier debugging.  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Fix invalid type in test function.

commit   : 93eb2f8362489eba2e6a6deee6b0d0c6dd3e6a2c    
  
author   : David Steele <[email protected]>    
date     : Wed, 19 May 2021 14:03:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 19 May 2021 14:03:42 -0400    

Click here for diff

This should have been char * from the start, but of course void * worked fine.  

M test/src/common/harnessTest.c
M test/src/common/harnessTest.intern.h

Update negate to indicate that it should not be used for new options.

commit   : 4d48850aad23042b3d7bf22221e2f5d34c00de07    
  
author   : David Steele <[email protected]>    
date     : Wed, 19 May 2021 12:58:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 19 May 2021 12:58:48 -0400    

Click here for diff

M src/build/config/config.yaml

Add missing header.

commit   : 870bcf30af1a516802a67ac173e8bd7a369c6530    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 May 2021 16:48:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 May 2021 16:48:30 -0400    

Click here for diff

This worked because all places where this header was included, common/type/stringId.h was included first.  

M src/protocol/helper.h

Fix typo in struct typedef.

commit   : 4fa95af447d1b9caecbaacb5e7a5ef4b334f7331    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 May 2021 16:38:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 May 2021 16:38:57 -0400    

Click here for diff

This worked because the type is never declared as struct ExecPub.  

M src/common/exec.h

Simplify defaults for --force option.

commit   : c92bc43c1eca6d3deac288f68332b323139e6405    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 18 May 2021 16:33:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 May 2021 16:33:45 -0400    

Click here for diff

All commands have the same default so it is more efficient to set the default for the entire option.  

M doc/xml/release.xml
M src/build/config/config.yaml
M src/config/parse.auto.c

Remove stanza archive spool path on restore.

commit   : 320c6e1aad2cc9a8dabecb2bc91a9b34464de6ec    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 May 2021 15:49:22 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 18 May 2021 15:49:22 -0400    

Click here for diff

Remove stanza archive spool path so existing files do not interfere with the new cluster. For instance, old archive-push acknowledgements could cause a new cluster to skip archiving. This should not happen if a new timeline is selected but better to be safe. Missing stanza spool paths are ignored.
  

  
Also add new path expression STORAGE_SPOOL_ARCHIVE to easily access this path.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/command/help/help.auto.c
M src/command/restore/restore.c
M src/config/parse.auto.c
M src/storage/helper.c
M src/storage/helper.h
M test/expect/mock-all-001.log
M test/src/module/command/restoreTest.c
M test/src/module/storage/posixTest.c

Move includes after typedef in info/manifest.h.

commit   : dc0e6645cd5c4baa3f301feda4acfce55ba363f7    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 May 2021 09:08:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 May 2021 09:08:06 -0400    

Click here for diff

This allows the Manifest type to be used in headers that need it, e.g. command/backup/common.h.  

M src/info/manifest.h

Add automatic GCS authentication for GCE instances.

commit   : 9af033194abc4b1cd3f06c3a97fff71c966c3827    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 May 2021 14:55:50 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 17 May 2021 14:55:50 -0400    

Click here for diff

When running on a GCE instance the authentication token can be pulled directly from the instance metadata. This is configured with repo-gcs-key-type=auto.
  

  
In a separate commit (26fefa6), move the code that parses the token response into a separate function, storageGcsAuthToken(), since it is now needed by two key types. This drastically improves the readability of the main commit.

M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/config/config.yaml
M src/command/help/help.auto.c
M src/config/config.auto.h
M src/config/parse.auto.c
M src/storage/gcs/storage.c
M src/storage/gcs/storage.h
M src/storage/helper.c
M test/src/module/storage/gcsTest.c

Remove default VM for test.pl --coverage-only option.

commit   : 0152075e6bc5fca2b78cbe6051a6b3d998c9030f    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 May 2021 11:35:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 17 May 2021 11:35:22 -0400    

Click here for diff

When running outside of our standard Vagrantfile the default will not be set correctly, so require the user to set it.  
  
In any case, this option is primarily useful for reporting so note that in the command line help.  

M test/test.pl

Add config/parse tests for config/env partial options.

commit   : 384f2470775bd2d867529e7267c12a798a259ab3    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 May 2021 09:33:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 17 May 2021 09:33:36 -0400    

Click here for diff

Partial option matching is valid on the command line but should never be used in config files or environment options.  

M test/src/module/config/parseTest.c

Move PostgreSQL version interface test functions to a test harness.

commit   : ae7f0af2023ec7b9261a5a4887351eec669c4442    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 May 2021 07:20:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 17 May 2021 07:20:28 -0400    

Click here for diff

Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core.  
  
Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.  

M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/interface/version.h
M src/postgres/interface/version.intern.h
M test/define.yaml
M test/lib/pgBackRestTest/Common/JobTest.pm
A test/src/common/harnessPostgres.c
A test/src/common/harnessPostgres.h
A test/src/common/harnessPostgres/harness083.c
A test/src/common/harnessPostgres/harness084.c
A test/src/common/harnessPostgres/harness090.c
A test/src/common/harnessPostgres/harness091.c
A test/src/common/harnessPostgres/harness092.c
A test/src/common/harnessPostgres/harness093.c
A test/src/common/harnessPostgres/harness094.c
A test/src/common/harnessPostgres/harness095.c
A test/src/common/harnessPostgres/harness096.c
A test/src/common/harnessPostgres/harness100.c
A test/src/common/harnessPostgres/harness110.c
A test/src/common/harnessPostgres/harness120.c
A test/src/common/harnessPostgres/harness130.c
A test/src/common/harnessPostgres/harnessVersion.intern.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/interfaceTest.c

Remove overzealous cast.

commit   : 18cc02238bbd15bfa9827ef86078aa172bdae52b    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 May 2021 07:15:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 17 May 2021 07:15:13 -0400    

Click here for diff

There does not seem to be a good reason to have an explicit cast for such a small number, so remove it.  
  
Leave PG_CONTROL_SIZE alone since it will be removed in an upcoming commit.  

M src/postgres/interface.c

Revert ignoring catalog version when identifying a PostgreSQL version.

commit   : 9235c62c6b597bc4ed812e0c0a3bc3a36ec30767    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 May 2021 07:12:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 17 May 2021 07:12:46 -0400    

Click here for diff

927d9adb changed the way CATALOG_VERSION_NO is used to identify PostgreSQL versions since PG_CONTROL_VERSION is generally bumped with each release. The goal was to make the beta/rc period less painful because any CATALOG_VERSION_NO bump renders pgBackRest inoperative.  
  
This worked, but in fact we'd rather be stricter about which CATALOG_VERSION_NO we accept when identifying a version of PostgreSQL. It is not just about identifying a major version, but making sure the build contains all the functions and catalogs we expect to make pgBackRest work correctly. It is better to reject early dev/beta/rc builds that may not work.  
  
Since 927d9adb was relatively recent the chance that this stricter checking will cause a problem seems minimal, so revert to checking CATALOG_VERSION_NO for every PostgreSQL version.  
  
Leave in place the code that pulls CATALOG_VERSION_NO from pg_control rather than the internal constant since the plan is still to allow catalog versions to "float" during the PostgreSQL beta/rc phase, which will be the subject of a future commit.  

M doc/xml/release.xml
M src/postgres/interface/version.intern.h
M src/postgres/interface/version.vendor.h

Fix issues with leftover spool files from a prior restore.

commit   : f45e76fa2e8ce0f62317a74fdd51083d52659cd5    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 May 2021 17:51:39 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 13 May 2021 17:51:39 -0400    

Click here for diff

If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue.
  

  
Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process.
  

  
Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c

Add config/parse tests for options and option args with spaces.

commit   : 34dd6636b83ac41cab592d6a6d18c082e1ee272e    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 May 2021 17:01:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 May 2021 17:01:21 -0400    

Click here for diff

If an option name has a space at the beginning then it will be considered an invalid command, but a space at the end is an invalid option. Add tests for these conditions.  
  
Spaces in option arguments should be preserved, so add a test to be sure this is true.  

M test/src/module/config/parseTest.c

Add config/parse tests for partial options.

commit   : 2b8d2daca15f2535672f195164c0e8daffdade90    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 May 2021 16:54:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 May 2021 16:54:25 -0400    

Click here for diff

Add coverage for partial options, i.e. an option that is partially specified but only prefix matches with a single valid option.  

M test/src/module/config/parseTest.c

Fix an ungrammatical error message in config parsing.

commit   : add0f9b77de4b9f1395338e4b092bf77b2da2cd5    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 May 2021 12:37:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 May 2021 12:37:59 -0400    

Click here for diff

M src/config/parse.c
M test/src/module/config/parseTest.c

Add config/parse test where the option/value are not in the same arg.

commit   : 0825428fef00e6b4b107526ffa462fa7558aace4    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 May 2021 12:35:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 May 2021 12:35:11 -0400    

Click here for diff

All the tests in this module use --option=value syntax so add one test with --option value syntax for coverage.  

M test/src/module/config/parseTest.c

Refactor GCS token response parsing into a separate function.

commit   : 26fefa6aeeae90bbb572ea57b605fb8fe526f1a0    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 May 2021 06:59:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 May 2021 06:59:57 -0400    

Click here for diff

This function is also useful for the upcoming auto authentication. Refactoring in a separate commit to make the feature commit more readable.  

M src/storage/gcs/storage.c

Use existing variable for GCS test server port.

commit   : 7b71604defa02d5ed9b7aa3407961cb59a52cc45    
  
author   : David Steele <[email protected]>    
date     : Wed, 12 May 2021 13:21:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 12 May 2021 13:21:59 -0400    

Click here for diff

Also fix incorrect separator comment.  

M test/src/module/storage/gcsTest.c

Convert option values in commands to StringId.

commit   : 5464ac83d121c653e95f337fa14870bbe35dc9f7    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 May 2021 17:24:30 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 11 May 2021 17:24:30 -0400    

Click here for diff

Convert most of the remaining options that benefit from being StringIds. Since all the command modules can include config.h directly it makes sense to auto-generate these values instead of manually creating an enum for each one.
  

  
For the time being StringIds are not being auto-generated because the StringId code does not exist in Perl. However, the *_Z zero-terminated constants for each allowed option value are now auto-generated.

M build/lib/pgBackRestBuild/Config/Build.pm
M doc/xml/release.xml
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/repo/ls.c
M src/command/restore/restore.c
M src/config/config.auto.h
M src/config/config.h
M src/config/load.c
M test/src/module/command/restoreTest.c

Change CentOS 7 documentation test to CentOS 8.

commit   : 87ba2ca253ec9546cb23635ec15017bb5ce7d6e9    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 May 2021 16:54:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 May 2021 16:54:42 -0400    

Click here for diff

The CentOS 7 documentation test relies on PostgreSQL 9.5 which has been removed from the yum.p.o repository package. Switch the test to CentOS 8 to fix the immediate issue, but a decision on the PostgreSQL 9.5 documentation will need to be made before the next release.  

M .github/workflows/test.yml
M test/ci.pl

Add instructions to install lcov for MacOS.

commit   : 2bfebff2ec73b0af6d937f303fbb9b8a0b8ee649    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 May 2021 15:51:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 May 2021 15:51:46 -0400    

Click here for diff

M test/Vagrantfile

Exclude Dockerfile from code count.

commit   : 54074a569708e40831d767a8b93074f41feec308    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 May 2021 07:52:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 May 2021 07:52:35 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Basic multi-architecture support for test containers.

commit   : baddec1e9ae37ad08840401e2602301f06bda021    
  
author   : David Steele <[email protected]>    
date     : Mon, 3 May 2021 16:31:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 3 May 2021 16:31:27 -0400    

Click here for diff

The tests worked fine on multiple architectures, but would only run "bare metal", i.e. tests that required containers could not be run.  
  
Enable basic multi-architecture support by allowing containers to be built using whatever architecture the host supports. Also allow cached containers to be defined for multiple architectures in container.yaml.  
  
Add a Dockerfile which can be used as a container for other containers to provide a consistent development environment.  
  
The primary goal is to allow development on Mac M1 but other architectures should find these improvements useful.  

A test/Dockerfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/test.pl

Convert BackupType enum to StringId.

commit   : 87df6d7a58a6ab3c4748ad75e4128cdad761b15a    
  
author   : David Steele <[email protected]>    
date     : Mon, 3 May 2021 12:15:39 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 3 May 2021 12:15:39 -0400    

Click here for diff

Allows removal of backupType()/backupTypeStr() and improves debug logging of the enum.
  

  
Move BackupType enum and string constants to info/infoBackup.h so they are available to more modules. Also convert InfoBackup to use BackupType instead of a String.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/backup/common.h
M src/command/expire/expire.c
M src/command/info/info.c
M src/common/type/stringId.h
M src/config/config.h
M src/config/load.c
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/manifest.c
M src/info/manifest.h
M test/define.yaml
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/info/infoBackupTest.c

Add new instructions for PostgreSQL news and include a sample.

commit   : 568dc0ba0c762376b8bd902d83e38f2649a156af    
  
author   : David Steele <[email protected]>    
date     : Mon, 3 May 2021 09:34:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 3 May 2021 09:34:22 -0400    

Click here for diff

It is no longer possible to pull news source from the PostgreSQL website so add a sample in the doc directory. Update the release instructions to reflect this change.  
  
Also note that it is no longer necessary to post separately to pgsql-announce.  

A doc/NEWS.md
M doc/RELEASE.md

Convert InfoPgType enum to StringId.

commit   : fb7ddce8076107c33c25832c9acb8baffbd57563    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 13:21:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 13:21:24 -0400    

Click here for diff

Improves debug logging by outputting archive/backup instead of 0/1.  

M doc/xml/release.xml
M src/info/infoPg.c
M src/info/infoPg.h

Update IoClient/IoSession to use StringIds.

commit   : bd0081fec87ee32fa57adf5c882a6cad5394eaa0    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 12:37:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 12:37:22 -0400    

Click here for diff

Using StringId for the client/session type removes String constants and some awkward referencing/dereferencing needed to use a String constant in the interface.  
  
Converting IoSessionRole to StringId removes a conditional in ioSessionToLog() and improves debug logging by outputting client/server instead of 0/1.  

M doc/xml/release.xml
M src/common/io/client.c
M src/common/io/client.intern.h
M src/common/io/session.c
M src/common/io/session.h
M src/common/io/session.intern.h
M src/common/io/socket/client.c
M src/common/io/socket/client.h
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M src/common/io/tls/session.c

Convert ArchivePushFileIoType enum to StringId.

commit   : 0ec91f61c636731a8a46fc69307bbe5d78ac8871    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 12:23:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 12:23:37 -0400    

Click here for diff

Improves debug logging by outputting open/write/close instead of 0/1/2.  

M doc/xml/release.xml
M src/command/archive/push/file.c

Convert ArchiveMode enum to StringId.

commit   : 8394577c6ac1a13884b4b05052ce6328d38ab067    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 12:05:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 12:05:01 -0400    

Click here for diff

Improves debug logging by outputting get/push instead of 0/1.  

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h

Convert ProtocolStorageType enum to StringId.

commit   : 7dd01897fd20a3629e7333dc47bed7d02035b501    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 11:59:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 11:59:04 -0400    

Click here for diff

Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.  

M doc/xml/release.xml
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/remote/protocol.c
M test/define.yaml
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c

Convert ProtocolParallelJobState enum to StringId.

commit   : bd68ed63ba38c841e286e0ab5390450f57c23b37    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 11:43:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 11:43:08 -0400    

Click here for diff

Allows removal of protocolParallelJobToConstZ(), which was used only for debugging.  

M doc/xml/release.xml
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h

Update CipherType/CipherMode to StringId.

commit   : 85fc3da4c37dc1cb9af3b8c5248121769ab56650    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 11:36:20 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 28 Apr 2021 11:36:20 -0400    

Click here for diff

As in 6cc521b, this allows option values and enums to be easily mapped together.

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/protocol.c
M src/command/check/check.c
M src/command/check/common.c
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/repo/get.c
M src/command/repo/put.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/upgrade.c
M src/command/verify/verify.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/common.c
M src/common/crypto/common.h
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/manifest.c
M src/protocol/helper.c
M test/src/common/harnessStorage.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/common/cryptoTest.c

Fix comments where 'output' was misspelled as 'ouput'.

commit   : c3b15fc3bdc332591c6b059c2b23b1cce990d387    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 28 Apr 2021 10:58:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Apr 2021 10:58:45 -0400    

Click here for diff

M src/command/help/help.c
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/LogTest.pm

Refactor String, Buffer, and Variant types with inline getters.

commit   : 066fbcf26826ee8332eb76db0db9d684047513ac    
  
author   : David Steele <[email protected]>    
date     : Tue, 27 Apr 2021 15:25:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 27 Apr 2021 15:25:10 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the String, Buffer, and Variant types.  

M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/variant.c
M src/common/type/variant.h
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c

Add cfgOptionDisplay()/cfgOptionIdxDisplay().

commit   : 1edcfde93e47c79a8b18b05f7ddb1e9fde29b0ed    
  
author   : David Steele <[email protected]>    
date     : Tue, 27 Apr 2021 12:12:43 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 27 Apr 2021 12:12:43 -0400    

Click here for diff

Centralize the formatting of the configuration value for display to the user or passing on a command line.
  

  
For the new functions, if the value was set by the user via the command line, config, etc., then that exact value will be displayed. This makes it easier for the user to recognize the value and saves having to format it into something reasonable, especially for time and size option types.
  

  
Note that cfgOptTypeHash and cfgOptTypeList option types are not supported by these functions, but they are generally not displayed to the user as a whole.
  

  
This also fixes a bug in config/load.c where time values where not being formatted correctly in an error message.

M doc/xml/release.xml
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/check/common.c
M src/command/command.c
M src/command/control/common.c
M src/command/control/start.c
M src/command/control/stop.c
M src/command/help/help.c
M src/command/local/local.c
M src/command/remote/remote.c
M src/command/repo/get.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/command/stanza/upgrade.c
M src/config/config.c
M src/config/config.h
M src/config/config.intern.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c

Preserve unused YAML fix for older Debian/Ubuntu distributions.

commit   : 95fb002b85897e71820c44519231975d883f87db    
  
author   : David Steele <[email protected]>    
date     : Tue, 27 Apr 2021 12:04:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 27 Apr 2021 12:04:50 -0400    

Click here for diff

This fix was not used since backports worked for older Debians and the old affected Ubuntus are EOL.  
  
Still, seems worth preserving just in case it comes up elsewhere.  

M build/lib/pgBackRestBuild/Config/Data.pm

Update storage module to use StringIds.

commit   : 6cc521b6b232cd10b5b1622b1841d988113d0c16    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Apr 2021 13:19:47 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 23 Apr 2021 13:19:47 -0400    

Click here for diff

Use StringIds for the storage types (e.g. STORAGE_S3_TYPE) and configuration settings, e.g. cfgOptS3KeyType.
  

  
Also add new config functions and harness config functions to support StringIds.

M doc/xml/release.xml
M src/command/repo/create.c
M src/config/config.c
M src/config/config.h
M src/config/load.c
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/storage.h
M src/storage/azure/write.c
M src/storage/cifs/storage.c
M src/storage/cifs/storage.h
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/gcs/storage.h
M src/storage/gcs/write.c
M src/storage/helper.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/storage.h
M src/storage/posix/storage.intern.h
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/read.h
M src/storage/read.intern.h
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/storage.h
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/storage.h
M src/storage/s3/write.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M src/storage/write.c
M src/storage/write.h
M src/storage/write.intern.h
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Do not write files atomically or sync paths during backup copy.

commit   : aa72c19a831fb0dd15a50a003fbaad427f335353    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Apr 2021 12:33:25 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 23 Apr 2021 12:33:25 -0400    

Click here for diff

There is no need to write the file atomically (e.g. via a temp file on Posix) because checksums are tested on resume after a failed backup. The path does not need be synced for each file because all paths are synced at the end of the backup.
  

  
This functionality was not lost during the migration -- it never existed in the Perl code, though these settings are used in restore. See 59f1353 where backupFile() was migrated to C.

M doc/xml/release.xml
M src/command/backup/file.c
M test/src/module/command/backupTest.c

Add help for all internal options valid for default roles.

commit   : aaa15b9709cd58ad56e5008858867d91f4abe803    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Apr 2021 11:46:03 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 23 Apr 2021 11:46:03 -0400    

Click here for diff

Fix the segfault when getting help for an internal option is requested by adding help for all internal options that are valid for a default command role.
  

  
Also print warnings about internal options in code rather than putting in each command/option description.

M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c
M src/command/help/help.c
M test/src/module/command/helpTest.c

Remove documentation about enum truncation and add tests.

commit   : 2ad497ea4cdc53f5966f2abab018696ff4694a2d    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Apr 2021 08:04:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Apr 2021 08:04:03 -0400    

Click here for diff

The enum truncation observed was due to the value getting passed via a protocol function which silently narrowed the enum.  
  
Even so, add some tests to ensure tested platforms support 64-bit enums.  

M src/common/type/stringId.h
M test/src/module/common/typeStringTest.c

Replace misused kvAdd() with kvPut().

commit   : bcc925b7406ae590cd24ff5a0bbb789a6d1ecd5a    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 20:04:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 20:04:27 -0400    

Click here for diff

Although kvAdd() works like kvPut() on the first call, kvPut() is more efficient when a key has a single value.  
  
Update the comment to clarify that kvAdd() is seldom required.  

M src/command/info/info.c
M src/common/io/filter/group.c
M src/common/stat.c
M src/common/type/keyValue.h
M src/protocol/server.c
M test/src/module/common/typeJsonTest.c
M test/src/module/storage/remoteTest.c

Remove redundant CompressType conversions.

commit   : a2f02f95ec5f2835d3c114d1b96fec8109e11402    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 19:28:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 19:28:11 -0400    

Click here for diff

These seem to be the result of overzealous copy/paste.  
  
The conversions do not even serve as a verification of locally available compression types since compressTypeEnum() and compressTypeStr() do not check that.  

M src/command/backup/backup.c

Remove auto-generated option String constants.

commit   : 468aa79ea8b1975b1d6fe0665611230432f16d75    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 19:10:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 19:10:13 -0400    

Click here for diff

These are rarely used and end up using more space than they save since most of them are never referenced.  
  
Replace VARSTR() with VARSTRDEF() where these constants are being used.  

M build/lib/pgBackRestBuild/Config/Build.pm
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/restore/restore.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/protocol/helper.c

Test pull requests in Cirrus CI.

commit   : 6a39c51f8f1d992c53bc3af80d9db0037c09fb41    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 18:18:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 18:18:34 -0400    

Click here for diff

Both FreedBSD and MacOS use clang as their default compiler and clang catches some errors that gcc does not. Specifically, wide integers being assigned to short integers resulting in possible truncations.  

M .cirrus.yml

Free no longer needed remotes so they do not timeout during restore.

commit   : 06fa18582a405417c8ab4ba667cb01a13a5c3103    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 12:23:08 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 22 Apr 2021 12:23:08 -0400    

Click here for diff

The remotes are no longer needed in the main process after the manifest is loaded. If the restore is long enough the connection will timeout and WARN at the end of the restore. This is harmless for the restore but distracting for the user.
  

  
To prevent this, free the remotes once they are no longer needed.

M doc/xml/release.xml
M src/command/restore/restore.c

Fix help when a valid option is invalid for the specified command.

commit   : 45f83558ea862c6f496a27b886e1160e6303bbe8    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 11:48:04 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 22 Apr 2021 11:48:04 -0400    

Click here for diff

Getting help for a valid option that was invalid for the command would segfault.
  

  
Add a check to ensure the option is valid for the command's default role.

M doc/xml/release.xml
M src/command/help/help.c
M test/src/module/command/helpTest.c

Use option constants in warnings.

commit   : 72dac5b10b7cd31a00212be9268df95c2d12615c    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 08:15:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 22 Apr 2021 08:15:36 -0400    

Click here for diff

In once case the constant replaces a formatted statement, which is more efficient, and in the other case plain text, which is more maintainable.  

M src/command/check/check.c
M src/command/stanza/create.c

Fix incorrect test title.

commit   : 20a018a8ae9ce8768044621e19ef716c5622f209    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Apr 2021 18:27:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Apr 2021 18:27:18 -0400    

Click here for diff

M test/src/module/command/archiveGetTest.c

Fix indentation.

commit   : c5b446d406937d35b277cd0d9131e6c058183f3b    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Apr 2021 10:09:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Apr 2021 10:09:07 -0400    

Click here for diff

M src/command/help/help.c

Add const to inline functions where appropriate.

commit   : fd69357302053a79fdef6decf1ff284a8a442e67    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Apr 2021 18:43:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Apr 2021 18:43:16 -0400    

Click here for diff

This lets the compiler know that these variables are not modified which should lead to better optimization.  
  
Smart compilers should be able to figure this out on their own, but marking parameters const is still good for documentation.  

M src/common/exec.h
M src/common/ini.h
M src/common/io/client.h
M src/common/io/fd.h
M src/common/io/filter/filter.h
M src/common/io/filter/group.h
M src/common/io/http/client.h
M src/common/io/http/header.h
M src/common/io/http/query.h
M src/common/io/http/request.h
M src/common/io/http/response.h
M src/common/io/http/session.h
M src/common/io/http/url.h
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/session.h
M src/common/io/write.h
M src/common/regExp.h
M src/common/type/buffer.h
M src/common/type/convert.h
M src/common/type/keyValue.h
M src/common/type/list.h
M src/common/type/mcv.h
M src/common/type/pack.h
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/common/type/variantList.h
M src/common/type/xml.h
M src/common/wait.h
M src/config/parse.c
M src/db/db.h
M src/info/infoArchive.h
M src/info/infoBackup.h
M src/info/manifest.h
M src/postgres/client.h
M src/protocol/client.h
M src/protocol/command.h
M src/protocol/parallel.h
M src/protocol/parallelJob.h
M src/protocol/server.h
M src/storage/read.h
M src/storage/storage.h
M src/storage/storage.intern.h
M src/storage/write.h
M test/src/common/harnessConfig.c

Add StringId type.

commit   : ed0d48f52cb5332f6804737ac01c43bf76d11a50    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Apr 2021 15:22:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Apr 2021 15:22:42 -0400    

Click here for diff

It is often useful to represent identifiers as strings when they cannot easily be represented as an enum/integer, e.g. because they are distributed among a number of unrelated modules or need to be passed to remote processes. Strings are also more helpful in debugging since they can be recognized without cross-referencing the source. However, strings are awkward to work with in C since they cannot be directly used in switch statements leading to less efficient if-else structures.  
  
A StringId encodes a short string into an integer so it can be used in switch statements but may also be readily converted back into a string for debugging purposes. StringIds may also be suitable for matching user input providing the strings are short enough.  
  
This patch includes a sample of StringId usage by converting protocol commands to StringIds. There are many other possible use cases. To list a few:  
  
* All "types" in storage, filters. IO , etc. These types are primarily for identification and debugging so they fit well with this model.  
  
* MemContext names would work well as StringIds since these are entirely for debugging.  
  
* Option values could be represented as StringIds which would mean we could remove the functions that convert strings to enums, e.g. CipherType.  
  
* There are a number of places where enums need to be converted back to strings for logging/debugging purposes. An example is protocolParallelJobToConstZ. If ProtocolParallelJobState were defined as:  
  
typedef enum  
{  
    protocolParallelJobStatePending = STRID5("pend", ...),  
    protocolParallelJobStateRunning = STRID5("run", ...),  
    protocolParallelJobStateDone = STRID5("done", ...),  
} ProtocolParallelJobState;  
  
then protocolParallelJobToConstZ() could be replaced with strIdToZ(). This also applies to many enums that we don't covert to strings for logging, such as CipherMode.  
  
As an example of usage, convert all protocol commands from strings to StringIds.  

M doc/xml/release.xml
M src/Makefile.in
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/archive/get/protocol.h
M src/command/archive/push/protocol.c
M src/command/archive/push/protocol.h
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/protocol.c
M src/command/backup/protocol.h
M src/command/restore/protocol.c
M src/command/restore/protocol.h
M src/command/restore/restore.c
M src/command/verify/protocol.c
M src/command/verify/protocol.h
M src/command/verify/verify.c
A src/common/type/stringId.c
A src/common/type/stringId.h
M src/config/protocol.c
M src/config/protocol.h
M src/db/db.c
M src/db/protocol.c
M src/db/protocol.h
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.c
M src/protocol/command.h
M src/protocol/server.c
M src/protocol/server.h
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M test/define.yaml
M test/src/module/command/backupTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/protocol/protocolTest.c

Add db-exclude option.

commit   : 292f836f1243e51d102bb4be202e7b21a7a386b1    
  
author   : Stefan Fercot <[email protected]>    
date     : Mon, 19 Apr 2021 21:01:00 +0200    
  
committer: GitHub <[email protected]>    
date     : Mon, 19 Apr 2021 21:01:00 +0200    

Click here for diff

Restore excluding the specified databases. Databases excluded will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, those databases will not be accessible but can be removed with the drop database command. The --db-exclude option can be passed multiple times to specify more than one database to exclude.
  

  
When used in combination with the --db-include option, --db-exclude will only apply to standard system databases (template0, template1, and postgres).

M doc/xml/reference.xml
M doc/xml/release.xml
M src/build/config/config.yaml
M src/command/help/help.auto.c
M src/command/restore/restore.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/src/module/command/helpTest.c
M test/src/module/command/restoreTest.c

Add storageDriver() inlines missed in b715c70b.

commit   : a75b413ac228452acfd33aa3d296d8edbeae9604    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Apr 2021 14:22:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Apr 2021 14:22:36 -0400    

Click here for diff

M src/storage/storage.c
M test/src/module/storage/posixTest.c

Update to remove files after archive-get test completes.

commit   : d859fe8c4d60e961cf688f17fb80d0cdc8fb810d    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Apr 2021 15:48:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Apr 2021 15:48:04 -0400    

Click here for diff

This cleans up the spool path for the next test. Currently there isn't one but there will be.  

M test/src/module/command/archiveGetTest.c

Remove lockClear().

commit   : 040ad71f8cd7b57ec6546e4733749c827ed0aaa3    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Apr 2021 15:41:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Apr 2021 15:41:55 -0400    

Click here for diff

This function has not been used since the switch to the fork/exec model.  
  
lockClear() was still used in one test (other than the lock test) so update the test and remove the function.  

M src/common/lock.c
M src/common/lock.h
M test/src/module/command/archiveGetTest.c
M test/src/module/common/lockTest.c

Define DEBUG in build.auto.c.

commit   : c2d4a0286e5899f3382bd34f9d285f122753bca5    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 18:06:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 18:06:07 -0400    

Click here for diff

Both NDEBUG and DEBUG were used in the code, which was a bit confusing.  
  
Define DEBUG in build.auto.c so it is available in all C and header files and stop using NDEBUG. This is preferable to using NDEBUG everywhere since there are multiple DEBUG* defines, e.g. DEBUG_COVERAGE.  
  
Note that NDEBUG is still required since it is used by the C libraries.  

M src/build.auto.h.in
M src/common/assert.h
M src/common/debug.h
M src/common/error.c
M src/common/error.h
M src/common/io/filter/group.h
M src/common/io/read.intern.h
M src/common/stackTrace.c
M src/common/stackTrace.h
M test/src/common/harnessConfig.c
M test/src/common/harnessDebug.h
M test/src/common/harnessInfo.c
M test/src/common/harnessLog.c
M test/src/common/harnessPack.c
M test/src/common/harnessPq.c
M test/src/common/harnessServer.c
M test/src/common/harnessStorage.c
M test/src/common/harnessTest.c
M test/src/module/common/stackTraceTest.c
M test/src/test.c

Update Cirrus CI FreeBSD release to 12.2.

commit   : c1aae434ca18ce06a6ea56b779fbdb5702ee1034    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 17:03:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 17:03:28 -0400    

Click here for diff

M .cirrus.yml

Add THIS_PUB() macro to simplify inline object accessors.

commit   : a0e24d492f8df97788018fc91d6e408c1c787e15    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 16:58:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 16:58:02 -0400    

Click here for diff

In combination with the thisPub() function, this macro simplifies accessing the public part of a private object struct.  
  
thisPub() asserts this != NULL so the caller does not need to do it.  

M CODING.md
M doc/xml/coding.xml
M src/common/exec.h
M src/common/io/client.h
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.h
M src/common/io/http/client.h
M src/common/io/http/request.h
M src/common/io/http/response.h
M src/common/io/http/url.h
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/session.h
M src/common/io/write.h
M src/common/type/keyValue.h
M src/common/type/list.h
M src/common/type/object.h
M src/common/type/xml.h
M src/common/wait.h
M src/db/db.h
M src/info/info.h
M src/info/infoArchive.h
M src/info/infoBackup.h
M src/info/infoPg.h
M src/info/manifest.h
M src/protocol/client.h
M src/protocol/parallelJob.h
M src/protocol/server.h
M src/storage/read.h
M src/storage/storage.h
M src/storage/storage.intern.h
M src/storage/write.h

Refactor remaining common/io modules with inline getters/setters.

commit   : 9fec4ce98c69044eae4e299e6e95bc45990d6d5c    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 14:37:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 14:37:02 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the remaining common/io modules.  

M src/common/exec.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/client.c
M src/common/io/client.h
M src/common/io/client.intern.h
M src/common/io/fdRead.c
M src/common/io/fdWrite.c
M src/common/io/http/response.c
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/session.c
M src/common/io/session.h
M src/common/io/session.intern.h
M src/common/io/socket/client.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/session.c
M src/common/io/write.c
M src/common/io/write.h
M src/storage/gcs/read.c
M src/storage/posix/read.c
M src/storage/posix/write.c
M src/storage/read.intern.h
M src/storage/remote/read.c
M src/storage/remote/write.c
M src/storage/write.intern.h
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c

Clarify that repo-s3-role is not an ARN.

commit   : 5bf160643b53699a28a8f0920c36d6ded767d249    
  
author   : Isaacwhyuenac <[email protected]>    
date     : Tue, 13 Apr 2021 14:02:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 14:02:20 -0400    

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c

Add bash to Cirrus CI FreeBSD package install.

commit   : 2c9c59170f6c1a3aacfdf0fb5c6197feb87afcc9    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 12:38:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Apr 2021 12:38:17 -0400    

Click here for diff

It appears bash was removed from the base image so install explicitly.  

M .cirrus.yml

Refactor common/io/filter module with inline getters/setters.

commit   : 8844ced384bf0d5feb5fe395560ec976cc292a73    
  
author   : David Steele <[email protected]>    
date     : Mon, 12 Apr 2021 16:05:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 12 Apr 2021 16:05:40 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the common/io/filter module.  

M src/command/backup/pageChecksum.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/compress/zst/compress.c
M src/common/compress/zst/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/sink.c
M src/common/io/filter/size.c
M src/info/info.c
M src/storage/gcs/write.c
M test/src/common/harnessInfo.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/ioTest.c
M test/src/module/performance/storageTest.c

Refactor common/type/keyValue module with inline getters/setters.

commit   : 3b9bed95186118a3fbf47e3e0900e74f4ad01217    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 17:29:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 17:29:14 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the common/type/keyValue module.  

M src/common/type/keyValue.c
M src/common/type/keyValue.h
M test/src/module/common/typeKeyValueTest.c

Refactor common/type/xml module with inline getters/setters.

commit   : c434b9ed885e083baed79c76426a7996d1a17d30    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 17:03:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 17:03:55 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the common/type/xml module.  

M src/common/type/xml.c
M src/common/type/xml.h

Remove unused XML constructors.

commit   : e18f3c1701b83037431354c41871e9daf16db9d4    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 15:56:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 15:56:42 -0400    

Click here for diff

These alternate constructors may have been used in the Perl days but they are no longer used so remove them.  

M src/common/type/xml.c
M src/common/type/xml.h
M test/src/module/common/typeXmlTest.c

Refactor common/exec module with inline getters/setters.

commit   : ddd37ebf7dcef30322346c34f09aae44a0ca35a2    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 15:31:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 15:31:31 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the common/exec module.  

M src/common/exec.c
M src/common/exec.h
M test/src/module/common/execTest.c
M test/src/module/protocol/protocolTest.c

Refactor db module with inline getters/setters.

commit   : 1d674c352f1d6fc1875946d32aa0f5db5c0082b7    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 14:32:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 14:32:21 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the db module.  

M src/db/db.c
M src/db/db.h
M test/src/module/db/dbTest.c

Refactor info modules with inline getters/setters.

commit   : 442b2e41b1561a5d7f5e06097116279ee173e626    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 13:48:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Apr 2021 13:48:40 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the info modules.  

M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoPg.c
M src/info/infoPg.h
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/manifestTest.c

Refactor protocol modules with inline getters/setters.

commit   : 4937653a3df1948ec8c5d9f62075e0f287852ce8    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Apr 2021 16:46:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Apr 2021 16:46:42 -0400    

Click here for diff

Extend the pattern introduced in 79a2d02c to the protocol modules.  

M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.h
M src/protocol/helper.h
M src/protocol/parallel.h
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M src/protocol/server.c
M src/protocol/server.h
M test/src/module/protocol/protocolTest.c

Add const keyword to module scope constants.

commit   : 235e32b57db36ca43bc47c5028ad9b619c223a68    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Apr 2021 14:47:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Apr 2021 14:47:53 -0400    

Click here for diff

These pointers should never be modified, so mark them const.  

M src/common/io/http/common.c

Add Ubuntu 20.04 test VM definition.

commit   : 6c005afd5bcd7260976908dd2ddb2cdb9031996a    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Apr 2021 14:38:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Apr 2021 14:38:20 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/VmTest.pm

Replace OBJECT_DEFINE_MOVE() and OBJECT_DEFINE_FREE() with inlines.

commit   : d30ec9c9aeaaa454078b67a5998614f0b4c9c81d    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Apr 2021 10:04:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Apr 2021 10:04:57 -0400    

Click here for diff

Inline functions are more efficient and if they are not used are automatically omitted from the binary.  
  
This also makes the implementation of these functions easier to find and removes the need for a declaration. That is, the complete implementation is located in the header rather than being spread between the header and C file.  

M src/Makefile.in
M src/command/backup/pageChecksum.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/exec.c
M src/common/exec.h
M src/common/ini.c
M src/common/ini.h
M src/common/io/client.c
M src/common/io/client.h
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/http/header.c
M src/common/io/http/header.h
M src/common/io/http/query.c
M src/common/io/http/query.h
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/common/io/http/response.c
M src/common/io/http/response.h
M src/common/io/http/session.c
M src/common/io/http/session.h
M src/common/io/http/url.c
M src/common/io/http/url.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/session.c
M src/common/io/session.h
M src/common/io/socket/client.c
M src/common/io/write.c
M src/common/io/write.h
M src/common/regExp.c
M src/common/regExp.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/keyValue.c
M src/common/type/keyValue.h
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/mcv.c
M src/common/type/mcv.h
A src/common/type/object.c
M src/common/type/object.h
M src/common/type/pack.c
M src/common/type/pack.h
M src/common/type/xml.c
M src/common/type/xml.h
M src/common/wait.c
M src/common/wait.h
M src/db/db.c
M src/db/db.h
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoPg.h
M src/info/manifest.c
M src/info/manifest.h
M src/postgres/client.c
M src/postgres/client.h
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.c
M src/protocol/command.h
M src/protocol/parallel.c
M src/protocol/parallel.h
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M src/protocol/server.c
M src/protocol/server.h
M src/storage/azure/read.c
M src/storage/gcs/read.c
M src/storage/read.c
M src/storage/read.h
M src/storage/storage.c
M src/storage/storage.h
M src/storage/write.c
M src/storage/write.h
M test/define.yaml
M test/src/module/common/typeListTest.c
M test/src/module/common/typeObjectTest.c
M test/src/module/storage/posixTest.c

Replace OBJECT_DEFINE_FREE_RESOURCE_BEGIN() with normal functions.

commit   : 351e7db4c4df6329f87ceb63b7c0fb253bed1463    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Apr 2021 16:27:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Apr 2021 16:27:55 -0400    

Click here for diff

OBJECT_DEFINE_MOVE() and OBJECT_DEFINE_FREE() will be replaced with inlines so this would be the only macro left that is constructing functions.  
  
It is not a great pattern anyway since it makes it hard to find the function implementation.  

M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/compress/zst/compress.c
M src/common/compress/zst/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/exec.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/session.c
M src/common/regExp.c
M src/common/type/object.h
M src/common/type/xml.c
M src/db/db.c
M src/postgres/client.c
M src/protocol/client.c
M src/storage/posix/read.c
M src/storage/posix/write.c
M src/storage/remote/write.c
M test/src/module/common/typeObjectTest.c

Replace OBJECT_DEFINE_GET() with *Pub struct pattern.

commit   : cc85c4f03d78492cf8158cf02c51e8eb8bab24e6    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Apr 2021 14:27:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Apr 2021 14:27:57 -0400    

Click here for diff

This macro was originally intended to simplify the creation of simple getters but it has been superseded by the pattern introduced in 79a2d02c.  
  
Remove instances of OBJECT_DEFINE_GET() to avoid confusion with the new pattern.  

M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/common/io/http/response.c
M src/common/io/http/response.h
M src/common/type/object.h
M src/common/wait.c
M src/common/wait.h
M test/src/module/common/ioHttpTest.c

Refactor storage modules with inline getters/setters.

commit   : b715c70b464d4bb2eaa347b6b83902040caa8cd8    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Apr 2021 14:04:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Apr 2021 14:04:38 -0400    

Click here for diff

Extended the pattern introduced in 79a2d02c to the storage modules: Storage, StorageRead, StorageWrite.  

M src/storage/azure/storage.h
M src/storage/cifs/storage.h
M src/storage/gcs/storage.h
M src/storage/posix/storage.h
M src/storage/read.c
M src/storage/read.h
M src/storage/read.intern.h
M src/storage/remote/storage.h
M src/storage/s3/storage.h
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M src/storage/write.c
M src/storage/write.h
M src/storage/write.intern.h
M test/src/module/command/backupTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Refactor List, StringList, and VariantList for performance.

commit   : 79a2d02c9c8c00a2db6c34ddf73223f159288c56    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Apr 2021 12:50:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Apr 2021 12:50:33 -0400    

Click here for diff

Introduce a standard pattern for exposing public struct members (as documented in CODING.md) and use it to inline lstSize() which should improve the performance of iterating large lists.  
  
Since many functions in these modules are just thin wrappers of other functions, inline where appropriate.  
  
Remove strLstExistsZ() and strLstInsertZ() since they were only used in tests, where the String version of the function is sufficient.  
  
Move strLstNewSplitSizeZ() to command/help/help.c and remove strLstNewSplitSize(). This function has only ever been used by help and does not seem widely applicable.  

M CODING.md
M doc/xml/coding.xml
M src/command/help/help.c
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/common/type/variantList.c
M src/common/type/variantList.h
M test/define.yaml
M test/src/common/harnessConfig.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/info/infoBackupTest.c

Update config.sub to latest version.

commit   : 904738a5f180ee3040fd7626c894b134dbaf164a    
  
author   : David Steele <[email protected]>    
date     : Mon, 5 Apr 2021 10:25:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 5 Apr 2021 10:25:01 -0400    

Click here for diff

M src/build/config.sub

Begin v2.34 development.

commit   : 8461961e5c832ffd8e255b76ccd0125e0cd83e00    
  
author   : David Steele <[email protected]>    
date     : Mon, 5 Apr 2021 10:23:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 5 Apr 2021 10:23:16 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h