pgBackRest 2.38 commit log

v2.37: TLS Server

commit   : f18f2d9991da29533ad8e9859074c4f39185420f    
  
author   : David Steele <[email protected]>    
date     : Mon, 3 Jan 2022 08:43:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 3 Jan 2022 08:43:55 -0500    

Click here for diff

Bug Fixes:  
  
* Fix restore delta link mapping when path/file already exists. (Reviewed by Reid Thompson. Reported by Younes Alhroub.)  
* Fix socket leak on connection retries. (Reviewed by Reid Thompson. Reported by James Coleman.)  
  
Features:  
  
* Add TLS server. (Reviewed by Stephen Frost, Reid Thompson, Andrew L'Ecuyer.)  
* Add --cmd option. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele. Suggested by Virgile CREVON.)  
  
Improvements:  
  
* Check archive immediately after backup start. (Reviewed by Reid Thompson, David Christensen.)  
* Add timeline and checkpoint checks to backup. (Reviewed by Stefan Fercot, Reid Thompson.)  
* Check that clusters are alive and correctly configured during a backup. (Reviewed by Stefan Fercot.)  
* Error when restore is unable to find a backup to match the time target. (Reviewed by Reid Thompson, Douglas J Hunley. Suggested by Douglas J Hunley.)  
* Parse protocol/port in S3/Azure endpoints. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Add warning when checkpoint_timeout exceeds db-timeout. (Contributed by Stefan Fercot. Reviewed by David Steele.)  
* Add verb to HTTP error output. (Contributed by Christoph Berg. Reviewed by David Steele.)  
* Allow y/n arguments for boolean command-line options. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Make backup size logging exactly match info command output. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Mahomed Hussein.)  
  
Documentation Improvements:  
  
* Display size option default and allowed values with appropriate units. (Reviewed by Reid Thompson.)  
* Fix typos and improve documentation for the tablespace-map-all option. (Reviewed by Reid Thompson. Suggested by Reid Thompson.)  
* Remove obsolete statement about future multi-repository support. (Suggested by David Christensen.)  

M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

commit   : 62fbee72ad319f92c9410ac8dbab2f81fe945a36    
  
author   : David Steele <[email protected]>    
date     : Sat, 1 Jan 2022 10:50:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 1 Jan 2022 10:50:16 -0500    

Click here for diff

M LICENSE
M src/common/macro.h
M src/postgres/interface/pageChecksum.vendor.c
M src/postgres/interface/static.vendor.h
M src/postgres/interface/version.vendor.h

Remove dead test code.

commit   : d6ebf6e2d67331a838f53beda1c186c527b56a8e    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Dec 2021 18:54:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Dec 2021 18:54:36 -0500    

Click here for diff

M test/src/module/command/backupTest.c

Add release note regarding IANA approval of the default TLS port.

commit   : fccb7f7dd45c6c373d0cfa74b90d69ca483aa3af    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Dec 2021 17:39:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Dec 2021 17:39:22 -0500    

Click here for diff

M doc/xml/release.xml

Parse protocol/port in S3/Azure endpoints.

commit   : 6a12458440168f13cb05d70f36ea54b1860e390c    
  
author   : Reid Thompson <[email protected]>    
date     : Thu, 16 Dec 2021 10:30:59 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 16 Dec 2021 10:30:59 -0500    

Click here for diff

Utilize httpUrlNewParseP() to parse endpoint and port from the URL in the S3 and Azure helpers to avoid issues where protocol was not expected to be part of the URL.

M doc/xml/release.xml
M src/storage/azure/helper.c
M src/storage/s3/helper.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Add TLS server documentation.

commit   : f06101de77a980c7e4115762f2fc301280aa4127    
  
author   : David Steele <[email protected]>    
date     : Thu, 16 Dec 2021 09:47:04 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 16 Dec 2021 09:47:04 -0500    

Click here for diff

Add documentation and make the feature visible.  

M README.md
M doc/xml/index.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M test/src/module/command/helpTest.c

Fix socket leak on connection retries.

commit   : 615bdff4030a31bfedfe7df04676e3948ec9c2c0    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Dec 2021 14:53:41 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 14 Dec 2021 14:53:41 -0500    

Click here for diff

This leak was caused by the file descriptor variable getting clobbered after a long jump. Mark it as volatile to fix.
  

  
Testing this is a bit complex because the issue only happens in optimized builds, if at all. Put the test into the performance suite, which is always optimized, until a better idea presents itself.

M doc/xml/release.xml
M src/common/io/socket/client.c
M test/define.yaml
M test/src/module/performance/typeTest.c

commit   : a73fe4eb966f9685f6e4179c397a10c1e7f15f19    
  
author   : David Steele <[email protected]>    
date     : Fri, 10 Dec 2021 15:53:40 -0500    
  
committer: GitHub <[email protected]>    
date     : Fri, 10 Dec 2021 15:53:40 -0500    

Click here for diff

If a path/file was remapped to a link using either --link-map or --link-all there would be no affect if the path/file already existed. If a link existed it would be properly updated and converting a link to a path/file also worked.
  

  
The issue happened during delta cleanup, which failed to check if the existing path/file had been remapped to a link.
  

  
Add checks for newly mapped path/file links and remove the old path/file we required.

M doc/xml/release.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Close expect log file when unit test completes.

commit   : 19a7ec69debfe6587fcc1163451896590c96bf21    
  
author   : David Steele <[email protected]>    
date     : Fri, 10 Dec 2021 15:04:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 10 Dec 2021 15:04:55 -0500    

Click here for diff

This did not cause any issues, but it is better to explicitly close open files.  

M test/src/common/harnessLog.c

Add verb to HTTP error output.

commit   : c38e2d31709804eb4b9125a15ad84c8fc813f366    
  
author   : Christoph Berg <[email protected]>    
date     : Wed, 8 Dec 2021 15:00:19 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 8 Dec 2021 15:00:19 -0500    

Click here for diff

This makes it easier to debug HTTP errors.  

M doc/xml/release.xml
M src/common/io/http/request.c
M test/src/module/common/ioHttpTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/s3Test.c

Error when restore is unable to find a backup to match the time target.

commit   : be4ac3923cb77873da298a30aca5d847b3c635af    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Dec 2021 13:57:26 -0500    
  
committer: GitHub <[email protected]>    
date     : Wed, 8 Dec 2021 13:57:26 -0500    

Click here for diff

This was previously a warning but the warning is easy to miss so a lot of time may be lost restoring and recovering a backup that will not hit the target.
  

  
Since this is technically a breaking change, add an "important note" about the change to the release.

M doc/xml/release.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Add warning when checkpoint_timeout exceeds db-timeout.

commit   : 672330593789f07aaad90bbafcd2597cbc602686    
  
author   : Stefan Fercot <[email protected]>    
date     : Wed, 8 Dec 2021 18:29:20 +0100    
  
committer: GitHub <[email protected]>    
date     : Wed, 8 Dec 2021 18:29:20 +0100    

Click here for diff

In the backup command, add a warning if start-fast is disabled and the PostgreSQL checkpoint_timeout is greater than db-timeout.
  

  
In such cases, we might timeout before the checkpoint occurs and the backup really starts.

M doc/xml/release.xml
M src/db/db.c
M src/db/db.h
M test/src/common/harnessPq.h
M test/src/module/db/dbTest.c

Check that clusters are alive and correctly configured during a backup.

commit   : bd2ba802db11c505ec69943fa81b2b379073fbf4    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Dec 2021 10:16:41 -0500    
  
committer: GitHub <[email protected]>    
date     : Wed, 8 Dec 2021 10:16:41 -0500    

Click here for diff

Fail the backup if a cluster stops or the standby is promoted. Previously, shutting down the primary would cause an error but it was not detected until the end of the backup. Now the error will happen sooner and a promotion on the standby will also cause an error.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/db/db.c
M src/db/db.h
M test/src/module/command/backupTest.c
M test/src/module/db/dbTest.c

Add SIGTERM and SIGHUP handling to TLS server.

commit   : 7b3ea883c7c010aafbeb14d150d073a113b703e4    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Dec 2021 18:18:43 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 7 Dec 2021 18:18:43 -0500    

Click here for diff

SIGHUP allows the configuration to be reloaded. Note that the configuration will not be updated in child processes that have already started.
  

  
SIGTERM terminates the server process gracefully and sends SIGTERM to all child processes. This also gives the tests an easy way to stop the server.

M doc/xml/release.xml
M src/command/server/server.c
M src/command/server/server.h
M src/common/io/socket/server.c
M src/main.c
M test/src/module/command/serverTest.c
M test/src/module/common/ioTlsTest.c

Add timeline and checkpoint checks to backup.

commit   : 49145d72bac16498cdbf5eeb3cd6128ea0be0667    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Dec 2021 09:21:07 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 7 Dec 2021 09:21:07 -0500    

Click here for diff

Add the following checks:
  

  
* Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
  
* Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
  
* Standby is on the same timeline as the primary. If not, this standby is not following the primary.
  
* Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.
  

  
This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.
  

  
The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/stanza/common.c
M src/db/db.c
M src/db/db.h
M src/db/helper.c
M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/interface/version.intern.h
M test/src/common/harnessPostgres.c
M test/src/common/harnessPostgres/harnessVersion.intern.h
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/db/dbTest.c
M test/src/module/postgres/interfaceTest.c

Add error type and message to CHECK() macro.

commit   : 9c76056dd0d1d2b07a89646b087c5c8d36ab97f5    
  
author   : David Steele <[email protected]>    
date     : Tue, 30 Nov 2021 16:21:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 30 Nov 2021 16:21:15 -0500    

Click here for diff

A CHECK() worked exactly like ASSERT() except that it was compiled into production code. However, over time many checks have been added that should not throw AssertError, which should be reserved for probable coding errors.  
  
Allow the error code to be specified so other error types can be thrown. Also add a human-readable message since many of these could be seen by users even when there is no coding error.  
  
Update coverage exceptions for CHECK() to match ASSERT() since all conditions will never be covered.  

M src/build/common/yaml.c
M src/build/config/main.c
M src/build/config/parse.c
M src/build/config/render.c
M src/build/error/main.c
M src/build/help/main.c
M src/build/help/parse.c
M src/build/help/render.c
M src/command/archive/common.c
M src/command/backup/backup.c
M src/command/help/help.c
M src/command/remote/remote.c
M src/command/restore/restore.c
M src/command/server/server.c
M src/common/assert.h
M src/common/io/filter/filter.c
M src/common/io/tls/common.c
M src/common/io/tls/session.c
M src/common/log.c
M src/common/type/pack.c
M src/common/type/string.c
M src/common/type/stringId.c
M src/config/protocol.c
M src/db/db.c
M src/info/info.c
M src/info/infoPg.c
M src/info/manifest.c
M src/postgres/client.c
M src/postgres/interface.c
M src/protocol/client.c
M src/protocol/helper.c
M src/protocol/server.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/helper.c
M src/storage/s3/storage.c
M src/storage/storage.c
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/src/module/common/logTest.c
M test/src/module/storage/posixTest.c

Add HRN_PG_CONTROL_PUT() and HRN_PG_CONTROL_TIME().

commit   : 0895cfcdf7d3f15b8029f73ed62c6094d30724b3    
  
author   : David Steele <[email protected]>    
date     : Tue, 30 Nov 2021 13:23:11 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 30 Nov 2021 13:23:11 -0500    

Click here for diff

These macros simplify management of pg_control test files.  
  
Centralize time updates for pg_control in the command/backup module. This caused some time updates in the logs.  
  
Finally, move the postgres module after the storage module so it can use storage macros.  

M test/define.yaml
M test/src/common/harnessPostgres.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/postgres/interfaceTest.c

Autogenerate test system identifiers.

commit   : 01ac6b6cac86ea857e54a3b1c45077df1e128a75    
  
author   : David Steele <[email protected]>    
date     : Tue, 30 Nov 2021 08:28:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 30 Nov 2021 08:28:36 -0500    

Click here for diff

hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL.  
  
Add constants to represent version system ids in tests. These will eventually be auto-generated.  
  
This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.  

M test/src/common/harnessPostgres.c
M test/src/common/harnessPostgres.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/info/infoPgTest.c

Ensure ASSERT() macro is always available in test modules.

commit   : 3f7409019df112ec50efb6c3db6f7780c9a63c87    
  
author   : David Steele <[email protected]>    
date     : Wed, 24 Nov 2021 16:09:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 24 Nov 2021 16:09:45 -0500    

Click here for diff

Tests that run without DEBUG for performance did not have ASSERT() and were using CHECK() instead.  
  
Instead ensure that the ASSERT() macro is always available in tests.  

M test/define.yaml
M test/src/common/harnessLog.c
M test/src/common/harnessTest.h
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c

Revert changes to backupFilePut() made in 1e77fc3d.

commit   : dcb4f09d8315e92c0877b589f3fa9b7f0fa65f93    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 23 Nov 2021 09:37:12 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Nov 2021 09:37:12 -0500    

Click here for diff

These changes were made obsolete by a3d7a23a.  

M doc/xml/release.xml
M src/command/backup/backup.c

Use ASSERT() or TEST_RESULT*() instead of CHECK() in test modules.

commit   : 7e35245dc3416238a84a43abbecdf976170dea91    
  
author   : David Steele <[email protected]>    
date     : Tue, 23 Nov 2021 08:07:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Nov 2021 08:07:31 -0500    

Click here for diff

M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/typeListTest.c
M test/src/module/protocol/protocolTest.c

Use infoBackupDataByLabel() to log backup size.

commit   : a3d7a23a9d90611a3d31947598fbea240b250710    
  
author   : Reid Thompson <[email protected]>    
date     : Mon, 22 Nov 2021 12:52:37 -0500    
  
committer: GitHub <[email protected]>    
date     : Mon, 22 Nov 2021 12:52:37 -0500    

Click here for diff

Eliminate summing and passing of copied files sizes for logging backup size.
  

  
Instead, utilize infoBackupDataByLabel() to pull the backup size for the log message.

M doc/xml/release.xml
M src/command/backup/backup.c
M test/src/module/command/backupTest.c

Allow y/n arguments for boolean command-line options.

commit   : 1a0560d363d28737befb8c222647783d4fc2ca29    
  
author   : Reid Thompson <[email protected]>    
date     : Fri, 19 Nov 2021 12:22:09 -0500    
  
committer: GitHub <[email protected]>    
date     : Fri, 19 Nov 2021 12:22:09 -0500    

Click here for diff

This allows boolean boolean command-line options to work like their config file equivalents.
  

  
At least for now this behavior will remain undocumented since all examples in the documentation will continue to use the standard syntax. The idea is that it will "just work" when options are copied out of config files rather than generating an error.

M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c

Rename server-start command to server.

commit   : 2d963ce9471808172f879916c3f3accc35f14d56    
  
author   : David Steele <[email protected]>    
date     : Thu, 18 Nov 2021 17:23:11 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 18 Nov 2021 17:23:11 -0500    

Click here for diff

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/config/config.auto.h
M src/config/parse.auto.c
M src/main.c
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/src/module/command/serverTest.c
M test/src/module/protocol/protocolTest.c

Check archive immediately after backup start.

commit   : 1f14f45dfb0d1677a695719381cbd5a8a3c6c986    
  
author   : David Steele <[email protected]>    
date     : Thu, 18 Nov 2021 16:18:10 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 18 Nov 2021 16:18:10 -0500    

Click here for diff

Previously the archive was only checked at the end of the backup to ensure all WAL required to make the backup consistent was present. The problem was that if archiving was not functioning then the backup had to complete before the user found out, which could be a while if the database was large enough.
  

  
Add an archive check immediately after backup start so failures are reported earlier.
  

  
The trick is to determine which WAL to check. If the repo is new there may not be any WAL in it and pg_start_backup() will not switch the WAL segment if it is empty. These are both likely scenarios when setting up and/or testing pgBackRest.
  

  
If the WAL segment is switched by pg_start_backup(), then check the archive for the segment that was detected prior to backup start. This should be common on normal running clusters with regular activity. Note that this might not be the segment immediately prior to the backup start segment if WAL volume is high.
  

  
If pg_start_backup() did not switch the WAL then we can force a switch on PostgreSQL >= 9.3 by creating a restore point. In that case the WAL to check will be the backup start WAL. This is most likely to happen on idle systems, during testing, or immediately after a repo switch.
  

  
An advantage of this approach other than earlier notification is that the backup directory will not be created so no resume will be attempted on the next backup.
  

  
Note that some additional churn was created in backup.c because the load of archive.info needs to be done earlier.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/db/db.c
M src/db/db.h
M test/src/common/harnessPq.h
M test/src/module/command/backupTest.c
M test/src/module/db/dbTest.c

Remove obsolete statement about future multi-repository support.

commit   : dea752477ab8e812cdbd717eb2091baf3f5d0906    
  
author   : David Steele <[email protected]>    
date     : Wed, 17 Nov 2021 16:39:04 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 17 Nov 2021 16:39:04 -0500    

Click here for diff

M doc/xml/release.xml
M doc/xml/user-guide.xml

Add linefeed and remove space.

commit   : 0949b4d35fdd04c55927eb6a107d881376dbe73c    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Nov 2021 18:26:21 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 16 Nov 2021 18:26:21 -0500    

Click here for diff

M src/command/backup/backup.c
M src/db/db.c

Add infoBackupLabelExists().

commit   : 809f0bbc638cdd95540e2257383147919f82e8f9    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Nov 2021 11:34:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 16 Nov 2021 11:34:53 -0500    

Click here for diff

This is easier to read than using infoBackupDataByLabel() != NULL.  
  
It also allows an assertion to be added to infoBackupDataByLabel() to ensure that a NULL return value is not used unsafely.  

M src/command/expire/expire.c
M src/command/verify/verify.c
M src/info/infoBackup.c
M src/info/infoBackup.h
M test/src/module/info/infoBackupTest.c

Include backup_label and tablespace_map file sizes in log output.

commit   : 1e77fc3d75490b7a1b6a0b31be9298c995ec672f    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 16 Nov 2021 10:21:32 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 16 Nov 2021 10:21:32 -0500    

Click here for diff

In cases where they are returned by postgres, include backup_label and tablespace_map file sizes in the backup size value output in the log.

M doc/xml/release.xml
M src/command/backup/backup.c

Add findutils package to RHEL 8 documentation container.

commit   : 6b5322cdad7163d91b43d37d9d8eeaa39ac7f214    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Nov 2021 09:27:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 16 Nov 2021 09:27:15 -0500    

Click here for diff

This package was dropped from the most recent Rocky Linux 8 image.  

M doc/xml/user-guide.xml

Fix typos and improve documentation for the tablespace-map-all option.

commit   : df89eff429e9b8fbc68d9e9895badf9719fd31d2    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 Nov 2021 16:53:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Nov 2021 16:53:41 -0500    

Click here for diff

M doc/xml/release.xml
M src/build/help/help.xml

Fix parameter test logging in parseOptionIdxValue().

commit   : fcae9d35038d454c674921c65beb02b195981480    
  
author   : Reid Thompson <[email protected]>    
date     : Mon, 15 Nov 2021 16:42:46 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Nov 2021 16:42:46 -0500    

Click here for diff

M src/config/parse.c

Add tablespace_map file to command/backup test module.

commit   : b3a5f7a8e27768c445458e47dad626609814fbb7    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 Nov 2021 14:32:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Nov 2021 14:32:22 -0500    

Click here for diff

The code worked fine but better to have explicit tests for this file.  

M test/src/module/command/backupTest.c

Add path to pgbench used for stress test in user guide.

commit   : e62ba8e85eaf469052960c4fd71ffaf26c1a1baa    
  
author   : David Steele <[email protected]>    
date     : Fri, 12 Nov 2021 17:15:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 12 Nov 2021 17:15:45 -0500    

Click here for diff

This allows the stress test to run on RHEL.  

M doc/xml/user-guide.xml

Revive archive performance test.

commit   : 43cfa9cef776360e592882c0b787704dbeb36cb3    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Nov 2021 12:14:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Nov 2021 12:14:41 -0500    

Click here for diff

This test was lost due to a syntax issue in a58635ac.  
  
Update the test to use system() to better mimic what postgres does and add logging so pgBackRest timing can be determined.  

M test/define.yaml
M test/lib/pgBackRestTest/Module/Performance/PerformanceArchivePerlTest.pm

Refactor postgres/client module with inline getters/setters.

commit   : dd96c29f963609fad38dac3349d7fa41e40722bb    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Nov 2021 07:53:46 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Nov 2021 07:53:46 -0500    

Click here for diff

Extend the pattern introduced in 79a2d02 to the postgres/client module.  

M src/postgres/client.c
M src/postgres/client.h

Update contributor for 6e635764.

commit   : afe77e76e0adf948138d797e227a6f4c7d47c2eb    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Nov 2021 07:31:02 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Nov 2021 07:31:02 -0500    

Click here for diff

M doc/xml/release.xml

Match backup log size with size reported by info command.

commit   : 6e635764a66278d5a8c2b4d30b23063bc3923067    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 9 Nov 2021 13:24:56 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 9 Nov 2021 13:24:56 -0500    

Click here for diff

Properly log the size of files copied during the backup, matching the backup size returned from the info command.
  

  
In the reference issue, the incremental backup after switchover logs the size of all files evaluated rather than only the size of the files copied in the backup.

M doc/xml/release.xml
M src/command/backup/backup.c
M test/expect/mock-all-001.log
M test/src/module/command/backupTest.c

Do not delete manifests individually during stanza delete.

commit   : d05d6b87142347cb4891304833db389dcf7f9a81    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Nov 2021 09:39:58 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Nov 2021 09:39:58 -0500    

Click here for diff

This appears to have been an attempt to not delete files that we don't recognize, but it only works in narrow cases and could leave the user is a position of not being able to complete the stanza delete without manual intervention. It seems better just to proceed with the delete, especially since the info files have already been removed.  
  
In addition, deleting the manifests individually could be slow on object stores if there were a very large number of backups.  

M src/command/stanza/delete.c
M test/src/module/command/stanzaTest.c

Refactor switch statements in strIdBitFromZN().

commit   : bb03b3f41942d0b781931092a76877ad309001ef    
  
author   : David Steele <[email protected]>    
date     : Thu, 4 Nov 2021 09:44:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 4 Nov 2021 09:44:31 -0400    

Click here for diff

Coverity does not like fall-throughs either to or from the default case so refactor to avoid that.  

M src/common/type/stringId.c

Optional parameters for tlsClientNew().

commit   : 676b9d95dd2467d4bddd402b5cd2b4f445c71944    
  
author   : David Steele <[email protected]>    
date     : Thu, 4 Nov 2021 08:19:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 4 Nov 2021 08:19:18 -0400    

Click here for diff

There are a number of optional parameters with the same type so this makes them easier to track and reduces churn when new ones are added.  

M src/command/server/ping.c
M src/common/io/client.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M src/protocol/helper.c
M src/storage/azure/storage.c
M src/storage/gcs/storage.c
M src/storage/s3/storage.c
M test/src/module/common/ioTlsTest.c
M test/src/module/protocol/protocolTest.c

Display size option default and allowed values with appropriate units.

commit   : 038abaa71d816cc87b382bd81d3df62ddec9455a    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 Nov 2021 15:23:08 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 3 Nov 2021 15:23:08 -0400    

Click here for diff

Size option default and allowed values were displayed in bytes, which was confusing for the user.
  

  
This also lays the groundwork for adding units to time options.
  

  
Move option parsing functions into a common module so they can be used from the build module.

M doc/xml/release.xml
M src/Makefile.in
M src/build/config/config.yaml
M src/build/config/render.c
M src/build/help/help.xml
M src/command/help/help.c
A src/config/common.c
A src/config/common.h
M src/config/exec.c
M src/config/parse.auto.c
M src/config/parse.c
M test/define.yaml
M test/src/module/command/helpTest.c
M test/src/module/config/execTest.c
M test/src/module/config/parseTest.c

Use void * instead of List * to avoid Coverity false positives.

commit   : 1b93a772369bbb3a936099e0d9d5cc79bad1e0f6    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 Nov 2021 12:14:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 Nov 2021 12:14:17 -0400    

Click here for diff

Coverity complains that this should be "List" but that is clearly not correct.  

M src/command/backup/backup.c
M src/command/restore/restore.c

Add --cmd option.

commit   : 2a576477b316238473525e56bc8fc8ea5790455f    
  
author   : Reid Thompson <[email protected]>    
date     : Wed, 3 Nov 2021 11:36:34 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 3 Nov 2021 11:36:34 -0400    

Click here for diff

Allows users to provide an executable to be used when pgbackrest generates command strings that expect to invoke pgbackrest. These generated commands are written to files by pgbackrest, e.g. recovery.conf.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/restore/restore.c
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c
M test/src/module/command/helpTest.c
M test/src/module/command/restoreTest.c
M test/src/module/config/parseTest.c

Simplify error handler.

commit   : c5b5b5880619d0994ab4a8feb3f60ab52170b61b    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 Nov 2021 10:36:31 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 3 Nov 2021 10:36:31 -0400    

Click here for diff

The error handler used a loop to process try, catch, and finally blocks. This worked fine but static analysis tools like Coverity did not understand that the finally block would always run and so there were false positives about double-free, unfreed resource, etc.
  

  
This implementation removes the loop, which simplifies everything, and makes it clear that the finally block will always run. This cuts down on Coverity false positives.
  

  
This implementation also catches lack of coverage on empty catch blocks so a few test fixes were committed separately in d74fe7a.
  

  
A small refactor in backup.c is required because gcc 10.3.1 on Fedora 33 complains that the reason variable may be used uninitialized. It's not clear why this is the case, but reducing the scope of the TRY block fixes the issue.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/common/error.c
M src/common/error.h
M test/src/module/common/errorTest.c

Centralize logic to build value lists during config rendering.

commit   : cff961ede7e41fa8035ffe7451a22eb5ea0e46c1    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 Nov 2021 07:38:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 Nov 2021 07:38:06 -0400    

Click here for diff

This reduces duplication and makes it easier to add new types.  

M src/build/config/render.c

Add StringId as an option type.

commit   : 7f6c513be925c77bc6a177408efcf79f624ffc94    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 Nov 2021 07:27:26 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 3 Nov 2021 07:27:26 -0400    

Click here for diff

Rather the converting String to StringIds at runtime, store defaults in StringId format in parse.auto.c and convert user input to StringId during parsing.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/config/parse.h
M src/build/config/render.c
M src/command/backup/backup.c
M src/config/config.c
M src/config/config.intern.h
M src/config/load.c
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M src/protocol/helper.c
M test/src/module/build/configTest.c
M test/src/module/command/repoTest.c
M test/src/module/config/parseTest.c

Use cfgOptionStrId() instead of cfgOptionStr() where appropriate.

commit   : b13844086d419dc3070bcce4e918b2353bf4887c    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 17:35:19 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 1 Nov 2021 17:35:19 -0400    

Click here for diff

The compress-type, repo-type and log-level-* options have allow lists, which means it is more efficient to treat them as StringIds.
  

  
For compress-type and log-level-* also update the functions that convert them to enums.

M doc/xml/release.xml
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/restore/restore.c
M src/common/compress/helper.c
M src/common/compress/helper.h
M src/common/log.c
M src/common/log.h
M src/config/load.c
M src/info/manifest.c
M test/src/module/common/compressTest.c
M test/src/module/common/logTest.c
M test/src/module/config/loadTest.c

Remove placeholder bz2 helper data.

commit   : b237d0cd592bbc6c6ee9280fb7aed264bf79eb9d    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 10:43:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 10:43:08 -0400    

Click here for diff

This placeholder data should have been removed when bz2 support was added in a021c9fe053.  

M src/common/compress/helper.c

Remove unused protocol log level.

commit   : f4e281399a81835821547ea5c78ed7a189914d3d    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 10:27:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 10:27:57 -0400    

Click here for diff

This log level was used in the Perl code but was never ported to C.  

M src/common/log.c
M src/common/logLevel.h

Simplify strIdFrom*() functions.

commit   : bc352fa6a8cff7cc08b6c7f3cdfac664d2b0805f    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 10:08:56 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 1 Nov 2021 10:08:56 -0400    

Click here for diff

The strIdFrom*() forced the caller to pick an encoding, which led to a number of TRY...CATCH blocks in the code. In practice the caller does not care which encoding is used as long as the string is valid for some encoding.
  

  
Update the strIdFrom*() function to try all possible encodings and only throw an error when the string is not valid for any of them.

M doc/xml/release.xml
M src/build/common/render.c
M src/common/type/stringId.c
M src/common/type/stringId.h
M src/config/config.c
M src/config/parse.c
M src/info/infoBackup.c
M src/info/manifest.c
M test/src/module/command/backupTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/config/parseTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/protocol/protocolTest.c

Update automake version.

commit   : a92d7938197d1035e362390ce467ae827cbae051    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 09:11:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 09:11:43 -0400    

Click here for diff

There were no changes to install.sh in this version.  

M doc/RELEASE.md

Begin v2.37 development.

commit   : 904b897f5e89542784af64b364a49205e7a6e040    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 09:03:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 Nov 2021 09:03:42 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h