pgBackRest 2.32 commit log

v2.31: Minor Bug Fixes and Improvements

commit   : e116b535e6ad21248c00a6d0d948575ed06c6e38    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Dec 2020 09:55:00 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Dec 2020 09:55:00 -0500    

Click here for diff

Bug Fixes:  
  
* Allow [, #, and space as the first character in database names. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Jefferson Alexandre.)  
* Create standby.signal only on PostgreSQL 12 when restore type is standby. (Fixed by Stefan Fercot. Reviewed by David Steele. Reported by Keith Fiske.)  
  
Features:  
  
* Expire history files. (Contributed by Stefan Fercot. Reviewed by David Steele.)  
* Report page checksum errors in info command text output. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)  
* Add repo-azure-endpoint option. (Reviewed by Cynthia Shang, Brian Peterson. Suggested by Brian Peterson.)  
* Add pg-database option. (Reviewed by Cynthia Shang.)  
  
Improvements:  
  
* Improve info command output when a stanza is specified but missing. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele. Suggested by uspen.)  
* Improve performance of large file lists in backup/restore commands. (Reviewed by Cynthia Shang, Oscar.)  
* Add retries to PostgreSQL sleep when starting a backup. (Reviewed by Cynthia Shang. Suggested by Vitaliy Kukharik.)  
  
Documentation Improvements:  
  
* Replace RHEL/CentOS 6 documentation with RHEL/CentOS 8.  

M CONTRIBUTING.md
M README.md
M doc/release.pl
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Add RHEL/CentOS 8 documentation.

commit   : 31becf05b7445f201250d5956e3b725dd4b9338e    
  
author   : David Steele <[email protected]>    
date     : Fri, 4 Dec 2020 10:59:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 4 Dec 2020 10:59:57 -0500    

Click here for diff

Update RHEL/CentOS 7 to cover the versions that were previously covered by RHEL/CentOS 6.  
  
Since RHEL/CentOS 7/8 work the same update the documentation logic and labels to reflect this compatibility.  

M doc/README.md
M doc/release.pl
M doc/xml/documentation.xml
M doc/xml/release.xml
M doc/xml/user-guide-index.xml
M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Common/VmTest.pm

Add pg-database option.

commit   : b0ea3379651f610113cb63f0743b6245dd9d5ffe    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Dec 2020 22:42:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Dec 2020 22:42:50 -0500    

Click here for diff

In some rare cases there is no postgres database so this option may be used to specify an alternate database.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/parse.auto.c
M src/db/helper.c
M src/db/protocol.c
M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/db/dbTest.c

Fix incorrect usage of TEST_RESULT_DOUBLE().

commit   : 1913ecee639e0d40e0d7bfedb898f2d881ac49f0    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Dec 2020 22:42:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Dec 2020 22:42:05 -0500    

Click here for diff

These work fine but it's better to use the correct macro.  

M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeVariantTest.c

Add retries to PostgreSQL sleep when starting a backup.

commit   : d4211d3aaf6137f0a2e635c1ac9b089771356bfd    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Dec 2020 22:41:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Dec 2020 22:41:14 -0500    

Click here for diff

Inaccuracies in sleep time or clock skew might make a single sleep insufficient to reach the next second.  
  
Add a few retries to make the process more reliable but still avoid an infinite loop if something is seriously wrong.  

M doc/xml/release.xml
M src/command/backup/backup.c
M test/src/module/command/backupTest.c

Remove CentOS 6 from tests and documentation.

commit   : ec9f23d31f809902156466a0e4c96ef042d31edb    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Dec 2020 16:23:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Dec 2020 16:23:05 -0500    

Click here for diff

CentOS6 EOL'd and the mirrors were swiftly deleted, leading to failures in tests and documentation.  
  
Remove CentOS 6 for now to get builds going again with the intention to replace it in the near future with CentOS 8.  

M .travis.yml
M doc/RELEASE.md
M doc/release.pl
M doc/xml/documentation.xml
M doc/xml/user-guide-index.xml
M doc/xml/user-guide.xml
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/test.pl

Improve error message when PQgetCancel() returns NULL.

commit   : ffc50719d90832d1c24ca9b56ac36f76f5bfcaa6    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Dec 2020 15:15:35 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Dec 2020 15:15:35 -0500    

Click here for diff

There is not a lot to be done in this case since it looks like PostgreSQL disconnected while the query was running, but at least improve the error message and remove the assert, which indicates a coding error.  

M src/postgres/client.c
M test/src/common/harnessPq.c
M test/src/module/postgres/clientTest.c

Remove extraneous storageRepo() calls.

commit   : d1d25c710d736a0946d53164b814f1c914603bdd    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Nov 2020 18:03:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Nov 2020 18:03:24 -0500    

Click here for diff

These calls are not required since cipher info is passed explicitly. They are probably a copy-pasto from some past time when one of these functions required it.  

M src/command/archive/get/protocol.c
M src/command/restore/protocol.c
M src/command/verify/protocol.c

Report page checksum errors in info command text output.

commit   : 5488de8b6a77e8fcd635af707cbd604f985d1b9a    
  
author   : Stefan Fercot <[email protected]>    
date     : Wed, 25 Nov 2020 18:14:03 +0100    
  
committer: GitHub <[email protected]>    
date     : Wed, 25 Nov 2020 18:14:03 +0100    

Click here for diff

This feature currently only works for text output. JSON output is planned for the future.

M doc/xml/release.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c

Conform retry in lockAcquireFile() to the common retry pattern.

commit   : 3ed7b93b90eb61d40221701e0aef1318c1ea2b2b    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 24 Nov 2020 09:40:44 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 24 Nov 2020 09:40:44 -0500    

Click here for diff

M doc/xml/release.xml
M src/common/lock.c

Prepare configuration module for multi-repository support.

commit   : 117f03eba15378cfa15097828041331b67d16f12    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Nov 2020 15:55:46 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Nov 2020 15:55:46 -0500    

Click here for diff

Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum).  
  
Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage.  
  
Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption.  
  
Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy.  
  
Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn.  
  
Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related.  
  
Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.  

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/check/common.c
M src/command/command.c
M src/command/help/help.c
M src/common/type/stringz.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.h
A src/config/config.intern.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/define.c
M src/config/define.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M src/config/protocol.c
M src/db/helper.c
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/helper.c
M src/storage/helper.h
M test/define.yaml
M test/src/common/harnessConfig.c
M test/src/module/command/localTest.c
D test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Allow multiple remote locks from the same main process.

commit   : 7fda83b31e73467c881dca43cc6165e4a3ae3963    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Nov 2020 12:41:54 -0500    
  
committer: GitHub <[email protected]>    
date     : Mon, 23 Nov 2020 12:41:54 -0500    

Click here for diff

Improve locking on remote processes by introducing an exec-id that is unique to the main process and passed to all remote processes. This allows the remote processes to determine if a lock is held by a remote from the same main process. If so, the lock is allowed.
  

  
The exec-id is also useful for associating remote logs with main logs for debugging purposes.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/control/stop.c
M src/command/info/info.c
M src/command/remote/remote.c
M src/command/restore/restore.c
M src/common/lock.c
M src/common/lock.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/src/common/harnessConfig.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/protocol/protocolTest.c

Update Vagrant box version.

commit   : 77198dce97dd70fdf59ade2886b41e9029553217    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Nov 2020 16:26:30 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Nov 2020 16:26:30 -0500    

Click here for diff

M test/Vagrantfile

Create standby.signal only on PostgreSQL 12 when restore type is standby.

commit   : 191b8ec18ba6a2fceb55fb280c64f1a49767b646    
  
author   : Stefan Fercot <[email protected]>    
date     : Thu, 19 Nov 2020 22:57:19 +0100    
  
committer: GitHub <[email protected]>    
date     : Thu, 19 Nov 2020 22:57:19 +0100    

Click here for diff

When restore type standby is provided, the recovery.signal isn't needed and may lead to some confusion (see #1236).
  

  
Lately, when using pg_basebackup --write-recovery-conf, only the standby.signal file is created. This change would then align with that behaviour.

M doc/xml/release.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Move retrieval of archiveResult before while loop in verifyArchive().

commit   : 9aacd3c54a82ca27d6950300a34fbad8bd1ec45c    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 18 Nov 2020 18:19:49 -0500    
  
committer: GitHub <[email protected]>    
date     : Wed, 18 Nov 2020 18:19:49 -0500    

Click here for diff

The result structure for the archive id being processed only needs to be retrieved once so moving it outside of the WAL path list processing loop is more efficient.

M src/command/verify/verify.c

Rename hrnCfgEnvId*() to hrnCfgEnvKey*().

commit   : 62d9f2396127c305be2a2463cfe98ad872709403    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Nov 2020 10:48:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Nov 2020 10:48:40 -0500    

Click here for diff

The hrnCfg*() functions were renamed before commit but these got missed.  

M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h

Fix override of user option reset.

commit   : dd708e6d27ac7b5917704fd07b40bb6fe2a2eee7    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:37:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:37:05 -0500    

Click here for diff

If a user reset an option such as pg-default on the command-line then an override in the code would not take effect.  
  
Ignore a reset when the code explicitly sets an option to prevent this.  

M src/config/exec.c
M test/src/module/config/execTest.c

Fix test title in storage/remote test.

commit   : 6acd76b60ce9bacb7180da1661745f3eaa7d6674    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:28:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:28:53 -0500    

Click here for diff

M test/src/module/storage/remoteTest.c

Fix spacing and typos in backup/manifest modules.

commit   : 63c12d25418d6252da643d50020f17cc496f0a11    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:26:43 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:26:43 -0500    

Click here for diff

M src/info/manifest.c
M test/src/module/command/backupTest.c
M test/src/module/info/manifestTest.c

Remove obsolete call to storageRepo() in archivePushProtocol().

commit   : 3f7a66fffc1062027166056b9cad2459d160c613    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:19:37 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:19:37 -0500    

Click here for diff

This call to storageRepo() was used to fetch cipher options from a remote to determine if a repo cipher was enabled.  
  
Now the main process does this work and passes the cipher options directly to the local so there is no need to pre-load the repo storage here.  

M src/command/archive/push/protocol.c

Check archive push queue limit before checking repository.

commit   : cbb9b8fd2b8579ea1e9f129c2b79e4a7151a2757    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:16:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:16:41 -0500    

Click here for diff

If the push queue limit has been exceeded then nothing will be pushed to the repo so there is no point in checking it. Worse, a failure in the check would cause drop not to run and potentially fill up the disk, exactly the case this feature was designed to prevent.  
  
The async version already checks the push queue limit before checking the repository so now both versions have the same behavior.  

M src/command/archive/push/push.c

Log warnings in archive-push async log.

commit   : 8cf47f82e49cb3151cd48c8430a1a38c7b3ccaa0    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:10:59 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Nov 2020 16:10:59 -0500    

Click here for diff

These warnings were only being reported to PostgreSQL on the console. Now they are also recorded in the async log increasing the chance that they will be seen.  
  
This also improves coverage by requiring a warning during async processing to have a test case, which has been added.  

M src/command/archive/push/push.c
M test/src/module/command/archivePushTest.c

Use a constant to check restore target action.

commit   : d5d1ec6f6f4820808f30f4e9015bf22f2534de4e    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 Nov 2020 11:09:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 4 Nov 2020 11:09:05 -0500    

Click here for diff

Checking the default here was fragile. If the default were to change the code would break.  
  
This also removes the only dependency on cfgOptionDefault() outside of the help command.  

M src/command/restore/restore.c

Fix typos.

commit   : 021a34a55bef5370441c7a416514efebb7086177    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 29 Oct 2020 17:16:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 29 Oct 2020 17:16:15 -0400    

Click here for diff

M src/config/parse.c

Spell out three exclamation points in git-history.cache.

commit   : f87a117b71ce0d99137b6df5405fa1fd592af4b4    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Oct 2020 08:59:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Oct 2020 08:59:43 -0400    

Click here for diff

Three exclamation points are used by convention as a marker for code that needs attention before it can be committed to integration.  
  
If the markers are in this file they come up in every search.  

M doc/resource/git-history.cache

Use harnessCfgLoad() in config/load unit tests.

commit   : d25e965c475e0a36aad8e4a2df7e0d840185cded    
  
author   : David Steele <[email protected]>    
date     : Tue, 27 Oct 2020 13:29:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 27 Oct 2020 13:29:58 -0400    

Click here for diff

The tests were originally written by loading values directly into the configuration before the parser was available.  
  
Update to use harnessCfgLoad() to simplify the tests and make them compatible with upcoming config changes.  

M test/src/module/config/loadTest.c

Fix comment.

commit   : d91e6014bd1b6ff6f68abee7944250ed16ee254c    
  
author   : David Steele <[email protected]>    
date     : Tue, 27 Oct 2020 13:18:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 27 Oct 2020 13:18:25 -0400    

Click here for diff

M src/config/parse.c

Improve info command output when a stanza is specified but missing.

commit   : abe9d90c891fe61b0b0e7d1729dc89ca10b3b2e8    
  
author   : Stefan Fercot <[email protected]>    
date     : Tue, 27 Oct 2020 13:34:18 +0100    
  
committer: GitHub <[email protected]>    
date     : Tue, 27 Oct 2020 13:34:18 +0100    

Click here for diff

Return a path missing error when a stanza is specified for the info command but the stanza does not exist in the repository.
  

  
Previously [] was returned, which is still the case if no stanza is specified and the repository does not exist.

M doc/xml/release.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c

Improve performance of large file lists in backup/restore commands.

commit   : 770b65de804fc56a3e6a00d3916958c99774880d    
  
author   : David Steele <[email protected]>    
date     : Mon, 26 Oct 2020 12:18:45 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 26 Oct 2020 12:18:45 -0400    

Click here for diff

lstRemoveIdx(list, 0) resulted in the entire list being moved down to the first position which could take a long time for big lists. This is a common pattern in backup/restore when processing file queues.
  

  
Instead simply move the list pointer up when first item is removed. Then on insert check if there is space at the beginning when there is no longer space at the end and do the move then. This way if a list is built and then drained without any new inserts then no move is required.

M doc/xml/release.xml
M src/common/type/list.c
M test/define.yaml
M test/src/module/common/typeListTest.c
M test/src/module/performance/typeTest.c

Use zero-based indexes when referring to option indexes.

commit   : d452e9cc38001e2258cc05137508ab692d226df2    
  
author   : David Steele <[email protected]>    
date     : Mon, 26 Oct 2020 10:25:16 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 26 Oct 2020 10:25:16 -0400    

Click here for diff

There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa.
  

  
Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user.
  

  
Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/check/common.c
M src/command/check/common.h
M src/command/restore/restore.c
M src/command/stanza/common.c
M src/command/verify/verify.c
M src/db/helper.c
M src/db/helper.h
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/helper.c
M src/storage/helper.h
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/db/dbTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c

Remove unused FUNCTION_LOG_VERIFY_WAL_RANGE* defines.

commit   : ae35c4f0292e410c06e71eed6c5d8565017351d1    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 26 Oct 2020 07:41:08 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 26 Oct 2020 07:41:08 -0400    

Click here for diff

The defines for FUNCTION_LOG_VERIFY_WAL_RANGE* are not used in the current verify.c and are currently not planned in the continuing development of the verify command, so they are dead code and are therefore being removed.

M src/command/verify/verify.c

Replace misuse of bufSize() with bufUsed().

commit   : ea49151746a2d349ccb3c25e8e41df1e88d59a06    
  
author   : David Steele <[email protected]>    
date     : Sat, 24 Oct 2020 13:18:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 24 Oct 2020 13:18:02 -0400    

Click here for diff

bufSize() should only be used whem checking the total size of the buffer, not how much of it is currently used.  
  
In these cases bufUsed() and bufSize() are returning the same value but benign-looking code changes could break this assumption.  

M src/common/crypto/cipherBlock.c
M src/common/io/read.c
M src/common/type/xml.c
M src/storage/s3/storage.c

Clarify intended usage of harnessCfgLoadRaw().

commit   : 80a770ed2ea8d060d8ea77dabb239cf7e861774d    
  
author   : David Steele <[email protected]>    
date     : Sat, 24 Oct 2020 11:16:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 24 Oct 2020 11:16:06 -0400    

Click here for diff

M test/src/common/harnessConfig.h

Allow [, #, and space as the first character in database names.

commit   : 76cfd8ca703f4d6a71c6fe10b683209a2040368c    
  
author   : David Steele <[email protected]>    
date     : Sat, 24 Oct 2020 11:07:07 -0400    
  
committer: GitHub <[email protected]>    
date     : Sat, 24 Oct 2020 11:07:07 -0400    

Click here for diff

iniLoad() was trimming lines which meant that a leading space would not pass checksum validation when a manifest was reloaded. Remove the trims since files we write should never contain extraneous spaces. This further diverges the format for the functions that read conf files (e.g. pgbackrest.conf) and those that read info (e.g. manifest) files.
  

  
While we are at it also allow [ and # as initial characters. # was reserved for comments but we never put comments into info files. [ denotes a section but we can get around this by never allowing arrays as values in info files, so if a line ends in ] it must be a section. This is currently the case but enforce it by adding an assert to info/info.c.

M doc/xml/release.xml
M src/common/ini.c
M src/common/ini.h
M src/info/info.c
M test/src/module/command/backupTest.c
M test/src/module/common/iniTest.c
M test/src/module/info/manifestTest.c

Add indexed option > 1 to config/exec unit test.

commit   : c573ef281489e274c9b59185275f9c595885f359    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Oct 2020 16:14:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Oct 2020 16:14:03 -0400    

Click here for diff

M test/src/module/config/execTest.c

Use harnessCfgLoadRaw() in command/command and common/exit unit tests.

commit   : 176cf0bf609ac2fc49c411537d4aa518dccd33d9    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Oct 2020 14:54:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Oct 2020 14:54:28 -0400    

Click here for diff

The tests were originally written by loading values directly into the configuration before the parser was available.  
  
Update to use harnessCfgLoadRaw() to simplify the tests and make them compatible with upcoming config changes.  
  
Note that some unreachable conditions were removed since they could not be reached via a parsed config, only by munging values directly into the config. cfgOptionTest(optionId) was removed because a non-default value must always be set. cfgOptionValid(cfgOptLogTimestamp) was removed because it is true for all commands except for cfgCmdNone, which is checked with an assert.  

M src/command/command.c
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/module/command/commandTest.c
M test/src/module/common/exitTest.c

Get target-action default from cfgOptionDefault() in restore command.

commit   : 156b7d48cce789cc6e435219062ceeadbbb7b494    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Oct 2020 12:39:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Oct 2020 12:39:23 -0400    

Click here for diff

cfgDefOptionDefault() worked but the default is available without having to peek into config definitions.  

M src/command/restore/restore.c

Remove cfgOptionId() and replace it with cfgParseOption().

commit   : 41789d70d16a3921c68132187f08a1b8fa058cf1    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Oct 2020 11:24:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Oct 2020 11:24:26 -0400    

Click here for diff

cfgOptionId() did not recognize deprecated options which made the help command throw errors when they were specified on the command line. cfgParseOption() will correctly identify deprecated options.  
  
cfgParseOption() can also be used in cfgParse() to reduce code duplication when parsing info out of the option value returned by optionFind().  
  
Finally, code the option key index separately in parse.auto.c. For now they are simply added back together but future code will need them separated.  

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/command/help/help.c
M src/config/config.c
M src/config/config.h
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M src/config/protocol.c
M test/src/module/command/helpTest.c
M test/src/module/config/configTest.c
M test/src/module/config/parseTest.c

Remove ConfigDefineCommand enum.

commit   : 6414ae9707ab637f298df1ce55fe4bcf46fbe910    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 18:17:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 18:17:47 -0400    

Click here for diff

This has always been equivalent to the ConfigCommand enum so it just adds complexity.  
  
It was created for symmetry with ConfigDefineOption, which will also be removed soon.  

M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M src/command/command.c
M src/command/help/help.c
M src/command/restore/restore.c
M src/config/config.c
M src/config/config.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/define.c
M src/config/define.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M src/protocol/helper.c
M test/src/common/harnessConfig.c
M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/parseTest.c

Fix repo/pg-local option being overwritten.

commit   : ba29e24684414c610e58374c000cb66e28315975    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 17:17:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 17:17:47 -0400    

Click here for diff

This option would get overwritten when the option values were moved to index 1.  
  
Set the repo/pg-local option after options are moved to fix this.  

M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Reset additional options in real/all integration test.

commit   : 2d38d2fc824db1bb3f56899addd51ab018977d12    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 17:06:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 17:06:52 -0400    

Click here for diff

Currently indexes above 1 do not have dependencies checked, so this doesn't error.  
  
In a future commit we will enable those checks and this will error if it is not fixed.  

M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Use harnessCfgLoadRaw() in command/help unit test.

commit   : 996387ea9312459db664e59d68b94455c9c56acc    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 16:27:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 16:27:52 -0400    

Click here for diff

This correctly set defaults from cfgLoadUpdateOption(), e.g. repo-host-cmd.  

M test/src/module/command/helpTest.c

Remove extraneous calls to cfgOptionValid() in unit tests.

commit   : 303b9b17e700dca00aa12a3afe23b1f7c9741752    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 14:19:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 14:19:28 -0400    

Click here for diff

M test/src/module/command/repoTest.c
M test/src/module/storage/remoteTest.c

Remove indexed option constants.

commit   : 7d069a2b9178498293372913b0fa287005aa93b4    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 14:03:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Oct 2020 14:03:48 -0400    

Click here for diff

These constants don't scale well as the index total is increased for an option.  
  
The core code rarely uses these options and they are easily replaced with cfgOptionName().  
  
The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.  

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/command/archive/common.c
M src/command/archive/push/push.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/protocol/helper.c
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Expire history files.

commit   : 86275c4f85668d4576c16c85c629a7f3515ade59    
  
author   : Stefan Fercot <[email protected]>    
date     : Fri, 16 Oct 2020 13:42:03 +0200    
  
committer: GitHub <[email protected]>    
date     : Fri, 16 Oct 2020 13:42:03 +0200    

Click here for diff

WAL timeline history files were not being expired because they were small and generally not very plentiful.
  

  
However, in some cases large numbers of history files may be generated so it makes sense to remove useless history files to keep things tidy.
  

  
The history file for the oldest retained timeline is kept for debugging purposes even though it is not used for recovery.

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/expire/expire.c
M test/src/module/command/expireTest.c

Assert when buffer used is greater than size limit.

commit   : 4b8dfd34c0a9ced0977c43c2b4159eb700eadf28    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Oct 2020 17:09:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Oct 2020 17:09:21 -0400    

Click here for diff

099281c5 reduced used but this could lead to surprising behavior so assert instead.  
  
Call bufUsedSet() before bufLimitSet() to avoid the assertion.  

M doc/xml/release.xml
M src/common/type/buffer.c
M test/src/module/common/typeBufferTest.c

Optimize small reads in IoRead.

commit   : 401e4f1df41a40860c784f1908994c6ee51f6a51    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Oct 2020 16:09:38 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 8 Oct 2020 16:09:38 -0400    

Click here for diff

Instead of using memmove() to manage the internal output buffer for every small read, track the current buffer position and only move data when the small read cannot be satisfied and more data is needed.

M doc/xml/release.xml
M src/common/io/read.c

Add option groups.

commit   : e0f09687e49805a4349933f5e244a2e88b27220b    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Oct 2020 10:52:19 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 8 Oct 2020 10:52:19 -0400    

Click here for diff

Group related options together so operations (e.g. valid, test, index total) can be performed on all options in the group.
  

  
Previously, options at the top of the hierarchy of the related options were used to do these tests. This was prone to error as option relationships changed and it was not always clear which option (or options) should be used.

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/command/check/check.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.h
M src/config/load.c
M src/db/helper.c
M test/src/module/config/configTest.c
M test/src/module/config/loadTest.c

Add missing assertion in lstSort().

commit   : dc8a9dab1b4e9b00a0e3b1394f519464c702cd60    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Oct 2020 08:37:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Oct 2020 08:37:53 -0400    

Click here for diff

M src/common/type/list.c

Reduce buffer used when size limit set.

commit   : 099281c51070fce5c801f90388b8f4316cea7f64    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 18:02:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 18:02:05 -0400    

Click here for diff

If the size is reduced then the amount used should be reduced as well.  
  
This is a bug but does not affect current use cases.  

M src/common/type/buffer.c
M test/src/module/common/typeBufferTest.c

Add bufSizeAlloc().

commit   : 39aca3269c440e635ae6ff04235628a9f9d30a79    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 16:34:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 16:34:13 -0400    

Click here for diff

M src/common/type/buffer.h
M test/src/module/common/typeBufferTest.c

Add missing assertion in lstFind().

commit   : c8a94d9eb6bb3972071eb3a4eb8e54ece1a5b56e    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 15:17:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 15:17:16 -0400    

Click here for diff

M src/common/type/list.c

Remove unused cfgDefOptionPrefix() function and data.

commit   : 98f6a1cffdd6eb2667a3d042aa5ec3c192129700    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 12:15:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 12:15:50 -0400    

Click here for diff

This function was made obsolete when Perl was removed in f0ef73db.  

M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M src/config/define.auto.c
M src/config/define.c
M src/config/define.h
M test/src/module/config/defineTest.c

Declare cfgDefOptionAllowListValue/Total() static.

commit   : 434282d343c131387ffc080ec5db9aa02ba3c413    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 12:03:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Oct 2020 12:03:23 -0400    

Click here for diff

These have not been used externally since Perl was removed in f0ef73db.  

M src/config/define.c
M src/config/define.h

Add repo-azure-endpoint option.

commit   : 9377d050724d334a18706b49090575f2160433f1    
  
author   : David Steele <[email protected]>    
date     : Tue, 6 Oct 2020 17:15:48 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 6 Oct 2020 17:15:48 -0400    

Click here for diff

This option allows alternate endpoints (e.g. Azure Government) to be configured.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/storage/azure/storage.c
M src/storage/azure/storage.h
M src/storage/helper.c
M test/src/module/command/helpTest.c
M test/src/module/storage/azureTest.c

Update config.guess and config.sub to latest versions.

commit   : 3ce6b3e5058a7b1893aeb0c587fe8938956bbef5    
  
author   : David Steele <[email protected]>    
date     : Mon, 5 Oct 2020 13:12:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 5 Oct 2020 13:12:48 -0400    

Click here for diff

M src/build/config.guess
M src/build/config.sub

Begin v2.31 development.

commit   : 3c8774fbab80a4300fa1d6935bc87036e41057dc    
  
author   : David Steele <[email protected]>    
date     : Mon, 5 Oct 2020 12:02:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 5 Oct 2020 12:02:31 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h