pgBackRest 2.33 commit log

v2.32: Repository Commands

commit   : aadc9e2fe6ac54783e61c1ff1660feea4d7b1da0    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 09:08:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 09:08:16 -0500    

Click here for diff

Bug Fixes:  
  
* Fix resume after partial delete of backup by prior resume. (Reviewed by Cynthia Shang. Reported by Tom Swartz.)  
  
Features:  
  
* Add repo-ls command. (Reviewed by Cynthia Shang, Stefan Fercot.)  
* Add repo-get command. (Contributed by Stefan Fercot, David Steele. Reviewed by Cynthia Shang.)  
* Add archive-mode-check option. (Contributed by Stefan Fercot. Reviewed by David Steele, Michael Banck.)  
  
Improvements:  
  
* Improve archive-get performance. (Reviewed by Cynthia Shang.)  

M CONTRIBUTING.md
M README.md
M doc/release.pl
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/index.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Improve expire command documentation.

commit   : d350d1cc2139038d6f45cfcb5b59a5dc5994d269    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 5 Feb 2021 11:48:07 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Feb 2021 11:48:07 -0500    

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/help/help.auto.c

Add repo-get command.

commit   : b65c370346948708af6f762770836aa27b1acf86    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Feb 2021 10:39:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Feb 2021 10:39:03 -0500    

Click here for diff

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c
M test/src/module/command/helpTest.c

Add repo-ls command.

commit   : 218cd078a65f4cf15191cc9232eda8d576523592    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Feb 2021 10:07:43 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Feb 2021 10:07:43 -0500    

Click here for diff

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c
M test/src/module/command/helpTest.c

Add missed comment.

commit   : 73f61a1f60856d6d7a05d36458981c46222cf3cf    
  
author   : David Steele <[email protected]>    
date     : Thu, 4 Feb 2021 17:24:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 4 Feb 2021 17:24:42 -0500    

Click here for diff

M src/config/parse.c

Add -g accidentally removed in 4e8d469f.

commit   : 9154d730308f4bf7a37e4a7c513d45b845d2a2cd    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 Feb 2021 17:05:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 Feb 2021 17:05:55 -0500    

Click here for diff

The tests all run fine without debug info but gdb and valgrind are a lot less useful without it.  

M test/lib/pgBackRestTest/Common/JobTest.pm

Add archive-mode-check option.

commit   : 4b46115345c8eddaa658bc2051dba30676444fef    
  
author   : Stefan Fercot <[email protected]>    
date     : Tue, 2 Feb 2021 19:43:14 +0100    
  
committer: GitHub <[email protected]>    
date     : Tue, 2 Feb 2021 19:43:14 +0100    

Click here for diff

This option disallows the PostgreSQL archive_mode=always setting and disabling it allows the setting.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/check/common.c
M src/command/help/help.auto.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/src/module/command/checkTest.c

Log configuration parameters in config test harness.

commit   : 101bf5d11466990092530709e419e83b0d9ce4d4    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Feb 2021 14:50:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 Feb 2021 14:50:22 -0500    

Click here for diff

This makes it easier to find the current configuration in tests.  

M test/src/common/harnessConfig.c

Set dry-run before updating options in config test harness.

commit   : 9baf5d7da7bc7107584c613ccf8ef7192d304ef5    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Feb 2021 14:42:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 Feb 2021 14:42:22 -0500    

Click here for diff

This is the same order that is used in cfgLoad() so apply it here as well.  
  
Also remove vestigial HRN_FEATURE_STORAGE define.  

M test/src/common/harnessConfig.c

Add bufEmpty().

commit   : 67d444b9e8673aafa6c68ef14e8a7352dda9571e    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Feb 2021 09:22:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 Feb 2021 09:22:01 -0500    

Click here for diff

This seems more readable than bufUsed() == 0, just like 7d6c0319.  

M src/common/compress/lz4/compress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/io/filter/filter.c
M src/common/io/filter/group.c
M src/common/io/http/request.c
M src/common/io/read.c
M src/common/io/write.c
M src/common/type/buffer.h
M src/common/type/pack.c
M src/common/type/xml.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/remote/protocol.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M test/src/common/harnessServer.c
M test/src/common/harnessStorage.c

Add lstEmpty(), strLstEmpty(), and varLstEmpty().

commit   : 7d6c0319f0cafa1b0ac0d62db57f3970290b40e2    
  
author   : David Steele <[email protected]>    
date     : Fri, 29 Jan 2021 14:27:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 29 Jan 2021 14:27:56 -0500    

Click here for diff

This seems more readable than lst*Size() == 0.  
  
Hopefully this will also eliminate usage of lst*Size() > 0/lst*Size() != 0 variants for the inverse.  

M src/command/archive/common.c
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/command.c
M src/command/expire/expire.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/repo/rm.c
M src/command/restore/restore.c
M src/command/stanza/delete.c
M src/command/verify/verify.c
M src/common/io/http/client.c
M src/common/io/http/query.c
M src/common/io/http/request.c
M src/common/type/json.c
M src/common/type/list.h
M src/common/type/pack.c
M src/common/type/string.c
M src/common/type/stringList.h
M src/common/type/variantList.h
M src/config/parse.c
M src/info/infoPg.c
M src/protocol/parallel.c
M src/storage/azure/storage.c
M test/src/common/harnessTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/performance/typeTest.c

Add configurable error handlers.

commit   : 5281e314221538ca2c09a36cd0a3110458457dbe    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 17:25:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 17:25:13 -0500    

Click here for diff

The stackTrace and memContext error handlers were hard-coded which made testing the error module in isolation impossible.  
  
Making the error handlers configurable also makes adding new ones in the future easier.  

M src/common/error.c
M src/common/error.h
M src/main.c
M test/define.yaml
M test/src/module/common/errorTest.c
M test/src/test.c

Add HRN_INTEST_* define to indicate when a test is being run.

commit   : 8e9f04cc3290f378b90120243b3694dfb113fe16    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 16:54:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 16:54:41 -0500    

Click here for diff

This is useful for initialization that needs to be done for the test and all subsequent tests.  
  
Use the new defines to implement initialization for sockets and statistics.  

M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/src/module/common/ioTlsTest.c
M test/src/module/common/statTest.c
M test/src/test.c

Update expire command log messages with repo prefix.

commit   : d5b919e65772557f98a211a1e1628c8192053df2    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 27 Jan 2021 16:33:01 -0500    
  
committer: GitHub <[email protected]>    
date     : Wed, 27 Jan 2021 16:33:01 -0500    

Click here for diff

In preparation for multi-repo support, a repo tag is added in this commit to the expire command log and error messages. This change also affects the expect logs and the user-guide. The format of the tag is "repoX:" where X is the repo key used in the configuration.
  

  
Until multi-repo support has been completed, this tag will always be "repo1:".

M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/expire/expire.c
M test/expect/mock-all-001.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/expireTest.c

Move cvtDoubleToStr() to strNewDbl().

commit   : 5d34bf3f38ff8ce3f88c8aed6c6ff01eb929d62b    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 11:50:10 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 11:50:10 -0500    

Click here for diff

This is a more logical location and it reduces the dependencies required to compile the common/convert module.  

M src/command/archive/push/push.c
M src/command/command.c
M src/command/help/help.c
M src/common/type/convert.c
M src/common/type/convert.h
M src/common/type/string.c
M src/common/type/string.h
M src/config/exec.c
M test/define.yaml
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeStringTest.c

Make unit test builds incremental based on coverage in prior tests.

commit   : 87eb081a8f7bf35f7f993316a17d5d42ad225176    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 10:57:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 10:57:42 -0500    

Click here for diff

When building tests only include files covered by the current test or by prior tests. This increases performance (less compilation and linking) and also helps detect cross-dependencies in the code. Since there are currently cross-dependencies the depend option is used to document them and allow compilation. The idea is to resolve them incrementally over time.  
  
Add the harness option to include harness modules when the minimum requirements for compilation are met.  
  
Add the feature option to indicate which features are now available in the harness (based on source modules already tested). This allows conditional compilation in harness modules when some features are not yet available.  

M test/define.yaml
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/src/common/harnessConfig.c
M test/src/common/harnessDebug.h
M test/src/common/harnessFork.h
M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h
M test/src/module/common/ioTlsTest.c
M test/src/module/common/statTest.c
M test/src/test.c

Disable DEBUG_TEST_TRACE when NDEBUG defined.

commit   : 73a8e91868f80b21e35685966aec343cca6a9c7a    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 10:29:23 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Jan 2021 10:29:23 -0500    

Click here for diff

The underlying functions used by the FUNCTION_TEST*() macros will not be able when NDEBUG is defined so disable them.  
  
It would be better to change #ifndef NDEBUG to #ifdef DEBUG_TEST_TRACE where needed to bring these into alignment but that will be a larger effort best undertaken separately.  

M src/common/debug.h

Fix indentation in .cirrus.yml.

commit   : 088df03355be88150e8a91c9d51dd73c895f5e4c    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:59:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:59:14 -0500    

Click here for diff

M .cirrus.yml

Fix .editorconfig indent for yaml files.

commit   : 14f2b41bdaf44eb697b80cbdf58289235234bb74    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:58:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:58:28 -0500    

Click here for diff

M .editorconfig

Remove storageLocal() dependency from common/parse module.

commit   : 59e57c7231488006e5a3681dc72c9328d1998450    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:55:20 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:55:20 -0500    

Click here for diff

storageLocal() requires the storage helper which is a huge dependency for this module.  
  
Pass in a storage object instead.  

M src/config/load.c
M src/config/parse.c
M src/config/parse.h
M test/src/common/harnessConfig.c
M test/src/module/config/parseTest.c

Replace storageLocal() with storageTest in postgres/interface test.

commit   : 468507843f0217f5eb7e9ba437691f4b6dcdb760    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:34:39 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:34:39 -0500    

Click here for diff

M test/src/module/postgres/interfaceTest.c

Add test in common/error module.

commit   : 77823af53db1e8efb43def926e5b5e63d989c8c4    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:31:48 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 19:31:48 -0500    

Click here for diff

This is required for coverage when the common/error module is run with just the source files required to make it run, rather than all source files as we do now.  
  
Likely something in the harness is providing coverage, but cover it explicitly so the coverage won't be lost if the harness changes.  

M test/src/module/common/errorTest.c

Remove version comment left over from Perl.

commit   : 85ad1aa2be9a2702dbcd923bb1a4e08abc4c7445    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 12:47:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 12:47:52 -0500    

Click here for diff

M src/version.h

Remove too-verbose braces in switch statements.

commit   : 456a300bb703024df46cbc34746649c7b54181b2    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 12:10:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 12:10:24 -0500    

Click here for diff

The original intention was to enclose complex code in braces but somehow braces got propagated almost everywhere.  
  
Document the standard for braces in switch statements and update the code to reflect the standard.  

M CODING.md
M doc/xml/coding.xml
M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/info/info.c
M src/command/repo/ls.c
M src/command/restore/restore.c
M src/common/compress/bz2/common.c
M src/common/compress/gz/common.c
M src/common/exit.c
M src/common/io/tls/session.c
M src/common/type/json.c
M src/common/type/list.c
M src/common/type/variant.c
M src/config/config.c
M src/config/parse.c
M src/info/manifest.c
M src/main.c
M src/postgres/client.c
M src/protocol/parallelJob.c
M test/src/common/harnessPack.c
M test/src/common/harnessServer.c
M test/src/common/harnessStorage.c
M test/src/common/harnessTest.c
M test/src/module/command/backupTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Use minio latest in documentation and integration tests.

commit   : f669da7dcc228da2167b6ad1966a9c035506553f    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 11:25:29 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jan 2021 11:25:29 -0500    

Click here for diff

At one time Minio had stability problems with latest but that appears to be resolved for the last year or so.  
  
Use latest so we'll know if something breaks since Minio is frequently used in production.  

M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm

Add backup verification to internal verify command.

commit   : 2e60b937097313be135bce551b59e3f6b9ebb51a    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 26 Jan 2021 11:21:36 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 26 Jan 2021 11:21:36 -0500    

Click here for diff

This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered.
  

  
Unit tests have been added and duplicate tests have been removed.

M doc/xml/release.xml
M src/command/verify/verify.c
M test/define.yaml
M test/src/module/command/verifyTest.c

Use explicit characters instead of Posix class in restore expression.

commit   : 5d48dd2fb3717167062acf7c157d0caffb77b2e4    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 25 Jan 2021 11:33:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Jan 2021 11:33:41 -0500    

Click here for diff

It is not clear how portable/supported the Posix character classes are. This way seems simpler and more portable.  
  
Updated from 5c98157b.  

M src/command/restore/restore.c

Add note about removing configuration to stanza-delete documentation.

commit   : e251ec574a29b9137defc0f493bc84ce5b82014a    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 25 Jan 2021 11:14:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Jan 2021 11:14:28 -0500    

Click here for diff

M doc/xml/reference.xml
M src/command/help/help.auto.c

Improve info command text output and --set handling.

commit   : 00fac1c0d18aef6c29067bb00500abc5867f702c    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 25 Jan 2021 09:19:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Jan 2021 09:19:05 -0500    

Click here for diff

The info command provides total sizes for files in the backup on the database as well as the repository. The text output and associated user documentation has been updated to provide more clarity regarding the sizes being displayed.  
  
In addition, the info command is updated to allow a user to optionally specify the repository when requesting a specific backup set. In this case, the text output will reflect the status of the stanza, the cipher types and archive min/max over all the repositories instead of a single repository when the repo option is specified.  

M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c
M src/command/info/info.c
M test/src/module/command/infoTest.c

Add XML::Checker::Parser to MacOS Perl modules.

commit   : 8e48308b0b4110cc11276231c4dee2067ca54e38    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 18:30:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 18:30:14 -0500    

Click here for diff

M test/Vagrantfile

Add .editorconfig to help enforce project style.

commit   : ee71eb28c4ea7986cfd86cd00a9cd30cd058669a    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 16:27:00 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 16:27:00 -0500    

Click here for diff

A .editorconfig

Use configure to generate Makefile variables for unit tests.

commit   : 4e8d469f4d02e0956fcbafc54e762252c8964513    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 16:24:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 16:24:14 -0500    

Click here for diff

The unit test Makefile generation was a hodge-podge of constants and rules based on distros/versions that easily got out of date and did not work on an unknown system. All of this dates from the mixed Perl/C unit test implementation.  
  
Instead use configure to generate most of the important Makefile variables, which allows the unit tests to run on multiple platforms, e.g. MacOS and FreeBSD.  
  
There is plenty of work to be done here and not all the unit tests work on MacOS and FreeBSD for various reasons.  
  
As a POC update the MacOS and FreeBSD tests on Cirrus-CI to run a few command unit tests.  

M .cirrus.yml
M src/Makefile.in
M test/Vagrantfile
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/module/common/debugOnTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/exitTest.c
M test/src/test.c
M test/test.pl

Add chmod to make file removal after tests more reliable.

commit   : ef2dc6d3f4fe10251e5ed67375723ec20b87e096    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:48:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:48:32 -0500    

Click here for diff

MacOS does not allow files to be removed recursively unless the owner has write and execute permissions on all the directories.  
  
Some tests leave the permissions in a bad state so fix them up before trying to delete.  

M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessTest.c

Allow the make command to be configured for test.pl.

commit   : 04e84da0effa5d3be312c123e5203a9ebc1d04d3    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:35:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:35:40 -0500    

Click here for diff

M test/lib/pgBackRestTest/Common/JobTest.pm
M test/test.pl

Cast constants explicitly in command/backupCommon test.

commit   : aafd2f528d0536f4a83cb7086b55bb6ff1e6d89c    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:29:10 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:29:10 -0500    

Click here for diff

Clang was not clear that these constants should be uint64_t.  
  
Found on MacOS M1.  

M test/src/module/command/backupCommonTest.c

Add wheel as possible group name in storage/posix test.

commit   : d922cb6b8e3995cf0303184e2a2c24ed36f2d56b    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:25:02 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:25:02 -0500    

Click here for diff

Some platforms use wheel as the group for symlinks instead of root.  

M test/src/module/storage/posixTest.c

More reliable log expect in common/log test.

commit   : 88e54ee6c7156da4359134cfb2838925d16bc315    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:22:29 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:22:29 -0500    

Click here for diff

The exact message is platform dependent so get the platform error to use in the expect.  
  
It doesn't matter what the message is as long as there is an error and it is logged.  

M test/src/module/common/logTest.c

Skip lz4 in performance/storage test when it is not present.

commit   : cbccae05b8619019a17dd8bfe2fca8cc7f7cb098    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:18:02 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:18:02 -0500    

Click here for diff

M test/src/module/performance/storageTest.c

Add casts to performance/storage test for 32-bit architectures.

commit   : fda105ebd18f050d8071f62add2933ecbff8b8a7    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:15:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:15:50 -0500    

Click here for diff

M test/src/module/performance/storageTest.c

Use YAML::Any module instead of YAML::XS in Perl.

commit   : d2057c53bd91ffb96d46d4ee71cc8029d30d6d49    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:06:38 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 15:06:38 -0500    

Click here for diff

YAML::XS requires libyaml so it not as portable as pure Perl versions of YAML.  
  
Instead of using YAML:PP just use the general YAML::Any module which uses whatever is installed. We are not concerned about performance for YAML so whatever works is fine.  

M build/lib/pgBackRestBuild/Error/Data.pm
M doc/xml/contributing.xml
M test/Vagrantfile
M test/ci.pl
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm

Add header required for test harness to compile on FreeBSD.

commit   : ea0dc8c2cb5c7e6e9d4b09b4fbf451b2d30bfdc8    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 13:39:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 13:39:13 -0500    

Click here for diff

M test/src/common/harnessServer.c

Use [[:blank:]] instead of \s for leading space matching in restore.

commit   : 5c98157bce2ae74a11c04cc6ecc01cd067fadb9a    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 11:46:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 11:46:55 -0500    

Click here for diff

This is more accurate since we don't really want lf/cr anyway, though the lines have already been split so that's not possible in this code for lf.  
  
Found on MacOS M1. FreeBSD also seems to be fine with the new expression.  

M src/command/restore/restore.c

Add optional warnings to configure.

commit   : 3b05f77d083d03c34f47a2372e7eeb684ad5b655    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 09:08:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 09:08:26 -0500    

Click here for diff

These warnings do not exist on all compilers/versions but are useful when present.  

M src/build/configure.ac
M src/configure

Remove semicolons from TEST_RESULT*() macros.

commit   : 4a83ca4f78c75e905b4b84aff1529ba91213c4dc    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 08:55:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 08:55:13 -0500    

Click here for diff

These prevented the macros from being used in single line statements (e.g. if-else).  
  
Also fix some missing semicolons in macro invocations.  

M test/src/common/harnessTest.h
M test/src/module/command/restoreTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/ioHttpTest.c

Add stderr to unit test error messages.

commit   : 5cb9f166ecf191891f7defcb41d03f4b0ea93475    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 08:23:59 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 08:23:59 -0500    

Click here for diff

Messages on stderr were being lost due to the error suppression used to customize the error message.  
  
Also update the formatting to be more informative and concise.  

M test/lib/pgBackRestTest/Common/ExecuteTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm

Fix logical -> bitwise boolean operator in backup unit test.

commit   : f95850c546ae6c5407eb0b1682362101d5e77c95    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 08:12:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 08:12:31 -0500    

Click here for diff

This unset more than the storageFeatureCompress flag but the test was not affected.  
  
Found on MacOS M1.  

M test/src/module/command/backupTest.c

Avoid NULL pointer arithmetic in MemContext unit test.

commit   : a7d32259cf00565986d065c93be25dc82bb1a024    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 08:05:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 08:05:31 -0500    

Click here for diff

Similar to b23a2a0b. In this case add 1 to avoid a NULL pointer.  
  
Found on MacOS M1.  

M test/src/module/common/memContextTest.c

Add MacOS error messages to RegExp unit test.

commit   : fe8ef3197e85ff30950867254cba15b9112f34bd    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 07:58:07 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Jan 2021 07:58:07 -0500    

Click here for diff

M test/src/module/common/regExpTest.c

Don't pass --ignore-missing-args to rsync on MacOS.

commit   : 483695cac6508b03b621742eef9940f4c1fe0bc9    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 13:16:48 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 13:16:48 -0500    

Click here for diff

MacOS has a very old version of rsync that does not support this option.  
  
Rather than require a newer version of rsync exclude the option since the plan is to remove the requirement for it.  

M test/test.pl

Move version check against release notes to release.pl.

commit   : 708c3e9135d69a2d7ac670fe8efbc4ff08b95fa5    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 12:52:30 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 12:52:30 -0500    

Click here for diff

This is a more appropriate place for the check and means test.pl can avoid loading any XML files if --no-gen is specified.  
  
The XML::Checker::Parser module originally selected for XML in Perl is not very portable so the requirement reduces the number of platforms where tests can be run.  

M doc/release.pl
M test/test.pl

Remove unused VerifyFileResult struct.

commit   : fdf1c299f90ccc3b02758af072a58e5d20eaf2c1    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 11:51:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 11:51:36 -0500    

Click here for diff

M src/command/verify/file.h

Use uintptr_t in ASSERT_ALLOC_VALID() to avoid pointer arithmetic.

commit   : b23a2a0baf51b27c3c02fc99074fb033ad61d4dd    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 10:48:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 10:48:22 -0500    

Click here for diff

Clang justifiably complains about pointer arithmetic on a known NULL value during testing. We know this is fine but use uintptr_t to silence the warnings.  
  
Found on MacOS M1.  

M src/common/memContext.c
M test/src/module/common/memContextTest.c

Add format attribute to stackTraceFmt().

commit   : 185a508f44bbb4cf939658424c2bdb8012cfcb0f    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 09:50:29 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 09:50:29 -0500    

Click here for diff

Let the compiler know this is a printf-style function so a non-literal format is allowed.  
  
Found on MacOS M1.  

M src/common/stackTrace.c

Use enum for signalType in exitSignalName().

commit   : a5ebaab249764ccb0727b481d4f7a65512117b34    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 09:16:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 09:16:36 -0500    

Click here for diff

This should have been an enum all along but was somehow missed.  
  
Found on MacOS M1.  

M src/common/exit.c

Use uint variant to store enum in verifyProtocol().

commit   : 547b297387b2097e457c2893beefc1cb1d3ccd1d    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 09:04:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Jan 2021 09:04:28 -0500    

Click here for diff

Avoid the need for a cast by using a matching type.  
  
Found on MacOS M1.  

M src/command/verify/protocol.c

Fix return type for PQresultStatus().

commit   : 8d883ea76b2e4d48bea3772367ca3e3257434a07    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 18:30:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 18:30:24 -0500    

Click here for diff

Found on MacOS M1.  

M src/postgres/client.c

Change variant to uint when building option lists.

commit   : bc25e9bf056b7082a1e69a7a59affb1b61ae6f51    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:36:33 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:36:33 -0500    

Click here for diff

Enums are uints on most platforms so this works without casting.  
  
Found on MacOS M1.  

M src/command/help/help.c

Cast result of cfgParseOptionId() to int.

commit   : 0869b8afffed2664331a68f72194648593dc7b45    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:29:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:29:24 -0500    

Click here for diff

It would probably make more sense to add an "invalid" enum value, but at least fix the cast for now. The function was originally designed to interface with Perl which required -1 in this case.  
  
Found on MacOS M1.  

M src/config/parse.c

Remove useless casts of vsnprintf() return value.

commit   : 3f72eca90954e26cb060037b97aada4f6f323ccf    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:26:12 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:26:12 -0500    

Click here for diff

The return value is not checked because we are happy with a truncated result in this case, which is guaranteed by passing the buffer size.  
  
Found on MacOS M1.  

M test/src/common/harnessConfig.c

Add missing mode_t casts.

commit   : 6a992a5884976555c20754d57b7ea1f1c30b7b5e    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:23:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:23:24 -0500    

Click here for diff

Found on MacOS M1 where mode_t is short.  

M src/command/restore/protocol.c
M src/common/type/convert.c
M src/info/manifest.c
M src/storage/remote/protocol.c
M src/storage/remote/storage.c

Fix cast for timeval.tv_usec.

commit   : 4b3200961ed8aca74e2fb47feff40b9206fd59b4    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:20:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:20:13 -0500    

Click here for diff

Found testing on MacOS M1.  

M src/common/time.c
M src/protocol/parallel.c

Remove useless assignment.

commit   : 6f529155b673fc5c04a51e6d2549655fe2667f3a    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:17:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 17:17:03 -0500    

Click here for diff

M test/src/common/harnessServer.c

Partial multi-repository implementation.

commit   : f32eb9b94ecf01d12f53a1bd6addeba81e811d7f    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 21 Jan 2021 15:21:50 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 21 Jan 2021 15:21:50 -0500    

Click here for diff

Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands.
  

  
Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations.
  

  
Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml
M src/command/archive/push/file.c
M src/command/archive/push/file.h
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M src/command/check/check.c
M src/command/help/help.auto.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/command/stanza/upgrade.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.intern.h
M src/config/load.c
M src/config/parse.auto.c
M src/config/parse.c
M src/protocol/helper.c
M src/protocol/helper.h
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Common/StorageRepo.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

Fix incorrect variable in parameter logging.

commit   : 1333748550d61b98c83f0ea1e730ff8360328a8c    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 21 Jan 2021 14:20:39 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 14:20:39 -0500    

Click here for diff

M src/command/stanza/delete.c

Use standard environment variable format in hrnCfgEnv*().

commit   : d3d484b27c8f9b695ef283a789796751cba43531    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 11:59:19 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jan 2021 11:59:19 -0500    

Click here for diff

The option portion was not being capitalized or replacing - with _.  
  
The parser does not care, but in cases where we have mixed hrnCfgEnv*()/setenv() calls the env variable might not get cleared, which can lead to funny test results.  

M test/src/common/harnessConfig.c

Compensate for numeric auto conversion in newer Perls.

commit   : 4e56948128878cbeb8ac6faf89d65aae644679bc    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Jan 2021 12:07:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Jan 2021 12:07:05 -0500    

Click here for diff

M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm

Remove errant todo comment.

commit   : d9fae0984853173f9f867f2eaf640a21e424ec32    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Jan 2021 11:59:47 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Jan 2021 11:59:47 -0500    

Click here for diff

The test is not a duplicate of the one above because it adds the archive-async option.  

M test/src/module/command/archiveGetTest.c

Remove duplicate mkdir for /tmp/pgbackrest in Vagrantfile.

commit   : a1365b26d4c2383eaacea3714eeb521435ca79aa    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Jan 2021 23:41:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Jan 2021 23:41:09 -0500    

Click here for diff

M test/Vagrantfile

Add pkg-config package to Vagrantfile.

commit   : 8322dfd7af97e307b952cfb37e6a65b65f55be0f    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Jan 2021 23:39:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Jan 2021 23:39:14 -0500    

Click here for diff

Newer Ubuntu versions do not install this package by default.  

M test/Vagrantfile

Use test lock path in archive-get test.

commit   : 49582c13cfbb49736cc50d333ec74e5be1b9388e    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Jan 2021 23:05:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Jan 2021 23:05:55 -0500    

Click here for diff

The default lock path should fail since the test VM gives ownership of /tmp to root.  
  
For some reason this was not working as expected under u18 but it fails under u20.  

M test/src/module/command/archiveGetTest.c

Improve test coverage list handling.

commit   : 065b5f93aec471b4ac5c189520eb354a3d078b5f    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Jan 2021 10:56:51 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Jan 2021 10:56:51 -0500    

Click here for diff

All unit tests now require full coverage so the "full" keyword is obsolete and has been removed.  
  
The covered code modules are simply listed, with only "no code" modules annotated.  

M test/define.yaml
M test/lib/pgBackRestTest/Common/DefineTest.pm

Improve archive-get performance.

commit   : a8fb285756853bbe2f7b2e88bc817236d23b7ed8    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Jan 2021 10:15:52 -0500    
  
committer: GitHub <[email protected]>    
date     : Fri, 15 Jan 2021 10:15:52 -0500    

Click here for diff

Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get.
  

  
Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process.
  

  
Cache results of storageList() when looking for multiple files to reduce storage I/O.
  

  
Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.

M doc/xml/release.xml
M src/command/archive/get/file.c
M src/command/archive/get/file.h
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M test/define.yaml
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/archiveGetTest.c

Add cfgOptionValid() check to assert in cfgOptionIdxTest().

commit   : 38583b6414d8d55c92a25a1d32c4bc9457b912c4    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Jan 2021 08:59:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Jan 2021 08:59:16 -0500    

Click here for diff

Skip the rest of the assert if the option is not valid since this function is intended to work with invalid options.  

M src/config/config.c

Fix inverted cfgOption*() calls.

commit   : 7f263dceb7aee6c8d43313ac954d279868be3661    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Jan 2021 08:30:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Jan 2021 08:30:56 -0500    

Click here for diff

These currently work because repoIdx is always 0.  

M src/config/config.c
M src/storage/helper.c

Remove non-conforming periods and reformat some comments.

commit   : 298cc4d5e5d79b400beac4b70352e625e56b0ed6    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Jan 2021 10:39:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Jan 2021 10:39:25 -0500    

Click here for diff

M src/command/info/info.c

Add co7 package to support llvm.

commit   : c2c702c09d4ac3ea858d0450695a789c999b99da    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Jan 2021 17:32:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Jan 2021 17:32:42 -0500    

Click here for diff

This is required for new package versions.  
  
Also remove the obsolete 9.2 package and update the supported versions list.  

M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm

Add duplicate WAL test to synchronous archive-get.

commit   : a57e0be395146aa13d136b583a015c8077adbadd    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Jan 2021 13:01:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Jan 2021 13:01:40 -0500    

Click here for diff

M test/src/module/command/archiveGetTest.c

Improve logging in archive-get command.

commit   : 22fd223fc393908f6f0d5f0a64ee8f93317ce24e    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Jan 2021 10:24:47 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Jan 2021 10:24:47 -0500    

Click here for diff

Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though).  
  
Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful.  
  
Add constants for messages that are used multiple times to ensure they stay consistent.  

M doc/xml/user-guide.xml
M src/command/archive/get/get.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/archiveGetTest.c

Add FUNCTION_LOG_RETURN_STRUCT() and update where appropriate.

commit   : c6aaf66e9d8c83ba7443bcce145802b1147a4daf    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Jan 2021 07:49:47 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Jan 2021 07:49:47 -0500    

Click here for diff

The FUNCTION_LOG_RETURN() macro requires logging macros (e.g. FUNCTION_LOG_*_TYPE and FUNCTION_LOG_*_FORMAT) when returning a struct but these macros don't deliver much value since they only output the name of the struct rather than the contents. A copy of the struct is also made during this operation, which is wasteful.  
  
FUNCTION_LOG_RETURN_STRUCT() does not make a copy of the struct and does not require any logging macros. Returned structures are logged as "struct" but this could be made more accurate using __typeof in the future.  
  
Structures as parameters are not addressed here and work as before, i.e. they require logging macros.  

M src/command/archive/get/file.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/verify/file.c
M src/command/verify/file.h
M src/command/verify/verify.c
M src/common/debug.h
M src/db/db.c
M src/db/helper.c

Check for missing files in queueNeed().

commit   : b21ed979821b247ed50ccd48c515ed3d742dc9da    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Jan 2021 18:47:19 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Jan 2021 18:47:19 -0500    

Click here for diff

Missing files would indicate that another process is running on the same spool path, which would be a very bad thing.  
  
This check doesn't cost any additional I/O so it seems like a good idea.  

M src/command/archive/get/get.c

Add tests for the archive-get command.

commit   : aae6f06bcf69794e843486bb88c490f180d5f2e5    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Jan 2021 18:20:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Jan 2021 18:20:28 -0500    

Click here for diff

Provide additional coverage for less common use cases.  

M test/src/module/command/archiveGetTest.c

Fix resume after partial delete of backup by prior resume.

commit   : aeee83044d97555723fdc6039be1c4d51e329d28    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Jan 2021 12:38:32 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 12 Jan 2021 12:38:32 -0500    

Click here for diff

If files other than backup.manifest.copy were left in a backup path by a prior resume then the next resume would skip the backup rather than removing it. Since the backup path still existed, it would be found during backup label generation and cause an error if it appeared to be later than the new backup label. This occurred if the skipped backup was full.
  

  
The error was only likely on object stores such as S3 because of the order of file deletion. Posix file systems delete from the bottom up because directories containing files cannot be deleted. Object stores do not have directories so files are deleted in whatever order they are provided by the list command. However, the issue can be reproduced on a Posix file system by manually deleting backup.manifest.copy from a resumable backup path.
  

  
Fix the issue by removing the resumable backup if it has no manifest files. Also add a new warning message for this condition.
  

  
Note that this issue could be resolved by running expire or a new full backup.

M doc/xml/release.xml
M src/command/backup/backup.c
M test/src/module/command/backupTest.c

Add job-retry and job-retry-interval options.

commit   : 96fd678662775032254906ed7365fad5c223ab74    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Jan 2021 15:15:25 -0500    
  
committer: GitHub <[email protected]>    
date     : Mon, 11 Jan 2021 15:15:25 -0500    

Click here for diff

These options specify the number of local worker job retries and the retry interval after one immediate retry.
  

  
There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests.
  

  
Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally.
  

  
These options are currently internal.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/command/help/help.auto.c
M src/command/local/local.c
M src/command/restore/restore.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.intern.h
M src/config/parse.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/common/harnessConfig.c
M test/src/module/config/parseTest.c

Restore storageRepo() call in archiveGetProtocol().

commit   : 8b86d5ea7a7dcdfaa087f1de05d4b1332235124d    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Jan 2021 11:34:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 11 Jan 2021 11:34:03 -0500    

Click here for diff

This call was removed by d1d25c71, which worked for archivePushProtocol() and verifyProtocol() since the encryption options are passed from the main process.  
  
archiveGetProtocol() still retrieves these options in the local process so the repo storage must be loaded first.  

M src/command/archive/get/protocol.c

Refactor common/archiveGet unit test.

commit   : f35d69c1c73719caaa5b6222ea6a836b84f933d5    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 16:48:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 16:48:32 -0500    

Click here for diff

The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor.  
  
Update using the new storage macros and reorganize the tests to provide better coverage.  

M test/define.yaml
M test/src/module/command/archiveGetTest.c

Make archive-get locality error generate a global.error file.

commit   : 8567b7f733ce821dad8955a8086209e9b592ebf2    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 16:29:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 16:29:56 -0500    

Click here for diff

Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.  

M src/command/archive/get/get.c
M test/src/module/command/archiveGetTest.c

Add storage test macros for common operations.

commit   : 7d3d6ecbe1e04d577687e8d4da7465c40f211344    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 16:14:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 16:14:26 -0500    

Click here for diff

The macros should make it much easier to write complex tests, especially when compression and encryption are involved.  
  
Update the command/archiveGet test to show how the new macros are used.  

M test/src/common/harnessInfo.c
M test/src/common/harnessInfo.h
M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/archiveGetTest.c

Add TEST_RESULT_STRLST_*() unit test macros.

commit   : 97c260bb56b14842e5fa272b6f5ef54cbc880dcb    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 12:49:33 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 12:49:33 -0500    

Click here for diff

This avoids the need for strLstJoin() when testing lists.  
  
Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.  

M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/config/execTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c

Add clarifications to unit test string diff messages.

commit   : 1ef36b403235320269bc4a4f232ff0157447f351    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 10:54:17 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 10:54:17 -0500    

Click here for diff

M test/src/common/harnessLog.c
M test/src/common/harnessTest.c

Improve formatting of unit test titles.

commit   : 17e29eb1bdeb0fd70516eb1f835e3b87062923ca    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 10:45:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 10:45:26 -0500    

Click here for diff

Add separation and some visual cues to help identify the start of a test.  
  
Also add a counter which can be used to search for a specific test, which is useful if there is a lot of debug output to search through.  

M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h

Add test paths for pg, repo, and spool.

commit   : dc0284412bf17cfc65218702f081dfca6da0b9a6    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 10:35:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 10:35:26 -0500    

Click here for diff

These may be used as standard locations for these paths in tests.  

M test/src/common/harnessConfig.h

Create TEST_PATH define in test.c.

commit   : 6bdbcdc7fe9edf51979ec082cf6616e978dc2b6e    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 10:32:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jan 2021 10:32:57 -0500    

Click here for diff

This allows paths to be constructed without using sprintf(), which makes the tests simpler and faster.  

M test/src/test.c

Remove forks from command/archive-get test.

commit   : ca9c1707f020ee52348d488352e2a27fc16e14ec    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Jan 2021 11:36:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 Jan 2021 11:36:42 -0500    

Click here for diff

These were required to deal with the legacy Perl code being unable to load new options between tests.  
  
The C code does not have this issue so remove the forks and update process ids in the log tests.  

M test/src/module/command/archiveGetTest.c

Remove archive-timeout from test in mock/archive.

commit   : 6e7a3eb38391b17fd82fccb4def1fae72532de4f    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Jan 2021 18:11:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Jan 2021 18:11:28 -0500    

Click here for diff

No timeout is expected here but the small timeout prevents errors from being thrown.  
  
This is not a bug since the error would be thrown on the next archive-get call but it does make the tests harder to debug when there is an error.  
  
It is not clear why there was a timeout here at all. It is likely cruft from a prior test or a copy/paste error.  

M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm

Remove duplicate tests from command/infoTest.

commit   : 656e7112965ddb2fb31234f3bf7cc9d979f6d4af    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 5 Jan 2021 15:14:16 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 5 Jan 2021 15:14:16 -0500    

Click here for diff

Tests that are duplicated are being removed from the info command unit tests. Specifically tests where the only thing different was whether a lock was held or not which affects only the status display. Removing these tests will reduce churn in the upcoming multi-repo support.

M test/src/module/command/infoTest.c

Set build-max to 2 for Github actions.

commit   : a7491182dccade65fa84fc40524cac878b32caa3    
  
author   : David Steele <[email protected]>    
date     : Mon, 4 Jan 2021 09:18:17 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 4 Jan 2021 09:18:17 -0500    

Click here for diff

Github runners only provide 2 virtual CPUs so the default build-max of 4 is not ideal.  

M .github/workflows/test.yml

commit   : 9345fe4cb2e85d2a1273bfb072ccf7a427399350    
  
author   : David Steele <[email protected]>    
date     : Sun, 3 Jan 2021 13:56:37 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 3 Jan 2021 13:56:37 -0500    

Click here for diff

M LICENSE
M src/common/macro.h
M src/postgres/interface/pageChecksum.vendor.c
M src/postgres/interface/static.vendor.h
M src/postgres/interface/version.vendor.h

Update test to work with different collations.

commit   : af17bc4b6d249d29617085aae696d630fcffac98    
  
author   : David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 15:34:07 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 15:34:07 -0500    

Click here for diff

The data returned by the protocol has not been sorted yet so it is vulnerable to differences in collation.  
  
Multiple records are not needed for this test so limit it to one path to solve this issue.  

M test/src/module/storage/remoteTest.c

Remove explicit command lists where they equal the default.

commit   : 0e1612cda16ddad3fa537351b8ef23a4912367d0    
  
author   : David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 12:29:11 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 12:29:11 -0500    

Click here for diff

This reduces noise in the file and new commands will automatically get these options.  

M build/lib/pgBackRestBuild/Config/Data.pm

Audit options valid for expire command.

commit   : 108038292c1aa96a261c152d8b7995a20ae01f51    
  
author   : David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 12:13:20 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 12:13:20 -0500    

Click here for diff

M build/lib/pgBackRestBuild/Config/Data.pm
M src/config/parse.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log

Audit options valid for start/stop commands.

commit   : 0acfcb669e7458ccc518d9fc1c704772be914c76    
  
author   : David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 11:10:48 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 11:10:48 -0500    

Click here for diff

M build/lib/pgBackRestBuild/Config/Data.pm
M src/config/parse.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log

Remove option exclusions now handled by command role validity.

commit   : 82a75268c1cc7bef2eedacec1c8464e26da00804    
  
author   : David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 10:22:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 10:22:09 -0500    

Click here for diff

These options were explicitly excluded because it was possible for them to be mangled by SSH if they contained spaces.  
  
They are now excluded by command role validity rules.  

M src/protocol/helper.c

Limit pg option validity and make it command-line only.

commit   : 09fdde359c8b70653fcafe9ad39fcb25bdbc672b    
  
author   : David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 10:08:58 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 10:08:58 -0500    

Click here for diff

The pg option only has one current usage, to let the backup local know which pg index it should copy files from.  
  
There are other possible uses for this option, but they need thought, tests, and documentation.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M src/config/parse.auto.c
M src/protocol/helper.c
M test/src/module/command/localTest.c
M test/src/module/config/parseTest.c
M test/src/module/protocol/protocolTest.c

Remove repo option.

commit   : 951cfa9e900a5ff98acc2a9ce8df9d83e0314862    
  
author   : David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 08:12:35 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 31 Dec 2020 08:12:35 -0500    

Click here for diff

This option was added in advance of the multi-repo functionality but it has no purpose and it is not clear what the validity rules should be.  
  
The option will be added back when multi-repo functionality is committed.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M src/command/help/help.auto.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/parse.auto.c
M src/config/parse.c
M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Call storageRemoteInfoParse() in prior context to avoid dups.

commit   : e3762945c2344d6d2f8fa727cbbc40e87300e75c    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Dec 2020 17:33:33 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Dec 2020 17:33:33 -0500    

Click here for diff

This results in fewer data duplications and makes the code less fragile since new data add in storageRemoteInfoParse() does not need to be added to an additional list for duplication.  

M src/storage/remote/storage.c

Add empty archive array to info command JSON when stanza is missing.

commit   : cc901632337ed9d50801037e1a31ad66ade6ad21    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 30 Dec 2020 16:17:56 -0500    
  
committer: GitHub <[email protected]>    
date     : Wed, 30 Dec 2020 16:17:56 -0500    

Click here for diff

There is an inconsistency when the JSON is output for the case when a stanza is requested and it does not exist in the repo. This was the only case where the archive array was not added to the JSON. Adding it will simplify the upcoming multi-repo support code.
  

  
Also, a redundant test was removed rather than updating it for this case.

M src/command/info/info.c
M test/src/module/command/infoTest.c

Do not pass pg-local/repo-local to a remote process.

commit   : 9bf7dbf6a2684c529da501eb95a1426ebb5d64ca    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Dec 2020 16:03:49 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Dec 2020 16:03:49 -0500    

Click here for diff

This was a hack to prevent the remote from loading host settings, which is now handled by option validity for command roles.  
  
These options are still useful so don't remove them, but do leave them internal for now.  

M build/lib/pgBackRestBuild/Config/Data.pm
M src/config/parse.auto.c
M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Remove redundant command list in repo-s3-key option.

commit   : 141466875f42f2dc09567efbcf757ad1000e715c    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Dec 2020 10:51:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Dec 2020 10:51:26 -0500    

Click here for diff

Use the repo-type command list as similar repo options do.  

M build/lib/pgBackRestBuild/Config/Data.pm

Limit option validity by command role.

commit   : abb8ebe58b7a1325dd95d967347d221e2b196ba3    
  
author   : David Steele <[email protected]>    
date     : Tue, 29 Dec 2020 15:49:37 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 29 Dec 2020 15:49:37 -0500    

Click here for diff

Building on 23f5712, limit option validity by role. This is mostly for options that weren't needed for certain roles but were harmless. However, the upcoming multi repository functionality requires the granularity implemented here.
  

  
The remote role benefits since host options can automatically excluded when building the options. Also, many options that are only required for the default role (e.g. repo-retention-full) no longer need to be passed in tests for other roles.

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/config/load.c
M src/config/parse.auto.c
M src/protocol/helper.c
M test/src/module/config/execTest.c
M test/src/module/protocol/protocolTest.c

Update unit tests for stricter option validity.

commit   : 26e81785d92775c6643e800644c6a73d01e54b53    
  
author   : David Steele <[email protected]>    
date     : Mon, 28 Dec 2020 15:22:21 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 28 Dec 2020 15:22:21 -0500    

Click here for diff

Some tests used options in contexts that are currently valid but are not correct usage, i.e. usage of internal options for the default role.  
  
Update these tests in advance of the option validity becoming stricter.  

M test/src/module/config/parseTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

Allow option validity to be determined by command role.

commit   : 23f5712d022ab34d370b7832813a597088842ca2    
  
author   : David Steele <[email protected]>    
date     : Mon, 28 Dec 2020 09:43:23 -0500    
  
committer: GitHub <[email protected]>    
date     : Mon, 28 Dec 2020 09:43:23 -0500    

Click here for diff

Validity by command was not granular enough so numerous options needed be marked internal so users would not stumble across them. Options were also needlessly being passed to roles that had no use for them.
  

  
Introduce per-role validity lists that depend on what roles are valid per command. Also add a check to ensure that only valid roles are used with a command.
  

  
This commit adds the functionality but does not introduce any new behavior, i.e. all options are valid for all roles that the command is valid for. A subsequent commit will introduce the new role restrictions to make the changes easier to audit.

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M src/command/command.c
M src/command/help/help.c
M src/config/config.h
M src/config/exec.c
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M test/src/common/harnessConfig.c
M test/src/module/config/parseTest.c

Fix indentation error preventing pull requests from being tested.

commit   : 715fa7a2f1f52dd1a82bb9ff3b0261ae046b1240    
  
author   : David Steele <[email protected]>    
date     : Mon, 28 Dec 2020 08:51:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 28 Dec 2020 08:51:22 -0500    

Click here for diff

M .github/workflows/test.yml

commit   : 9e9e7c4a0d59b9dbcdaaefb5b5a2cf51621ef64a    
  
author   : David Steele <[email protected]>    
date     : Thu, 17 Dec 2020 09:32:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 17 Dec 2020 09:32:31 -0500    

Click here for diff

Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl.  
  
Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement.  
  
Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.  

M build/lib/pgBackRestBuild/Config/Build.pm
D build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/Makefile.in
M src/command/command.c
M src/command/help/help.c
M src/config/config.auto.c
M src/config/config.c
M src/config/config.h
M src/config/config.intern.h
D src/config/define.auto.c
D src/config/define.c
D src/config/define.h
M src/config/exec.c
M src/config/parse.auto.c
R065 src/config/define.auto.h src/config/parse.auto.h
M src/config/parse.c
M src/config/parse.h
M src/protocol/helper.c
M src/storage/helper.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/common/harnessConfig.c
M test/src/module/command/helpTest.c
D test/src/module/config/defineTest.c
M test/src/module/config/parseTest.c
M test/test.pl

Move help data from define.auto.c/config.auto.c to a pack.

commit   : f520ecc89a211dbbe22a38c0b887e910bc1f866b    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Dec 2020 15:59:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Dec 2020 15:59:36 -0500    

Click here for diff

The help data can be represented more compactly in a pack and this separates data needed for help from data needed for parsing, freeing each to have a more appropriate representation.  

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
A build/lib/pgBackRestBuild/Config/BuildHelp.pm
A src/command/help/help.auto.c
M src/command/help/help.c
M src/config/config.auto.c
M src/config/config.c
M src/config/config.h
M src/config/define.auto.c
M src/config/define.c
M src/config/define.h
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/module/config/defineTest.c
M test/src/module/config/parseTest.c
M test/test.pl

Remove cfgCmdNone from CFG_COMMAND_TOTAL.

commit   : 996de0a3e6901219a84e7c1a20636835daf606a4    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Dec 2020 11:33:51 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Dec 2020 11:33:51 -0500    

Click here for diff

cfgCmdNone is used to indicate a missing or invalid command so should not be used in the total used for command process.  

M build/lib/pgBackRestBuild/Config/Build.pm
M src/command/help/help.c
M src/config/config.auto.h

Always advance id when pack field is NULL.

commit   : 558ff1e55587645ba033e30f147e56fe444b1276    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Dec 2020 09:59:48 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Dec 2020 09:59:48 -0500    

Click here for diff

This was done in the internal versions but not the user-facing function. That meant the field had to be explicitly read after determining it was NULL, which is wasteful.  
  
Since there is only one behavior now, remove pckReadDefaultNull() and move the logic to pckReadNullInternal().  

M src/common/type/pack.c
M src/common/type/pack.h
M test/src/module/common/typePackTest.c

Remove cfgDefOptionIndexTotal().

commit   : 39963f6aa5f86648da6cc6b36f856585654af28c    
  
author   : David Steele <[email protected]>    
date     : Mon, 14 Dec 2020 14:37:23 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 14 Dec 2020 14:37:23 -0500    

Click here for diff

This function was only used in one place, which was better served by cfgOptionGroupIdxTotal().  

M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M src/config/define.auto.c
M src/config/define.c
M src/config/define.h
M src/protocol/helper.c
M test/src/module/config/defineTest.c

Remove deprecated option from command/archive-push unit test.

commit   : 7f66ba5762703af2b5ddaa504d092685184ec7c8    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Dec 2020 17:58:59 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Dec 2020 17:58:59 -0500    

Click here for diff

M test/src/module/command/archivePushTest.c

Remove redundant distro specification.

commit   : 7429aca8fa4269e1370cc31283688b97b0848769    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Dec 2020 17:26:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Dec 2020 17:26:52 -0500    

Click here for diff

M .travis.yml

Move most tests to Github Actions.

commit   : d01669aa58cfabeeff859819334b697984e69988    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Dec 2020 15:19:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Dec 2020 15:19:01 -0500    

Click here for diff

Testing on Travis-CI has been getting slower (from ~18 minutes to 3-6 hours) and the travis-ci.org service will be terminated at the end of the year. Moving to travis-ci.com is an option but the quotas are too low for our purposes.  
  
Instead use Github Actions, which does not currently have quotas, and runs our current tests with just a few tweaks.  
  
This still leaves multi-architecture tests on Travis-CI but we may be able to run those and stay within the new quotas.  
  
Also fix a minor bug in restoreTest.c exposed by Github Actions using a different name for the user and group.  

A .github/workflows/test.yml
M .travis.yml
M doc/xml/contributing.xml
R095 test/travis.pl test/ci.pl
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/src/module/command/restoreTest.c

Add pack type.

commit   : 8361a97482c9b481863cf0d7300936a7d3dbe9df    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Dec 2020 12:05:14 -0500    
  
committer: GitHub <[email protected]>    
date     : Wed, 9 Dec 2020 12:05:14 -0500    

Click here for diff

The pack type is an architecture-independent format for serializing data compactly, inspired by ProtocolBuffers and Avro.
  

  
Also add ioReadSmall(), which is optimized for small binary reads, similar to ioReadLineParam().

M doc/xml/release.xml
M src/Makefile.in
M src/common/io/read.c
M src/common/io/read.h
M src/common/type/convert.h
A src/common/type/pack.c
A src/common/type/pack.h
M test/define.yaml
A test/src/common/harnessPack.c
A test/src/common/harnessPack.h
M test/src/module/common/ioTest.c
A test/src/module/common/typePackTest.c

Replace double type with time in config module.

commit   : 87996558d2cc12d3795709cf07e9c36ebb21e127    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Dec 2020 08:59:51 -0500    
  
committer: GitHub <[email protected]>    
date     : Wed, 9 Dec 2020 08:59:51 -0500    

Click here for diff

The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand.
  

  
Also remove Variant doubles since they are no longer used.
  

  
Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.

M CONTRIBUTING.md
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/contributing.xml
M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/command.c
M src/command/help/help.c
M src/command/local/local.c
M src/command/remote/remote.c
M src/command/repo/get.c
M src/command/repo/ls.c
M src/command/restore/restore.c
M src/command/verify/verify.c
M src/common/type/convert.c
M src/common/type/convert.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/config/config.c
M src/config/config.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/define.c
M src/config/define.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M src/db/helper.c
M src/db/protocol.c
M src/protocol/helper.c
M test/define.yaml
M test/src/module/command/backupTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/helpTest.c
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c

Update install-sh, config.guess, and config.sub to latest versions.

commit   : a137c6baaa17341ff4c9c85fe5954bbc02f47540    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Dec 2020 11:26:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Dec 2020 11:26:34 -0500    

Click here for diff

M doc/RELEASE.md
M src/build/config.guess
M src/build/config.sub
M src/build/install-sh

Begin v2.32 development.

commit   : ab0500789e762d3a8cedf3c692c2646ea8db9cd6    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Dec 2020 11:13:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Dec 2020 11:13:45 -0500    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h