pgBackRest 2.21 commit log

v2.20: Bug Fixes

commit   : 39fc2b7ad68d97cf5f23757e8a341428935621f4    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 08:20:21 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 08:20:21 -0500    

Click here for diff

Bug Fixes:  
  
* Fix archive-push/archive-get when PGDATA is symlinked. These commands tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call. (Reported by Stephen Frost, Milosz Suchy.)  
* Fix reference list when backup.info is reconstructed in expire command. Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual. This unlikely combination of events means this is probably not a problem in the field.  
* Fix segfault on unexpected EOF in gzip decompression. (Reported by Stephen Frost.)  

M README.md
M doc/RELEASE.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml

Allow timezones to be explicitly set for testing.

commit   : b031dbbcf8fa83ef60cfabc20fbf16046904f9aa    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 22:11:04 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 22:11:04 -0500    

Click here for diff

The TZ environment variable was not reliably pushed down to the test processes.  
  
Instead pass TZ via a command line parameter and set explicitly in the test process.  

M .travis.yml
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/test.c
M test/test.pl

Use localtime() to format time_t in cvtTimeToZ().

commit   : db5175915a0dc09b7591d71886ee8ce3bf5a12c3    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 21:51:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 21:51:32 -0500    

Click here for diff

Using gmtime() produced output skewed by the local timezone.  
  
Since this function is currently only used for debug logging this is not a live bug in the field.  

M src/common/type/convert.c

Make type in comment more specific.

commit   : 811ecd758d9e2ca12966995ec6cfc1e4ff8c0cfd    
  
author   : Stephen Frost <[email protected]>    
date     : Wed, 11 Dec 2019 14:57:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 14:57:32 -0500    

Click here for diff

M src/common/type/convert.c

Fix archive-push/archive-get when PGDATA is symlinked.

commit   : 0194a9867182435c7a181be6466ca2b91f05801d    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 14:36:39 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 14:36:39 -0500    

Click here for diff

Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked.  
  
If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.  

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c

Fix segfault on unexpected EOF in gzip decompression.

commit   : 8c840c28a65dae3cca581691e88e51839cd64bca    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 08:48:46 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 Dec 2019 08:48:46 -0500    

Click here for diff

If the compressed stream terminated early then the decompression process would get a flush request (NULL input buffer) since the filter was not marked as done.  This could happen on a zero-length or truncated (i.e. invalid) compressed file.  
  
Change the existing assertion to an error to catch this condition in production gracefully.  

M doc/xml/release.xml
M src/common/compress/gzip/decompress.c
M test/src/module/common/compressGzipTest.c

Remove obsolete --perl-option option.

commit   : c933f12f9c39d7c076f84252c081c3430be72e55    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Dec 2019 13:28:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Dec 2019 13:28:15 -0500    

Click here for diff

This option was used when Perl was executed instead of being embedded.  
  
It has been obsolete for a long time so remove it.  

M build/lib/pgBackRestBuild/Config/Data.pm
M lib/pgBackRest/LibCAuto.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/perl/embed.auto.c
M test/src/module/perl/configTest.c

Remove test point infrastructure.

commit   : d0ba8ff58c46c15facb4dba19eab8db83835b07c    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Dec 2019 13:16:47 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Dec 2019 13:16:47 -0500    

Click here for diff

82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.  
  
Since tests points are no longer used remove the infrastructure.  
  
Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.  

M build/lib/pgBackRestBuild/Config/Data.pm
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Main.pm
M lib/pgBackRest/Protocol/Base/Master.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Common/ExecuteTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/config/defineTest.c
M test/src/module/config/parseTest.c

Make buildPutDiffers() work with empty files.

commit   : d7d663c2b93fd6a3bdfb1c6a4bd1feae0cd8cc63    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Dec 2019 13:02:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Dec 2019 13:02:36 -0500    

Click here for diff

If the file was empty the timestamp was updated.  If the file is empty and there is no content then file should not be saved.  

M test/lib/pgBackRestTest/Common/BuildTest.pm

Remove stray uint type.

commit   : 800d2972b00bb381c92cdadead9a1c04141853be    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Dec 2019 18:28:20 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Dec 2019 18:28:20 -0500    

Click here for diff

This was probably copied from an example but some compilers don't like it.  

M test/src/common/harnessTls.c

Add stringz module to define some commonly used strings.

commit   : 471d54a738887dd5c99afced36420cf6fb5c631b    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Dec 2019 17:55:20 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Dec 2019 17:55:20 -0500    

Click here for diff

This module will eventually contain various useful zero-terminated string functions.  
  
For now, using NULL_Z instead of strPtr(NULL_STR) avoids a strict aliasing warning on RHEL 6.  This is likely a compiler issue, but adding these constants seems like a good idea anyway and we are not going to get a fix in a gcc that old.  

M src/Makefile.in
M src/command/check/common.c
M src/common/debug.c
M src/common/debug.h
M src/common/type/convert.c
M src/common/type/json.c
M src/common/type/string.c
A src/common/type/stringz.h
M src/common/type/variant.c
M src/db/db.c

Remove redundant test and move another test.

commit   : ca33545630a073579ef763294d897dd06820f5a5    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 9 Dec 2019 14:06:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Dec 2019 14:06:32 -0500    

Click here for diff

M test/src/module/info/manifestTest.c

Note that next repo format should use UTC for backup labels.

commit   : 61ad3a44c9e562a306064770428726a5664056e9    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Dec 2019 09:12:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Dec 2019 09:12:34 -0500    

Click here for diff

M doc/FORMAT.md

Add functions for building new manifests.

commit   : d3132dae26ad40c2a47db62d4cc5f4502465d7c6    
  
author   : David Steele <[email protected]>    
date     : Sun, 8 Dec 2019 18:43:47 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 8 Dec 2019 18:43:47 -0500    

Click here for diff

New manifests are built before a backup is performed.  
  
Reviewed by Cynthia Shang.  

M doc/xml/release.xml
M src/Makefile.in
M src/info/manifest.c
M src/info/manifest.h
M test/define.yaml
M test/src/module/info/manifestTest.c

Add pgLsnFromStr(), pgLsnToStr(), and pgLsnToWalSegment().

commit   : 2cfde18755f305008af134ac833aca6989321ca7    
  
author   : David Steele <[email protected]>    
date     : Sun, 8 Dec 2019 14:19:47 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 8 Dec 2019 14:19:47 -0500    

Click here for diff

M src/postgres/interface.c
M src/postgres/interface.h
M test/define.yaml
M test/src/module/postgres/interfaceTest.c

Update pq harness to play nicely with variable LSNs.

commit   : f517b141fb065efd21878fc774f51b52c014d887    
  
author   : David Steele <[email protected]>    
date     : Sun, 8 Dec 2019 14:15:23 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 8 Dec 2019 14:15:23 -0500    

Click here for diff

M test/src/common/harnessPq.h

Add backup functions to Db object.

commit   : d2587250da82a90257534268e2175bfcfd3e15ac    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 18:44:06 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 18:44:06 -0500    

Click here for diff

These functions implement the database backup functionality for all supported versions.  

M src/Makefile.in
M src/db/db.c
M src/db/db.h
M test/define.yaml
M test/src/common/harnessPq.h
M test/src/module/db/dbTest.c

Add protocolRemoteFree() to shutdown a specific remote.

commit   : 8766326da812f6aae00bdbfef4b1382805de8c3e    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:48:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:48:53 -0500    

Click here for diff

Sometimes it is useful to shutdown remotes that are no longer needed instead of waiting for them to be shutdown at program exit.  

M src/protocol/helper.c
M src/protocol/helper.h
M test/src/module/protocol/protocolTest.c

Add constant for pg_tblspc path.

commit   : 5175d52e9466c339f5a4cd5c7d7e8724c3bbb2e3    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:42:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:42:42 -0500    

Click here for diff

M src/postgres/interface.h

Remove unused Perl module.

commit   : 78b1e05b7e6f8d09a424a2e8864e317d63c474f3    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:39:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:39:25 -0500    

Click here for diff

M lib/pgBackRest/Protocol/Local/Master.pm
M src/perl/embed.auto.c

Pq test harness usability and error reporting improvements.

commit   : 35a262951a1ed74431cdfa92c36ee2c9ae867c1a    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:33:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:33:34 -0500    

Click here for diff

Pq script errors are now printed in test output in case they are being masked by a later error.  
  
Once a script error occurs, the same error will be thrown forever rather than throwing a new error on the next item in the script.  
  
HRNPQ_MACRO_CLOSE() is not required in scripts unless harnessPqScriptStrictSet(true) is called.  Most higher-level tests should not need to run in strict mode.  
  
The command/check test seems to require strict mode but there's no apparent reason why it should.  This would be a good thing to look into at some point.  

M test/src/common/harnessPq.c
M test/src/common/harnessPq.h
M test/src/module/command/checkTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/db/dbTest.c
M test/src/module/postgres/clientTest.c

Add log replacements to help test non-deterministic log output.

commit   : d6479ddd0efda9bb442f4bbfe937daa0ab46ac72    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:15:20 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:15:20 -0500    

Click here for diff

Some log output (e.g. time) is hard to test because the values can change between tests.  
  
Add expressions to replace substrings in the log with predictable values to simplify testing.  
  
This is similar to the log replacement facility available for Perl expect log testing.  

M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/common/harnessTest.c
M test/src/module/command/commandTest.c
M test/src/module/db/dbTest.c

Improve diff output in tests.

commit   : e4716ee036e8a4417f6dcb8d108c0d7b9e3f2b7d    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:02:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 17:02:41 -0500    

Click here for diff

Always compare expected vs actual (in that order) and give a hint in the error message to indicate what should be added and what removed.  

M test/src/common/harnessLog.c
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h

Improve storage harness test callback.

commit   : 8c47ee296a1f311682b46b3f1de98d4eb7dcaf6a    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 16:55:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 16:55:50 -0500    

Click here for diff

Add ability to omit the root (i.e. dot) path and get real size of compressed files.  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Recopy during backup when resumed file is missing or corrupt.

commit   : 1b3770e248f4a5b59ba16f6a4c6cba7cf23de883    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 09:48:33 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 09:48:33 -0500    

Click here for diff

A recopy would occur if the size or checksum was invalid but on error the backup would terminate.  
  
Instead, recopy the resumed file on any error.  If the error is systemic (e.g. network failure) then it should show up again during the recopy.  

M src/command/backup/file.c
M test/src/module/command/backupTest.c

commit   : d3f717c89208ef132e75b634ad2e91efbda25a04    
  
author   : David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 09:26:51 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 7 Dec 2019 09:26:51 -0500    

Click here for diff

Since there is only one driver that supports (or is likely to support) links (Posix), require the path feature to make logic in the code simpler.  
  
The checks are added just in case another driver supports links.  

M src/storage/storage.c

Fix backup labels in mock/all resume integration tests.

commit   : e632c605250438a9e06f133b55c88e578ea709cd    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 Dec 2019 11:48:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 Dec 2019 11:48:41 -0500    

Click here for diff

These were not getting updated to match the directory name when the manifests were copied.  
  
The Perl code didn't care but the C code expects labels to be set correctly.  

M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Allocate one extra process for backup from standby.

commit   : 5395aa0c92febadbf0c71f93799877cfe7268b6b    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Dec 2019 22:34:38 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Dec 2019 22:34:38 -0500    

Click here for diff

The connection to the primary is process-max + 1 so make space for it.  

M src/protocol/helper.c

Add functions to get the substring found by regExpMatch().

commit   : b2d82bd2484dbe3dee12479efb6e9aceba5aac57    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 Dec 2019 19:43:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 4 Dec 2019 19:43:26 -0500    

Click here for diff

For now this is only used in testing but there are places where it could be useful in the core code.  
  
Even if that turns out not to be true, it doesn't seem worth implementing a new version in testing just to capture a few values that we already have.  

M src/common/regExp.c
M src/common/regExp.h
M test/src/module/common/regExpTest.c

Add asserts to ensure hostId > 0.

commit   : 053af2f54db0eaaaca3ac67cbdd68ba1b701048a    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 Dec 2019 19:31:39 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 4 Dec 2019 19:31:39 -0500    

Click here for diff

hostId is 1-based (e.g. pg1-*) so it should always be > 0.  

M src/protocol/helper.c

Add missing header.

commit   : d1f49825b11ba88ac2024734c4c8a1c961f05854    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Dec 2019 11:54:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Dec 2019 11:54:45 -0500    

Click here for diff

M src/postgres/interface.h

Use more general error code when tablespace linked into PGDATA.

commit   : 8dfe0e48e25725978738ecb3fec24e5ce2e9076b    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Dec 2019 10:49:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Dec 2019 10:49:25 -0500    

Click here for diff

The specific error code was not that useful since we also test the error message which contains details of the link error.  

M build/error.yaml
M lib/pgBackRest/Common/ExceptionAuto.pm
M lib/pgBackRest/Manifest.pm
M src/common/error.auto.c
M src/common/error.auto.h
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Add flag to dbGet() to require a standby.

commit   : 33a63aae503adf8d7f3f4430ee6478b845d38e2c    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Dec 2019 07:39:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Dec 2019 07:39:42 -0500    

Click here for diff

This is needed from backup from standby functionality.  

M src/command/check/check.c
M src/command/stanza/common.c
M src/db/helper.c
M src/db/helper.h
M test/src/module/command/checkTest.c
M test/src/module/db/dbTest.c

Error in remote command when stop file exists.

commit   : 28116918ff3a00b133d74d524df452e29398082d    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Dec 2019 07:35:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Dec 2019 07:35:36 -0500    

Click here for diff

This duplicates the Perl functionality.  

M src/Makefile.in
M src/command/remote/remote.c
M test/src/module/command/remoteTest.c

Reduce the scope of mock/all exclusion tests.

commit   : fc291b6f28753329afdd7b65b115f90d31569df1    
  
author   : David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 17:47:47 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 17:47:47 -0500    

Click here for diff

Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Make MCV return false when a boolean tie.

commit   : d15ed338217667f0bd7ffff3eb9f5e24052477fd    
  
author   : David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 16:32:21 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 16:32:21 -0500    

Click here for diff

This is to maintain compatibility with the older Perl code that returned the lowest sorted order item in a tie.  
  
For other datatypes the C code returns the same value, often enough at least to not cause churn in the expect tests.  

M src/common/type/mcv.c
M test/src/module/common/typeMcvTest.c

Add PostgreSQL version where backup from standby is supported.

commit   : ab0974cc8ea0cff70a9f680ff8e1dc10b1ba3c48    
  
author   : David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 16:30:44 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 16:30:44 -0500    

Click here for diff

M src/postgres/version.h

Add PostgreSQL file and path names used for building manifests.

commit   : 218698088f0e4d5a978b8f4c8df6b114980889b9    
  
author   : David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 16:28:33 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 16:28:33 -0500    

Click here for diff

M src/postgres/interface.c
M src/postgres/interface.h

Add pgLsnName() and pgXactPath().

commit   : 56ee321a9565e7c1e076c58744e0cfc408801e28    
  
author   : David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 15:49:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 1 Dec 2019 15:49:34 -0500    

Click here for diff

M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

Add storage feature flags.

commit   : bc83354fb53a1ba8a86366478c5be871775dd04a    
  
author   : David Steele <[email protected]>    
date     : Fri, 29 Nov 2019 11:29:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 29 Nov 2019 11:29:16 -0500    

Click here for diff

Even though storagePathSync() is a noop when path sync is not supported, it is useful to know in advance if the function will do anything so add the storageFeaturePathSync flag.  
  
The storageFeatureSymLink and storageFeatureHardLink flags are currently informational only since links are not yet implemented in the storage interface.  

M src/storage/posix/storage.c
M src/storage/storage.c
M src/storage/storage.h

Remove unused reference to use pgBackRest::Backup::File.

commit   : 0933aeb1ad954f8c7b2ed6d4e160d7e132ada8b7    
  
author   : David Steele <[email protected]>    
date     : Fri, 29 Nov 2019 10:56:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 29 Nov 2019 10:56:13 -0500    

Click here for diff

M lib/pgBackRest/Protocol/Remote/Minion.pm
M src/perl/embed.auto.c

Fix reference list when backup.info is reconstructed in expire command.

commit   : 50eb062e0e7a6aac326e411ed28e0c8aa397bf01    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Nov 2019 09:34:19 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Nov 2019 09:34:19 -0500    

Click here for diff

Adding a manifest to backup.info was migrated to C in 4e4d1f41 but deduplication of the references was missed leading to a reference for every file being added to backup.info.  
  
Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual.  
  
This unlikely combination of events means this is probably not a problem in the field.  

M doc/xml/release.xml
M src/info/infoBackup.c
M test/src/module/info/infoBackupTest.c

Set archive-check option in manifest correctly when offline.

commit   : 686b6f91da4cfdbc8708b0f08ced460c10e4326c    
  
author   : David Steele <[email protected]>    
date     : Thu, 28 Nov 2019 08:27:21 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 28 Nov 2019 08:27:21 -0500    

Click here for diff

Archive check does not run when in offline backup mode but the option was set to true in the manifest.  It's harmless since these options are informational only but it could cause confusion when debugging.  

M lib/pgBackRest/Backup/Backup.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Remove obsolete call to perlExec() for stand-alone expire command.

commit   : 5506e5de27b1a61015f6e18d5582ca81f696e30f    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Nov 2019 15:09:07 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Nov 2019 15:09:07 -0500    

Click here for diff

This was missed in ecae5e34.  

M src/main.c

Remove unused reference to pgBackRest::Db.

commit   : 4683557c54d2f2511fd754ae7c0c5e34ac1e5ba4    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Nov 2019 17:22:30 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Nov 2019 17:22:30 -0500    

Click here for diff

M lib/pgBackRest/Archive/Common.pm
M src/perl/embed.auto.c

Remove obsolete Perl archive code.

commit   : 158e439689c8247ba2e53da4bfe65934e391b690    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Nov 2019 17:16:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Nov 2019 17:16:45 -0500    

Click here for diff

This should have been removed in a1c13a50 but was missed.  

D lib/pgBackRest/Archive/Get/File.pm
M lib/pgBackRest/Protocol/Remote/Minion.pm
M src/perl/embed.auto.c
M test/define.yaml
D test/lib/pgBackRestTest/Module/Command/CommandArchiveGetPerlTest.pm

Update integration tests in real/all that use test points.

commit   : 82df7e6f3bf53409b3c3a15eff7bbc33c13197f8    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Nov 2019 11:32:12 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Nov 2019 11:32:12 -0500    

Click here for diff

Test points are not supported by the new C code so these will be replaced with unit tests.  
  
The fact that the tests still pass even when the changes aren't made mid-backup (except application_name) shows how weak they were in the first place.  
  
Even so, this does represent a regression in (soon to be be removed) Perl coverage.  

M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Update missing manifest warning in BackupInfo.

commit   : b145c72b5c27ad299ad1fa7dcd84f45fb0797e0f    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 08:51:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 08:51:28 -0500    

Click here for diff

This brings the Perl message in line with C to reduce expect log churn.  

M lib/pgBackRest/Backup/Info.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log

Remove exclusions once they have been tested in mock/all.

commit   : 8800f32ad9b6b9b1351afd4280cc8dfe1ada2d92    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 08:35:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 08:35:26 -0500    

Click here for diff

The exclusions no longer have any effect after a restore and just add noise to the expect log.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Update integration tests in mock/all that use test points.

commit   : 9856fef5861fbf054007233f78582d3584c03565    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 07:48:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 07:48:52 -0500    

Click here for diff

Test points will not be available in the C code so update these tests as best as possible without using them.  
  
This represents a loss of coverage for the Perl code (soon to be removed) which will be made up in the C code with unit tests.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Remove start/stop --force integration tests in mock/all.

commit   : 3cd45a7411ce67630915c89cf5c69a37265d0e07    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 07:45:58 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 07:45:58 -0500    

Click here for diff

These tests require test points which are not being implemented in the C code.  
  
This functionality is fully tested in the command/control unit tests so integration tests are no longer required.  

M test/expect/mock-all-001.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Update Perl page checksum expression.

commit   : 01aefc563dbabc7c7b1b7dbed27716692898597f    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 07:37:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Nov 2019 07:37:09 -0500    

Click here for diff

This expression determines which files contain page checksums but it was also including the directory above the relation directories.  In a real PostgreSQL installation this not a problem because these directories don't contain any files.  
  
However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no.  
  
Update the expression to document the change and avoid churn in the expect logs later.  

M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Fix comment typo.

commit   : 18e43c5955217838ad55e68e176b83a9187533ec    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Nov 2019 09:24:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Nov 2019 09:24:52 -0500    

Click here for diff

M src/command/backup/file.c

Add hostId to protocolLocalGet().

commit   : cace54151f3dc9f686b99923f1dbd1c010ab84f4    
  
author   : David Steele <[email protected]>    
date     : Sat, 23 Nov 2019 10:32:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 23 Nov 2019 10:32:57 -0500    

Click here for diff

Previously this function was only creating locals that talked to the repository.  Backup will need to be able to talk to multiple PostgreSQL hosts.  

M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/restore/restore.c
M src/protocol/helper.c
M src/protocol/helper.h
M test/src/module/protocol/protocolTest.c

Add protocolStorageType*() to manage protocol storage types.

commit   : ab65ffdfacf47b6d182cbf6461d49a76c0cc8b00    
  
author   : David Steele <[email protected]>    
date     : Sat, 23 Nov 2019 10:22:11 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 23 Nov 2019 10:22:11 -0500    

Click here for diff

Abstract the string representation of storage types that are passed over the protocol layer.  

M src/Makefile.in
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/remote/protocol.c
M test/define.yaml
M test/src/module/protocol/protocolTest.c

Only install specific lcov version when required.

commit   : a4b9440d354c3cf95f7ca3aa68dc691c10c65e21    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 19:25:49 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 19:25:49 -0500    

Click here for diff

Installing lcov 1.14 everywhere turned out to be a problem just as using 1.13 on Ubuntu 19.04 was.  
  
Since we primarily use Ubuntu 18.04 for coverage testing and reporting, we definitely want to make sure that works.  So, revert to using the default packaged lcov except when specified otherwise in VmTest.pm.  
  
PostgreSQL minor version releases are also included since all containers have been rebuilt.  

M doc/release.pl
M test/Vagrantfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm

Revert "Forbid % character in parameters."

commit   : 52a3ba6b6f25443b5ed456969b794f63a809a606    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 15:18:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 15:18:56 -0500    

Click here for diff

The issue "fixed" in f01aa586 was caused by treating all strings as format strings while logging, which was fixed in 0c05df45.  
  
Revert because there no longer seems a reason for the extra logic, and it was only partially applied, i.e. not to env vars, command-line options, or config options.  

M src/config/parse.c
M test/src/module/config/parseTest.c

Fix walPath() when CWD is / and path is relative.

commit   : 381aecae4eeb15c538318e95c3101fa07a6712d4    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 14:30:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 14:30:56 -0500    

Click here for diff

The function would return a // prefix in this case, which works fine but looks odd while debugging.  

M src/command/archive/common.c
M test/src/module/command/archiveCommonTest.c

Add _FMT() logging macro variants.

commit   : 0c05df458287c77df1188a68bbd74e145ed0f059    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 13:33:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 13:33:26 -0500    

Click here for diff

Using the same macros for formatted and unformatted logging had several disadvantages.  
  
First, the compiler was unable to verify the format string against the parameters.  
  
Second, legitimate % characters in messages were being interpreted as format characters with garbage output ensuing.  
  
Add _FMT() variants and update all call sites to use the correct variant.  

M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/check/check.c
M src/command/control/start.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/upgrade.c
M src/common/debug.h
M src/common/exit.c
M src/common/io/http/client.c
M src/common/io/tls/client.c
M src/common/log.c
M src/common/log.h
M src/config/load.c
M src/config/parse.c
M src/db/helper.c
M src/info/infoBackup.c
M src/storage/s3/storage.c
M test/src/module/common/logTest.c

Add missing linefeeds.

commit   : 99edcca55b3b8b1d144bdeb32c4d98c8ee13bf34    
  
author   : David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 09:18:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 22 Nov 2019 09:18:24 -0500    

Click here for diff

M src/storage/s3/storage.c

Add missing static keyword.

commit   : 29ab06cf133b6a3d18d7bc39507f92a45927b2e8    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 17:30:18 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 17:30:18 -0500    

Click here for diff

M src/config/parse.c

Forbid % character in parameters.

commit   : f01aa5861d341a19d85ddf3d7307c250994fecd4    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 17:28:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 17:28:03 -0500    

Click here for diff

This character causes problems in C and in the shell if we try to output it in an error message.  
  
Forbid it completely and spell it out in error messages to avoid strange effects.  
  
There is likely a better way deal with the issue but this will do for now.  

M src/config/parse.c
M test/src/module/config/parseTest.c

Don't log secrets in cipherBlock module.

commit   : 6be9c7b63a61464d60d15133662eb44776401b1d    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 17:20:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 17:20:42 -0500    

Click here for diff

In practice there are lots of ways secrets can be leaked (e.g. info load, protocol) but at least remove this instance.  

M src/common/crypto/cipherBlock.c

Remove obsolete integration tests from mock/all.

commit   : c524ec4f95462bfbc1484e139d754280ec05e94c    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 16:06:27 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 16:06:27 -0500    

Click here for diff

The protocol timeout tests have been superceded by unit tests.  
  
The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto.  It didn't hurt anything but it did add 200ms to each test where it appeared.  
  
Catalog and control version tests were redundant.  The database version and system id tests covered the important code paths and the C code gets these values from a lookup table.  
  
Finally, fix an incomplete update to the backup.info file while munging for tests.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Safely initialize manifest object.

commit   : 53cd530bbf3921c9500963d12393344425574d08    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 12:09:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 12:09:24 -0500    

Click here for diff

Using a designated initializer is safer than zeroing the struct.  It is also better for debugging because Valgrind should be able to detect access to areas that are not initialized due to alignment.  

M src/info/manifest.c

Add manifestMove().

commit   : 270f9496e432109b243e8db5d497805f72c56d91    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 12:08:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 12:08:32 -0500    

Click here for diff

M src/Makefile.in
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/info/manifestTest.c

Rearrange manifest module.

commit   : c5a6631d2791ca5ae69a6178fac758b6c96f9781    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 11:44:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 11:44:40 -0500    

Click here for diff

Put functions with related functions, move getters above the helper functions, and rename manifestPgPath() to manifestPathPg().  

M src/command/info/info.c
M src/command/restore/restore.c
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/info/manifestTest.c

Allow storageInfo() to operate outside the base storage path.

commit   : 9f71a019c815af58bdd10ca2ceaaf6f1506fb560    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 10:55:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 10:55:03 -0500    

Click here for diff

It is occasionally useful to get information about a file outside of the base storage path.  storageLocal() can be used in some cases but when the storage is remote is doesn't seem worth creating a separate storage object for adhoc info requests.  
  
storageInfo() is a read-only operation so this seems pretty safe.  The noPathEnforce parameter will make auditing exceptions easy.  

M src/storage/storage.c
M src/storage/storage.h
M test/src/module/storage/posixTest.c

Allow adhoc enforcement in storagePath().

commit   : d3b1897625c5b200724fb087cedd975fce07672f    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 10:34:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 10:34:32 -0500    

Click here for diff

The ability to disable enforcement (i.e., the requested absolute path is within the storage path) globally will be removed after the Perl migration.  
  
The feature will still be needed occasionally so allow it in an adhoc fashion.  

M src/storage/storage.c
M src/storage/storage.h
M test/src/module/storage/posixTest.c

Rename storagePath() to storageP() in places where it was missed.

commit   : e1dad720a126e2145718ccc78614727fa67cd619    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 10:21:35 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 10:21:35 -0500    

Click here for diff

Correct this since it will be enforced in a subsequent patch.  

M libc/xs/storage/storage.xs
M src/command/check/check.c
M src/command/expire/expire.c
M src/command/restore/file.c
M src/perl/libc.auto.c
M test/src/module/storage/remoteTest.c

Process . in strPathAbsolute().

commit   : cef9f0f37f33fe46ffcb229535a82479700b8de2    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 09:40:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 09:40:15 -0500    

Click here for diff

A . in a link will always lead to an error since the destination will be inside PGDATA.  However, it is accepted symlink syntax so it's better to resolve it and get the correct error message.  
  
Also, we may have other uses for this function in the future.  

M src/common/type/string.c
M test/src/module/common/typeStringTest.c

Add contributor.

commit   : a6fc0bf2ca778844448885192f11d17bf933b551    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 08:15:37 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Nov 2019 08:15:37 -0500    

Click here for diff

M doc/xml/release.xml

Fix inverted page checksum load/save.

commit   : 3d2c0b0f3bcd4fbae0409a374f13d1aebb3bba81    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Nov 2019 14:24:58 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Nov 2019 14:24:58 -0500    

Click here for diff

This was not caught before because the on-disk format was correct even though the representation in memory was inverted.  

M src/info/manifest.c

Fix lcov build in Vagrantfile.

commit   : 63c4c148361c75ec53d2d6b53a3ec69389b925c3    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Nov 2019 20:52:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Nov 2019 20:52:01 -0500    

Click here for diff

-q was being instead of -s for silent mode which caused the build to fail.  

M test/Vagrantfile

Remove --force option from stanza-create documentation.

commit   : 5f034714453f28a04d8e48828e8ee53a61401c4e    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Nov 2019 15:56:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Nov 2019 15:56:13 -0500    

Click here for diff

This should have been removed when the support for the option was removed in c7333190.  
  
The option cannot be removed entirely because we don't want to error in the case where --force was specified but the stanza is valid.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M src/config/define.auto.c

Markdown update missed in 1db9e3b1.

commit   : c5ee56a724fe6c49b7e3496090ff1bf6d73e33f8    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Nov 2019 10:34:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Nov 2019 10:34:14 -0500    

Click here for diff

M CODING.md

Fix typo.

commit   : 20162ed3feec8d3de5de8e2cca7bb065a47bcc67    
  
author   : David Fetter <[email protected]>    
date     : Tue, 19 Nov 2019 10:32:18 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Nov 2019 10:32:18 -0500    

Click here for diff

M CODING.md
M doc/xml/coding.xml

Use variable parameter macros to make the storage interface tidier.

commit   : 74317f0ab6024d6c3a2f9635677f26489786e166    
  
author   : David Steele <[email protected]>    
date     : Sun, 17 Nov 2019 17:35:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 17 Nov 2019 17:35:03 -0500    

Click here for diff

Many functions don't take optional parameters, so tacking the struct onto the end was pretty burdensome.  
  
Add macros to get the interface under a variety of circumstances to make this practical.  

M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.intern.h

Remove *MP() macros variants.

commit   : 1db9e3b144d886ec4159176312345d154b7178b8    
  
author   : David Steele <[email protected]>    
date     : Sun, 17 Nov 2019 15:10:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 17 Nov 2019 15:10:40 -0500    

Click here for diff

Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer.  
  
-Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.  

M CODING.md
M CONTRIBUTING.md
M doc/xml/coding.xml
M doc/xml/contributing.xml
M libc/xs/storage/storage.xs
M libc/xs/storage/storage.xsh
M src/Makefile.in
M src/command/archive/common.c
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
M src/command/backup/file.c
M src/command/check/check.c
M src/command/control/common.c
M src/command/control/start.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/restore/file.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/common/lock.c
M src/common/type/list.h
A src/common/type/param.h
M src/config/parse.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/manifest.c
M src/perl/libc.auto.c
M src/postgres/interface.c
M src/storage/info.h
M src/storage/remote/protocol.c
M src/storage/storage.c
M src/storage/storage.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/storageTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Use structs to pass optional parameters to storage drivers.

commit   : 5c65d88f6254176c257f0bf58ca5c9f59bb99177    
  
author   : David Steele <[email protected]>    
date     : Sun, 17 Nov 2019 14:17:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 17 Nov 2019 14:17:03 -0500    

Click here for diff

New optional parameters can be added without breaking the interface and this reduces noise in the function prototypes.  

M src/storage/posix/storage.c
M src/storage/posix/storage.intern.h
M src/storage/posix/write.c
M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.intern.h

Add storageInfoList() support to remote storage driver.

commit   : 09e129886e10a036e75493d55625f01c34e15d85    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:47:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:47:42 -0500    

Click here for diff

M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M test/define.yaml
M test/src/module/storage/remoteTest.c

Allow zero-length substrings to be extracted from the end of a string.

commit   : 26e1da82e7970ac03b8300da5f55ed7da1c2ab96    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:32:49 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:32:49 -0500    

Click here for diff

The previous assert was a bit overzealous and did not allow this case.  It's not very common but still occasionally useful.  

M src/common/type/string.c
M test/src/module/common/typeStringTest.c

Add storageInfo() support to remote storage driver.

commit   : 8a3de1e05a427bf40b7742ac476c01ef219f6b46    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:30:08 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:30:08 -0500    

Click here for diff

M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M test/src/module/storage/remoteTest.c

Store base path for remote storage locally.

commit   : 8d6a8c3bf0401705578d05902f8e7f871f3ab266    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:12:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:12:16 -0500    

Click here for diff

It wasn't practical for the main process to be ignorant of the remote path, and in any case knowing the path makes debugging easier.  
  
Pull the remote path when connecting and pass the result of local storagePath() to the remote when making calls.  

M src/Makefile.in
M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M src/storage/storage.c
M test/expect/mock-archive-002.log
M test/src/module/storage/remoteTest.c

Add facility for reading and writing adhoc protocol output.

commit   : 6827a13f3a4b1943769601acf34676bd1f689b4b    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:05:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 17:05:34 -0500    

Click here for diff

Pushing output through a JSON blob is not practical if the output is extremely large, e.g. a backup manifest with 100K+ files.  
  
Add read/write routines so that output can be returned in chunks but errors will still be detected.  

M src/common/type/buffer.c
M src/common/type/buffer.h
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/server.c
M src/protocol/server.h
M test/src/module/protocol/protocolTest.c

Add mock module to CentOS6/7 Travis CI testing.

commit   : 90e19d99bab6b448484865cb6b46fb786352009c    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 09:38:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Nov 2019 09:38:42 -0500    

Click here for diff

This was dropped at some point but is important for integration coverage.  

M .travis.yml

Add user-id/group-id to hrnReplaceKey().

commit   : c8db11e65bed37ec18d83b527c895b8b86c9f11b    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Nov 2019 17:50:12 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Nov 2019 17:50:12 -0500    

Click here for diff

M test/src/common/harnessTest.c
M test/src/common/harnessTest.h

Allow "null" in jsonToStr().

commit   : 53a2d04ab0e0c6d3ef75afcef1663492d9db98e7    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Nov 2019 17:48:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Nov 2019 17:48:25 -0500    

Click here for diff

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Allow cipherPassSub() to be set in Info without the constructor.

commit   : 64ddc163c95b6bc87a7589462b1b16ca70cb08bc    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Nov 2019 10:50:17 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Nov 2019 10:50:17 -0500    

Click here for diff

This value is not always known when the object is created so allow it to be set later.  

M src/info/info.c
M src/info/info.h

Allow trailing / for relative paths in strPathAbsolute().

commit   : 48e8942e86b1471568116dc7fe604878ca9afd70    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Nov 2019 08:53:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Nov 2019 08:53:15 -0500    

Click here for diff

The trailing / does nothing but is nevertheless valid syntax.  

M src/common/type/string.c
M test/src/module/common/typeStringTest.c

Update varNew*() calls to Variant constant macros.

commit   : 83ab272171887092df85704e27ddfec22c9f335f    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Nov 2019 16:50:58 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Nov 2019 16:50:58 -0500    

Click here for diff

The constants created by the macros are faster and use less memory.  

M src/command/backup/pageChecksum.c
M src/storage/remote/protocol.c

Filter logged command options based on the command definition.

commit   : 3b879c2cb3cfdbcd03dad1d6cc898052f8d8863d    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Nov 2019 16:48:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Nov 2019 16:48:41 -0500    

Click here for diff

Previously, options were being filtered based on what was currently valid.  For chained commands (e.g. backup then expire) some options may be valid for the first command but not the second.  
  
Filter based on the command definition rather than what is currently valid to avoid logging options that are not valid for subsequent commands.  This reduces the number of options logged and will hopefully help avoid confusion and expect log churn.  

M src/command/command.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/commandTest.c

Modify InfoBackupData struct to use time_t for backup start/stop times.

commit   : c5b76d213bc24d703e7b973df171a2b01595dabc    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 12 Nov 2019 17:05:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Nov 2019 17:05:09 -0500    

Click here for diff

The uint64_t types worked but were not consistent with how timestamps are handled in other parts of the code.  

M src/command/info/info.c
M src/info/infoBackup.c
M src/info/infoBackup.h

Begin v2.20 development.

commit   : 6f65dde8d39436a31f90030e9b3cef5cf10d04e3    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Nov 2019 15:56:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Nov 2019 15:56:28 -0500    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h