pgBackRest 2.21 (upcoming) commit log

commit   : d3f717c89208ef132e75b634ad2e91efbda25a04    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 7 Dec 2019 09:26:51 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 7 Dec 2019 09:26:51 -0500    

Click here for diff

Since there is only one driver that supports (or is likely to support) links (Posix), require the path feature to make logic in the code simpler.  
  
The checks are added just in case another driver supports links.  

M src/storage/storage.c

Fix backup labels in mock/all resume integration tests.

commit   : e632c605250438a9e06f133b55c88e578ea709cd    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 6 Dec 2019 11:48:41 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 6 Dec 2019 11:48:41 -0500    

Click here for diff

These were not getting updated to match the directory name when the manifests were copied.  
  
The Perl code didn't care but the C code expects labels to be set correctly.  

M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Allocate one extra process for backup from standby.

commit   : 5395aa0c92febadbf0c71f93799877cfe7268b6b    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 5 Dec 2019 22:34:38 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 5 Dec 2019 22:34:38 -0500    

Click here for diff

The connection to the primary is process-max + 1 so make space for it.  

M src/protocol/helper.c

Add functions to get the substring found by regExpMatch().

commit   : b2d82bd2484dbe3dee12479efb6e9aceba5aac57    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 4 Dec 2019 19:43:26 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 4 Dec 2019 19:43:26 -0500    

Click here for diff

For now this is only used in testing but there are places where it could be useful in the core code.  
  
Even if that turns out not to be true, it doesn't seem worth implementing a new version in testing just to capture a few values that we already have.  

M src/common/regExp.c
M src/common/regExp.h
M test/src/module/common/regExpTest.c

Add asserts to ensure hostId > 0.

commit   : 053af2f54db0eaaaca3ac67cbdd68ba1b701048a    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 4 Dec 2019 19:31:39 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 4 Dec 2019 19:31:39 -0500    

Click here for diff

hostId is 1-based (e.g. pg1-*) so it should always be > 0.  

M src/protocol/helper.c

Add missing header.

commit   : d1f49825b11ba88ac2024734c4c8a1c961f05854    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 2 Dec 2019 11:54:45 -0500    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 2 Dec 2019 11:54:45 -0500    

Click here for diff

M src/postgres/interface.h

Use more general error code when tablespace linked into PGDATA.

commit   : 8dfe0e48e25725978738ecb3fec24e5ce2e9076b    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 2 Dec 2019 10:49:25 -0500    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 2 Dec 2019 10:49:25 -0500    

Click here for diff

The specific error code was not that useful since we also test the error message which contains details of the link error.  

M build/error.yaml
M lib/pgBackRest/Common/ExceptionAuto.pm
M lib/pgBackRest/Manifest.pm
M src/common/error.auto.c
M src/common/error.auto.h
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Add flag to dbGet() to require a standby.

commit   : 33a63aae503adf8d7f3f4430ee6478b845d38e2c    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 2 Dec 2019 07:39:42 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 2 Dec 2019 07:39:42 -0500    

Click here for diff

This is needed from backup from standby functionality.  

M src/command/check/check.c
M src/command/stanza/common.c
M src/db/helper.c
M src/db/helper.h
M test/src/module/command/checkTest.c
M test/src/module/db/dbTest.c

Error in remote command when stop file exists.

commit   : 28116918ff3a00b133d74d524df452e29398082d    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 2 Dec 2019 07:35:36 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 2 Dec 2019 07:35:36 -0500    

Click here for diff

This duplicates the Perl functionality.  

M src/Makefile.in
M src/command/remote/remote.c
M test/src/module/command/remoteTest.c

Reduce the scope of mock/all exclusion tests.

commit   : fc291b6f28753329afdd7b65b115f90d31569df1    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 17:47:47 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 17:47:47 -0500    

Click here for diff

Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Make MCV return false when a boolean tie.

commit   : d15ed338217667f0bd7ffff3eb9f5e24052477fd    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 16:32:21 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 16:32:21 -0500    

Click here for diff

This is to maintain compatibility with the older Perl code that returned the lowest sorted order item in a tie.  
  
For other datatypes the C code returns the same value, often enough at least to not cause churn in the expect tests.  

M src/common/type/mcv.c
M test/src/module/common/typeMcvTest.c

Add PostgreSQL version where backup from standby is supported.

commit   : ab0974cc8ea0cff70a9f680ff8e1dc10b1ba3c48    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 16:30:44 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 16:30:44 -0500    

Click here for diff

M src/postgres/version.h

Add PostgreSQL file and path names used for building manifests.

commit   : 218698088f0e4d5a978b8f4c8df6b114980889b9    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 16:28:33 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 16:28:33 -0500    

Click here for diff

M src/postgres/interface.c
M src/postgres/interface.h

Add pgLsnName() and pgXactPath().

commit   : 56ee321a9565e7c1e076c58744e0cfc408801e28    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 15:49:34 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 1 Dec 2019 15:49:34 -0500    

Click here for diff

M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

Add storage feature flags.

commit   : bc83354fb53a1ba8a86366478c5be871775dd04a    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 29 Nov 2019 11:29:16 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 29 Nov 2019 11:29:16 -0500    

Click here for diff

Even though storagePathSync() is a noop when path sync is not supported, it is useful to know in advance if the function will do anything so add the storageFeaturePathSync flag.  
  
The storageFeatureSymLink and storageFeatureHardLink flags are currently informational only since links are not yet implemented in the storage interface.  

M src/storage/posix/storage.c
M src/storage/storage.c
M src/storage/storage.h

Remove unused reference to use pgBackRest::Backup::File.

commit   : 0933aeb1ad954f8c7b2ed6d4e160d7e132ada8b7    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 29 Nov 2019 10:56:13 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 29 Nov 2019 10:56:13 -0500    

Click here for diff

M lib/pgBackRest/Protocol/Remote/Minion.pm
M src/perl/embed.auto.c

Fix reference list when backup.info is reconstructed in expire command.

commit   : 50eb062e0e7a6aac326e411ed28e0c8aa397bf01    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 28 Nov 2019 09:34:19 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 28 Nov 2019 09:34:19 -0500    

Click here for diff

Adding a manifest to backup.info was migrated to C in 4e4d1f41 but deduplication of the references was missed leading to a reference for every file being added to backup.info.  
  
Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual.  
  
This unlikely combination of events means this is probably not a problem in the field.  

M doc/xml/release.xml
M src/info/infoBackup.c
M test/src/module/info/infoBackupTest.c

Set archive-check option in manifest correctly when offline.

commit   : 686b6f91da4cfdbc8708b0f08ced460c10e4326c    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 28 Nov 2019 08:27:21 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 28 Nov 2019 08:27:21 -0500    

Click here for diff

Archive check does not run when in offline backup mode but the option was set to true in the manifest.  It's harmless since these options are informational only but it could cause confusion when debugging.  

M lib/pgBackRest/Backup/Backup.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Remove obsolete call to perlExec() for stand-alone expire command.

commit   : 5506e5de27b1a61015f6e18d5582ca81f696e30f    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 27 Nov 2019 15:09:07 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 27 Nov 2019 15:09:07 -0500    

Click here for diff

This was missed in ecae5e34.  

M src/main.c

Remove unused reference to pgBackRest::Db.

commit   : 4683557c54d2f2511fd754ae7c0c5e34ac1e5ba4    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 26 Nov 2019 17:22:30 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 26 Nov 2019 17:22:30 -0500    

Click here for diff

M lib/pgBackRest/Archive/Common.pm
M src/perl/embed.auto.c

Remove obsolete Perl archive code.

commit   : 158e439689c8247ba2e53da4bfe65934e391b690    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 26 Nov 2019 17:16:45 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 26 Nov 2019 17:16:45 -0500    

Click here for diff

This should have been removed in a1c13a50 but was missed.  

D lib/pgBackRest/Archive/Get/File.pm
M lib/pgBackRest/Protocol/Remote/Minion.pm
M src/perl/embed.auto.c
M test/define.yaml
D test/lib/pgBackRestTest/Module/Command/CommandArchiveGetPerlTest.pm

Update integration tests in real/all that use test points.

commit   : 82df7e6f3bf53409b3c3a15eff7bbc33c13197f8    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 26 Nov 2019 11:32:12 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 26 Nov 2019 11:32:12 -0500    

Click here for diff

Test points are not supported by the new C code so these will be replaced with unit tests.  
  
The fact that the tests still pass even when the changes aren't made mid-backup (except application_name) shows how weak they were in the first place.  
  
Even so, this does represent a regression in (soon to be be removed) Perl coverage.  

M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Update missing manifest warning in BackupInfo.

commit   : b145c72b5c27ad299ad1fa7dcd84f45fb0797e0f    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 08:51:28 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 08:51:28 -0500    

Click here for diff

This brings the Perl message in line with C to reduce expect log churn.  

M lib/pgBackRest/Backup/Info.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log

Remove exclusions once they have been tested in mock/all.

commit   : 8800f32ad9b6b9b1351afd4280cc8dfe1ada2d92    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 08:35:26 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 08:35:26 -0500    

Click here for diff

The exclusions no longer have any effect after a restore and just add noise to the expect log.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Update integration tests in mock/all that use test points.

commit   : 9856fef5861fbf054007233f78582d3584c03565    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 07:48:52 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 07:48:52 -0500    

Click here for diff

Test points will not be available in the C code so update these tests as best as possible without using them.  
  
This represents a loss of coverage for the Perl code (soon to be removed) which will be made up in the C code with unit tests.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Remove start/stop --force integration tests in mock/all.

commit   : 3cd45a7411ce67630915c89cf5c69a37265d0e07    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 07:45:58 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 07:45:58 -0500    

Click here for diff

These tests require test points which are not being implemented in the C code.  
  
This functionality is fully tested in the command/control unit tests so integration tests are no longer required.  

M test/expect/mock-all-001.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Update Perl page checksum expression.

commit   : 01aefc563dbabc7c7b1b7dbed27716692898597f    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 07:37:09 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 25 Nov 2019 07:37:09 -0500    

Click here for diff

This expression determines which files contain page checksums but it was also including the directory above the relation directories.  In a real PostgreSQL installation this not a problem because these directories don't contain any files.  
  
However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no.  
  
Update the expression to document the change and avoid churn in the expect logs later.  

M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Fix comment typo.

commit   : 18e43c5955217838ad55e68e176b83a9187533ec    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 24 Nov 2019 09:24:52 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 24 Nov 2019 09:24:52 -0500    

Click here for diff

M src/command/backup/file.c

Add hostId to protocolLocalGet().

commit   : cace54151f3dc9f686b99923f1dbd1c010ab84f4    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 23 Nov 2019 10:32:57 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 23 Nov 2019 10:32:57 -0500    

Click here for diff

Previously this function was only creating locals that talked to the repository.  Backup will need to be able to talk to multiple PostgreSQL hosts.  

M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/restore/restore.c
M src/protocol/helper.c
M src/protocol/helper.h
M test/src/module/protocol/protocolTest.c

Add protocolStorageType*() to manage protocol storage types.

commit   : ab65ffdfacf47b6d182cbf6461d49a76c0cc8b00    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 23 Nov 2019 10:22:11 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 23 Nov 2019 10:22:11 -0500    

Click here for diff

Abstract the string representation of storage types that are passed over the protocol layer.  

M src/Makefile.in
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/remote/protocol.c
M test/define.yaml
M test/src/module/protocol/protocolTest.c

Only install specific lcov version when required.

commit   : a4b9440d354c3cf95f7ca3aa68dc691c10c65e21    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 19:25:49 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 19:25:49 -0500    

Click here for diff

Installing lcov 1.14 everywhere turned out to be a problem just as using 1.13 on Ubuntu 19.04 was.  
  
Since we primarily use Ubuntu 18.04 for coverage testing and reporting, we definitely want to make sure that works.  So, revert to using the default packaged lcov except when specified otherwise in VmTest.pm.  
  
PostgreSQL minor version releases are also included since all containers have been rebuilt.  

M doc/release.pl
M test/Vagrantfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm

Revert "Forbid % character in parameters."

commit   : 52a3ba6b6f25443b5ed456969b794f63a809a606    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 15:18:56 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 15:18:56 -0500    

Click here for diff

The issue "fixed" in f01aa586 was caused by treating all strings as format strings while logging, which was fixed in 0c05df45.  
  
Revert because there no longer seems a reason for the extra logic, and it was only partially applied, i.e. not to env vars, command-line options, or config options.  

M src/config/parse.c
M test/src/module/config/parseTest.c

Fix walPath() when CWD is / and path is relative.

commit   : 381aecae4eeb15c538318e95c3101fa07a6712d4    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 14:30:56 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 14:30:56 -0500    

Click here for diff

The function would return a // prefix in this case, which works fine but looks odd while debugging.  

M src/command/archive/common.c
M test/src/module/command/archiveCommonTest.c

Add _FMT() logging macro variants.

commit   : 0c05df458287c77df1188a68bbd74e145ed0f059    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 13:33:26 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 13:33:26 -0500    

Click here for diff

Using the same macros for formatted and unformatted logging had several disadvantages.  
  
First, the compiler was unable to verify the format string against the parameters.  
  
Second, legitimate % characters in messages were being interpreted as format characters with garbage output ensuing.  
  
Add _FMT() variants and update all call sites to use the correct variant.  

M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/check/check.c
M src/command/control/start.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/upgrade.c
M src/common/debug.h
M src/common/exit.c
M src/common/io/http/client.c
M src/common/io/tls/client.c
M src/common/log.c
M src/common/log.h
M src/config/load.c
M src/config/parse.c
M src/db/helper.c
M src/info/infoBackup.c
M src/storage/s3/storage.c
M test/src/module/common/logTest.c

Add missing linefeeds.

commit   : 99edcca55b3b8b1d144bdeb32c4d98c8ee13bf34    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 09:18:24 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 22 Nov 2019 09:18:24 -0500    

Click here for diff

M src/storage/s3/storage.c

Add missing static keyword.

commit   : 29ab06cf133b6a3d18d7bc39507f92a45927b2e8    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 17:30:18 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 17:30:18 -0500    

Click here for diff

M src/config/parse.c

Forbid % character in parameters.

commit   : f01aa5861d341a19d85ddf3d7307c250994fecd4    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 17:28:03 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 17:28:03 -0500    

Click here for diff

This character causes problems in C and in the shell if we try to output it in an error message.  
  
Forbid it completely and spell it out in error messages to avoid strange effects.  
  
There is likely a better way deal with the issue but this will do for now.  

M src/config/parse.c
M test/src/module/config/parseTest.c

Don't log secrets in cipherBlock module.

commit   : 6be9c7b63a61464d60d15133662eb44776401b1d    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 17:20:42 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 17:20:42 -0500    

Click here for diff

In practice there are lots of ways secrets can be leaked (e.g. info load, protocol) but at least remove this instance.  

M src/common/crypto/cipherBlock.c

Remove obsolete integration tests from mock/all.

commit   : c524ec4f95462bfbc1484e139d754280ec05e94c    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 16:06:27 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 16:06:27 -0500    

Click here for diff

The protocol timeout tests have been superceded by unit tests.  
  
The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto.  It didn't hurt anything but it did add 200ms to each test where it appeared.  
  
Catalog and control version tests were redundant.  The database version and system id tests covered the important code paths and the C code gets these values from a lookup table.  
  
Finally, fix an incomplete update to the backup.info file while munging for tests.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Safely initialize manifest object.

commit   : 53cd530bbf3921c9500963d12393344425574d08    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 12:09:24 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 12:09:24 -0500    

Click here for diff

Using a designated initializer is safer than zeroing the struct.  It is also better for debugging because Valgrind should be able to detect access to areas that are not initialized due to alignment.  

M src/info/manifest.c

Add manifestMove().

commit   : 270f9496e432109b243e8db5d497805f72c56d91    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 12:08:32 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 12:08:32 -0500    

Click here for diff

M src/Makefile.in
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/info/manifestTest.c

Rearrange manifest module.

commit   : c5a6631d2791ca5ae69a6178fac758b6c96f9781    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 11:44:40 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 11:44:40 -0500    

Click here for diff

Put functions with related functions, move getters above the helper functions, and rename manifestPgPath() to manifestPathPg().  

M src/command/info/info.c
M src/command/restore/restore.c
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/info/manifestTest.c

Allow storageInfo() to operate outside the base storage path.

commit   : 9f71a019c815af58bdd10ca2ceaaf6f1506fb560    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 10:55:03 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 10:55:03 -0500    

Click here for diff

It is occasionally useful to get information about a file outside of the base storage path.  storageLocal() can be used in some cases but when the storage is remote is doesn't seem worth creating a separate storage object for adhoc info requests.  
  
storageInfo() is a read-only operation so this seems pretty safe.  The noPathEnforce parameter will make auditing exceptions easy.  

M src/storage/storage.c
M src/storage/storage.h
M test/src/module/storage/posixTest.c

Allow adhoc enforcement in storagePath().

commit   : d3b1897625c5b200724fb087cedd975fce07672f    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 10:34:32 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 10:34:32 -0500    

Click here for diff

The ability to disable enforcement (i.e., the requested absolute path is within the storage path) globally will be removed after the Perl migration.  
  
The feature will still be needed occasionally so allow it in an adhoc fashion.  

M src/storage/storage.c
M src/storage/storage.h
M test/src/module/storage/posixTest.c

Rename storagePath() to storageP() in places where it was missed.

commit   : e1dad720a126e2145718ccc78614727fa67cd619    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 10:21:35 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 10:21:35 -0500    

Click here for diff

Correct this since it will be enforced in a subsequent patch.  

M libc/xs/storage/storage.xs
M src/command/check/check.c
M src/command/expire/expire.c
M src/command/restore/file.c
M src/perl/libc.auto.c
M test/src/module/storage/remoteTest.c

Process . in strPathAbsolute().

commit   : cef9f0f37f33fe46ffcb229535a82479700b8de2    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 09:40:15 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 09:40:15 -0500    

Click here for diff

A . in a link will always lead to an error since the destination will be inside PGDATA.  However, it is accepted symlink syntax so it's better to resolve it and get the correct error message.  
  
Also, we may have other uses for this function in the future.  

M src/common/type/string.c
M test/src/module/common/typeStringTest.c

Add contributor.

commit   : a6fc0bf2ca778844448885192f11d17bf933b551    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 08:15:37 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 21 Nov 2019 08:15:37 -0500    

Click here for diff

M doc/xml/release.xml

Fix inverted page checksum load/save.

commit   : 3d2c0b0f3bcd4fbae0409a374f13d1aebb3bba81    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 20 Nov 2019 14:24:58 -0500    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 20 Nov 2019 14:24:58 -0500    

Click here for diff

This was not caught before because the on-disk format was correct even though the representation in memory was inverted.  

M src/info/manifest.c

Fix lcov build in Vagrantfile.

commit   : 63c4c148361c75ec53d2d6b53a3ec69389b925c3    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 19 Nov 2019 20:52:01 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 19 Nov 2019 20:52:01 -0500    

Click here for diff

-q was being instead of -s for silent mode which caused the build to fail.  

M test/Vagrantfile

Remove --force option from stanza-create documentation.

commit   : 5f034714453f28a04d8e48828e8ee53a61401c4e    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 19 Nov 2019 15:56:13 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 19 Nov 2019 15:56:13 -0500    

Click here for diff

This should have been removed when the support for the option was removed in c7333190.  
  
The option cannot be removed entirely because we don't want to error in the case where --force was specified but the stanza is valid.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M src/config/define.auto.c

Markdown update missed in 1db9e3b1.

commit   : c5ee56a724fe6c49b7e3496090ff1bf6d73e33f8    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 19 Nov 2019 10:34:14 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 19 Nov 2019 10:34:14 -0500    

Click here for diff

M CODING.md

Fix typo.

commit   : 20162ed3feec8d3de5de8e2cca7bb065a47bcc67    
  
author   : David Fetter <david@fetter.org>    
date     : Tue, 19 Nov 2019 10:32:18 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 19 Nov 2019 10:32:18 -0500    

Click here for diff

M CODING.md
M doc/xml/coding.xml

Use variable parameter macros to make the storage interface tidier.

commit   : 74317f0ab6024d6c3a2f9635677f26489786e166    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 17 Nov 2019 17:35:03 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 17 Nov 2019 17:35:03 -0500    

Click here for diff

Many functions don't take optional parameters, so tacking the struct onto the end was pretty burdensome.  
  
Add macros to get the interface under a variety of circumstances to make this practical.  

M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.intern.h

Remove *MP() macros variants.

commit   : 1db9e3b144d886ec4159176312345d154b7178b8    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 17 Nov 2019 15:10:40 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 17 Nov 2019 15:10:40 -0500    

Click here for diff

Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer.  
  
-Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.  

M CODING.md
M CONTRIBUTING.md
M doc/xml/coding.xml
M doc/xml/contributing.xml
M libc/xs/storage/storage.xs
M libc/xs/storage/storage.xsh
M src/Makefile.in
M src/command/archive/common.c
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
M src/command/backup/file.c
M src/command/check/check.c
M src/command/control/common.c
M src/command/control/start.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/restore/file.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/common/lock.c
M src/common/type/list.h
A src/common/type/param.h
M src/config/parse.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/manifest.c
M src/perl/libc.auto.c
M src/postgres/interface.c
M src/storage/info.h
M src/storage/remote/protocol.c
M src/storage/storage.c
M src/storage/storage.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/storageTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Use structs to pass optional parameters to storage drivers.

commit   : 5c65d88f6254176c257f0bf58ca5c9f59bb99177    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 17 Nov 2019 14:17:03 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 17 Nov 2019 14:17:03 -0500    

Click here for diff

New optional parameters can be added without breaking the interface and this reduces noise in the function prototypes.  

M src/storage/posix/storage.c
M src/storage/posix/storage.intern.h
M src/storage/posix/write.c
M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.intern.h

Add storageInfoList() support to remote storage driver.

commit   : 09e129886e10a036e75493d55625f01c34e15d85    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:47:42 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:47:42 -0500    

Click here for diff

M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M test/define.yaml
M test/src/module/storage/remoteTest.c

Allow zero-length substrings to be extracted from the end of a string.

commit   : 26e1da82e7970ac03b8300da5f55ed7da1c2ab96    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:32:49 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:32:49 -0500    

Click here for diff

The previous assert was a bit overzealous and did not allow this case.  It's not very common but still occasionally useful.  

M src/common/type/string.c
M test/src/module/common/typeStringTest.c

Add storageInfo() support to remote storage driver.

commit   : 8a3de1e05a427bf40b7742ac476c01ef219f6b46    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:30:08 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:30:08 -0500    

Click here for diff

M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M test/src/module/storage/remoteTest.c

Store base path for remote storage locally.

commit   : 8d6a8c3bf0401705578d05902f8e7f871f3ab266    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:12:16 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:12:16 -0500    

Click here for diff

It wasn't practical for the main process to be ignorant of the remote path, and in any case knowing the path makes debugging easier.  
  
Pull the remote path when connecting and pass the result of local storagePath() to the remote when making calls.  

M src/Makefile.in
M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M src/storage/storage.c
M test/expect/mock-archive-002.log
M test/src/module/storage/remoteTest.c

Add facility for reading and writing adhoc protocol output.

commit   : 6827a13f3a4b1943769601acf34676bd1f689b4b    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:05:34 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 17:05:34 -0500    

Click here for diff

Pushing output through a JSON blob is not practical if the output is extremely large, e.g. a backup manifest with 100K+ files.  
  
Add read/write routines so that output can be returned in chunks but errors will still be detected.  

M src/common/type/buffer.c
M src/common/type/buffer.h
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/server.c
M src/protocol/server.h
M test/src/module/protocol/protocolTest.c

Add mock module to CentOS6/7 Travis CI testing.

commit   : 90e19d99bab6b448484865cb6b46fb786352009c    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 09:38:42 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 16 Nov 2019 09:38:42 -0500    

Click here for diff

This was dropped at some point but is important for integration coverage.  

M .travis.yml

Add user-id/group-id to hrnReplaceKey().

commit   : c8db11e65bed37ec18d83b527c895b8b86c9f11b    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 15 Nov 2019 17:50:12 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 15 Nov 2019 17:50:12 -0500    

Click here for diff

M test/src/common/harnessTest.c
M test/src/common/harnessTest.h

Allow "null" in jsonToStr().

commit   : 53a2d04ab0e0c6d3ef75afcef1663492d9db98e7    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 15 Nov 2019 17:48:25 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 15 Nov 2019 17:48:25 -0500    

Click here for diff

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Allow cipherPassSub() to be set in Info without the constructor.

commit   : 64ddc163c95b6bc87a7589462b1b16ca70cb08bc    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 15 Nov 2019 10:50:17 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 15 Nov 2019 10:50:17 -0500    

Click here for diff

This value is not always known when the object is created so allow it to be set later.  

M src/info/info.c
M src/info/info.h

Allow trailing / for relative paths in strPathAbsolute().

commit   : 48e8942e86b1471568116dc7fe604878ca9afd70    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 15 Nov 2019 08:53:15 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 15 Nov 2019 08:53:15 -0500    

Click here for diff

The trailing / does nothing but is nevertheless valid syntax.  

M src/common/type/string.c
M test/src/module/common/typeStringTest.c

Update varNew*() calls to Variant constant macros.

commit   : 83ab272171887092df85704e27ddfec22c9f335f    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 14 Nov 2019 16:50:58 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 14 Nov 2019 16:50:58 -0500    

Click here for diff

The constants created by the macros are faster and use less memory.  

M src/command/backup/pageChecksum.c
M src/storage/remote/protocol.c

Filter logged command options based on the command definition.

commit   : 3b879c2cb3cfdbcd03dad1d6cc898052f8d8863d    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 14 Nov 2019 16:48:41 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 14 Nov 2019 16:48:41 -0500    

Click here for diff

Previously, options were being filtered based on what was currently valid.  For chained commands (e.g. backup then expire) some options may be valid for the first command but not the second.  
  
Filter based on the command definition rather than what is currently valid to avoid logging options that are not valid for subsequent commands.  This reduces the number of options logged and will hopefully help avoid confusion and expect log churn.  

M src/command/command.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/commandTest.c

Modify InfoBackupData struct to use time_t for backup start/stop times.

commit   : c5b76d213bc24d703e7b973df171a2b01595dabc    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Tue, 12 Nov 2019 17:05:09 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 12 Nov 2019 17:05:09 -0500    

Click here for diff

The uint64_t types worked but were not consistent with how timestamps are handled in other parts of the code.  

M src/command/info/info.c
M src/info/infoBackup.c
M src/info/infoBackup.h

Begin v2.20 development.

commit   : 6f65dde8d39436a31f90030e9b3cef5cf10d04e3    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 12 Nov 2019 15:56:28 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 12 Nov 2019 15:56:28 -0500    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h