pgBackRest 2.12 commit log

v2.12: C Implementation of Archive Push

commit   : 4e57b689165cffd8c686f0f36d0c5dccb553b3e1    
  
author   : David Steele <[email protected]>    
date     : Thu, 11 Apr 2019 09:14:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 11 Apr 2019 09:14:22 -0400    

Click here for diff

IMPORTANT NOTE: The new TLS/SSL implementation forbids dots in S3 bucket names per RFC-2818. This security fix is required for compliant hostname verification.  
  
Bug Fixes:  
  
* Fix issues when a path option is / terminated. (Reported by Marc Cousin.)  
* Fix issues when log-level-file=off is set for the archive-get command. (Reported by Brad Nicholson.)  
* Fix C code to recognize host:port option format like Perl does. (Reported by Kyle Nevins.)  
* Fix issues with remote/local command logging options.  
  
Improvements:  
  
* The archive-push command is implemented entirely in C.  
* Increase process-max limit to 999. (Suggested by Rakshitha-BR.)  
* Improve error message when an S3 bucket name contains dots.  
  
Documentation Improvements:  
  
* Clarify that S3-compatible object stores are supported. (Suggested by Magnus Hagander.)  

M README.md
M doc/RELEASE.md
M doc/resource/exe.cache
M doc/xml/index.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml

Fix C code to recognize host:port format like Perl does.

commit   : df12cbb1625ed61c74d43ae3a308445f943b0570    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Apr 2019 17:48:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Apr 2019 17:48:34 -0400    

Click here for diff

This was not an intentional feature in Perl, but it works, so it makes sense to implement the same syntax in C.  
  
This is a break from other places where a -port option is explicitly supplied, so it may make sense to support both styles going forward.  This commit does not address that, however.  
  
Reported by Kyle Nevins.  

M doc/xml/release.xml
M src/config/config.c
M src/config/config.h
M src/storage/driver/s3/storage.c
M src/storage/helper.c
M test/define.yaml
M test/src/module/config/configTest.c
M test/src/module/storage/s3Test.c

Fix compile flag accidentally removed in 5ee8388f.

commit   : 3aa521fed0838021209b4be36849feb2c7e40a2e    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Apr 2019 13:37:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Apr 2019 13:37:24 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/JobTest.pm

Add process id to C archive-get and archive-push logging.

commit   : 25cea0bd0a22de2f83a2ded786e2facc86ff4b10    
  
author   : David Steele <[email protected]>    
date     : Tue, 9 Apr 2019 11:08:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 9 Apr 2019 11:08:27 -0400    

Click here for diff

This was missed in the original migration.  There was no functional issue, but logging the process ids is useful for debugging.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c

Add process id to ProtocolParallelJob.

commit   : 8c202c77dac74d919962f69acb279b583b4272ff    
  
author   : David Steele <[email protected]>    
date     : Tue, 9 Apr 2019 11:01:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 9 Apr 2019 11:01:35 -0400    

Click here for diff

This exposes the process that performed to job to the caller for logging purposes.  

M doc/xml/release.xml
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M test/src/module/protocol/protocolTest.c

Allow process id in C logging

commit   : 4ace7edbd9e849d5a7a5d8b20e08c0c3e2098a02    
  
author   : David Steele <[email protected]>    
date     : Tue, 9 Apr 2019 10:54:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 9 Apr 2019 10:54:36 -0400    

Click here for diff

The default process id in C logging has always been zero.  This should have been updated when multi-processing was introduced in C, but it was missed.  

M doc/xml/release.xml
M src/common/exit.c
M src/common/log.c
M src/common/log.h
M test/src/module/common/logTest.c

Improve error message when an S3 bucket name contains dots.

commit   : 6099729e922893312e001a3394db43d0c2b341ad    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Apr 2019 19:38:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Apr 2019 19:38:06 -0400    

Click here for diff

The Perl lib we have been using for TLS allows dots in wildcards, but this is forbidden by RFC-2818.  The new TLS implementation in C forbids this pattern, just as PostgreSQL and curl do.  
  
However, this does present a problem for users who have been using bucket names with dots in older versions of pgBackRest.  Since this limitation exists for security reasons there appears to be no option but to take a hard line and do our best to notify the user of the issue as clearly as possible.  

M doc/xml/release.xml
M src/config/load.c
M test/src/module/config/loadTest.c

Fix issue setting log-level-file=off for the archive-get command.

commit   : 21c83eea59f5fb4d3c366a1e9782d26356801675    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Apr 2019 17:21:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Apr 2019 17:21:20 -0400    

Click here for diff

This problem was not specific to archive-get, but that was the only place it was expressing in the last release.  The new archive-push was also affected.  
  
The issue was with daemon processes that had closed all their file descriptors.  When exec'ing and setting up pipes to communicate with a child process the dup2() function created file descriptors that overlapped with the first descriptor (stdout) that was being duped into.  This descriptor was subsequently closed and wackiness ensued.  
  
If logging was enabled (the default) that increased all the file descriptors by one and everything worked.  
  
Fix this by checking if the file descriptor to be closed is the same one being dup'd into.  This solution may not be generally applicable but it works fine in this case.  
  
Reported by Brad Nicholson.  

M doc/xml/release.xml
M src/common/exec.c
M test/src/module/common/execTest.c

Close log file before exec().

commit   : 8ac422dca95c8761fc57450ed74061ff286ff0c3    
  
author   : David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 21:09:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 21:09:29 -0400    

Click here for diff

The log file should be closed before exec() so the child process does not retain a reference to it.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/common/exec.c
M src/common/log.c
M src/common/log.h
M test/src/module/common/logTest.c

Fix typo in error message.

commit   : ade20e5c657f1a05bbe00c1f693346bfadd8b795    
  
author   : David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 19:11:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 19:11:20 -0400    

Click here for diff

M src/common/exec.c

Skip coverage for macros with numbers in their name.

commit   : 9f3425f6dd2892721935103ff3f75555b7408dd5    
  
author   : David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 18:47:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 18:47:04 -0400    

Click here for diff

This was an oversight in the old regular expression.  

M doc/xml/release.xml
M test/src/lcov.conf

Increase process-max limit to 999.

commit   : c72a94cfec64096d0f18fa1385b1347918be4fdb    
  
author   : David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 18:20:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 18:20:03 -0400    

Click here for diff

Recent hardware has outstripped the old default so increase it to something that should last for a while.  
  
Suggested by Rakshitha-BR.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/config/define.auto.c

Allow three-digits process IDs in logging.

commit   : 9a7eab9428b8ce34b31b878bd5e6af4f842ce8d7    
  
author   : David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 18:12:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 7 Apr 2019 18:12:07 -0400    

Click here for diff

This is required to support process-max > 99 or else there will be formatting/alignment issues in the logs.  

M doc/xml/release.xml
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Config/Config.pm
M src/common/exec.c
M src/common/log.c
M src/common/log.h
M src/config/load.c
M src/config/parse.c
M src/perl/embed.auto.c
M test/src/common/harnessLog.c
M test/src/module/common/logTest.c

Add assert missed in 1b486847.

commit   : 2a85df95c121528d826e30526a2fd251638624e7    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 3 Apr 2019 17:48:45 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 Apr 2019 17:48:45 +0100    

Click here for diff

Contributed by Cynthia Shang.  

M src/command/archive/push/push.c

Add memContextSwitch() missed in 5bdaa35f.

commit   : 1e5f2713b2936bd3252e0081a979a9d84eba19d0    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 3 Apr 2019 16:40:19 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 Apr 2019 16:40:19 +0100    

Click here for diff

Contributed by Cynthia Shang.  

M src/command/archive/common.c

Add test and asserts missed in abba2bd1.

commit   : 5504f286eb6765d3131f2ff99cb2b464a02c4589    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 3 Apr 2019 12:32:56 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 Apr 2019 12:32:56 +0100    

Click here for diff

Contributed by Cynthia Shang.  

M src/common/type/stringList.c
M test/src/module/common/typeStringTest.c

Add assert missed in 1b486847.

commit   : be58b8fee799d29cc5f1df0147bce59b09982208    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 3 Apr 2019 12:04:44 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 Apr 2019 12:04:44 +0100    

Click here for diff

Contributed by Cynthia Shang.  

M src/command/archive/push/file.c

Clarify that S3-compatible object stores are supported.

commit   : e97b8241a59f4dd56ea374b00b3fb9905a1febc3    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 Apr 2019 11:40:12 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 Apr 2019 11:40:12 +0100    

Click here for diff

The documentation mentioned Amazon S3 frequently but failed to mention that other S3-compatible object stores are also supported.  
  
Tone down the specific mentions of Amazon S3 and replace them with "S3-compatible object store" when appropriate.  
  
Suggested by Magnus Hagander.  

M doc/xml/index.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml

Add forkSafe() to handle fork errors.

commit   : 4c63279a19a6baa1f6854eee59cc804f6f89cb23    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 Apr 2019 13:50:02 +0100    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 Apr 2019 13:50:02 +0100    

Click here for diff

fork() rarely fails but we should definitely detect when it does so.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/common/fork.c
M src/common/fork.h
M test/src/common/harnessFork.h
M test/src/module/common/forkTest.c

The archive-push command is implemented entirely in C.

commit   : 1b48684713220b3436b24b266e543196824e974f    
  
author   : David Steele <[email protected]>    
date     : Fri, 29 Mar 2019 13:26:33 +0000    
  
committer: David Steele <[email protected]>    
date     : Fri, 29 Mar 2019 13:26:33 +0000    

Click here for diff

This new implementation should behave exactly like the old Perl code with the exception of updated log messages.  
  
Remove as much of the Perl code as possible without breaking other commands.  

M doc/xml/release.xml
D lib/pgBackRest/Archive/Base.pm
M lib/pgBackRest/Archive/Common.pm
D lib/pgBackRest/Archive/Push/Async.pm
D lib/pgBackRest/Archive/Push/File.pm
D lib/pgBackRest/Archive/Push/Push.pm
M lib/pgBackRest/Main.pm
M lib/pgBackRest/Protocol/Helper.pm
M lib/pgBackRest/Protocol/Local/Minion.pm
M lib/pgBackRest/Protocol/Remote/Minion.pm
M src/Makefile
A src/command/archive/push/file.c
A src/command/archive/push/file.h
A src/command/archive/push/protocol.c
A src/command/archive/push/protocol.h
M src/command/archive/push/push.c
M src/command/archive/push/push.h
M src/command/local/local.c
M src/main.c
M src/perl/embed.auto.c
M src/postgres/interface.h
M src/protocol/parallelJob.c
M test/define.yaml
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Module/Command/CommandArchiveCommonPerlTest.pm
D test/lib/pgBackRestTest/Module/Command/CommandArchivePushPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Protocol/ProtocolHelperPerlTest.pm
M test/src/module/command/archivePushTest.c
M test/src/module/perl/execTest.c

Add locking capability to the remote command.

commit   : 251dbede8ff8a7206986106b8df8b08257cd7ebd    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Mar 2019 21:14:06 +0000    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Mar 2019 21:14:06 +0000    

Click here for diff

When a repository server is configured, commands that modify the repository acquire a remote lock as well as a local lock for extra protection against multiple writers.  
  
Instead of the custom logic used in Perl, make remote locking part of the command configuration.  
  
This also means that the C remote needs the stanza since it is used to construct the lock name.  We may need to revisit this at a later date.  

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/command/remote/remote.c
M src/config/config.auto.c
M src/config/config.c
M src/config/config.h
M src/protocol/helper.c
M src/protocol/server.c
M src/protocol/server.h
M test/src/module/command/remoteTest.c
M test/src/module/config/configTest.c
M test/src/module/protocol/protocolTest.c

Add protocolKeepAlive() to send noops to all remotes.

commit   : 7db8cedd68477dd511bb8bdf8b94b2d2925470dd    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Mar 2019 20:59:28 +0000    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Mar 2019 20:59:28 +0000    

Click here for diff

While the local processes are doing their jobs the remote connection from the main process may timeout.  
  
Send occasional noops to ensure that doesn't happen.  

M doc/xml/release.xml
M src/protocol/helper.c
M src/protocol/helper.h
M test/src/module/protocol/protocolTest.c

Add TEST_64BIT() macro to detect 64-bit platforms.

commit   : f8cbf5d78cbacfef253f8e75983e403e10049df0    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Mar 2019 20:52:23 +0000    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Mar 2019 20:52:23 +0000    

Click here for diff

This may not be the best way to detect 64-bit platforms but it seems to be working fine so far.  
  
Create a macro to make it clearer what is being done and to make it easier to change the implementation.  

M doc/xml/release.xml
M test/src/common/harnessTest.h
M test/src/module/common/memContextTest.c
M test/src/module/common/typeStringTest.c

Fix missing semicolon in code example.

commit   : 1fa103a4d23742af0fb16214a74b974cbe875724    
  
author   : David Steele <[email protected]>    
date     : Wed, 27 Mar 2019 18:14:00 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 27 Mar 2019 18:14:00 +0200    

Click here for diff

M test/src/common/harnessFork.h

Set WAL long header flag in test function missed in e938a892.

commit   : 3181cd0ab8f09cbeca87d97e670e0901feb412a3    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Mar 2019 18:46:44 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Mar 2019 18:46:44 +0200    

Click here for diff

This was missed because the unit tests were reusing a buffer without resetting it to zero, so this flag ended up still set when the test function was called.  
  
This was not a live issue since it only expressed in tests and this code is not used in master yet.  

M src/postgres/interface/version.intern.h
M test/src/module/postgres/interfaceTest.c

Add missing parens.

commit   : 7364d859da8b4ee67e4b0d76e9c0d08a15fa546d    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Mar 2019 16:00:15 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Mar 2019 16:00:15 +0200    

Click here for diff

M src/common/io/tls/client.c

Build test harness with the same warnings as code being tested.

commit   : 5ee8388f482adc52129a9960bdb077e8c054edb0    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Mar 2019 08:20:55 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Mar 2019 08:20:55 +0200    

Click here for diff

The test harness was not being built with warnings which caused some wackiness with an improperly structured switch.  Just use the same warnings as the code being tested.  
  
Also enable warnings on code that is not directly being tested since other code modules are frequently modified during testing.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessTls.c

Fix comment content and formatting.

commit   : f709334851cfbe254ce9f13266adba4504e9af64    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 23:59:56 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 23:59:56 +0200    

Click here for diff

M src/storage/driver/s3/storage.c

Add strLstMergeAnti() for merge anti-joins.

commit   : abba2bd13269b3d51802cb7bfba18a0abae47342    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 20:35:20 +0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 20:35:20 +0400    

Click here for diff

We deal with some pretty big lists in archive-push so a nested-loop anti-join looked like it would not be efficient enough.  
  
This merge anti-join should do the trick even though both lists must be sorted first.  

M doc/xml/release.xml
M src/common/type/stringList.c
M src/common/type/stringList.h
M test/define.yaml
M test/src/module/common/typeStringTest.c

Add storagePg()/storagePgWrite() to storage helper.

commit   : 538e2c7474394f1f24bb0899bc7d85bf7c68bd5a    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 09:45:36 +0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 09:45:36 +0400    

Click here for diff

Add read/write storage helper functions for the PostgreSQL data directory.  

M doc/xml/release.xml
M src/storage/helper.c
M src/storage/helper.h
M test/src/module/storage/posixTest.c

Allow warnings to be written by archiveAsyncStatusOkWrite().

commit   : 444b4f86403088f24883ffe0b940a6402159bfc0    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 09:17:15 +0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 09:17:15 +0400    

Click here for diff

Migrate this feature to the C code since archive-push will need it.  

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M test/src/module/command/archiveCommonTest.c

Use a single file to handle global errors in async archiving.

commit   : 8820d695747b40b73ead6d32a408e7d2804bd192    
  
author   : David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 08:12:38 +0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 25 Mar 2019 08:12:38 +0400    

Click here for diff

The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed.  On retry each of these error files would be removed, and if the error was persistent, they would then be recreated.  In a busy environment this could mean tens or hundreds of thousands of files.  
  
Another issue was that the error files could not be written until a list of WAL files to process had been generated.  This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log.  
  
Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file.  This reduces churn and allows more errors to be reported directly to PostgreSQL.  

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c

Update default WAL segment size comment.

commit   : 1f6f3f673e49a2dff4dff465d183a2f6928138e6    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Mar 2019 18:29:20 +0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Mar 2019 18:29:20 +0400    

Click here for diff

M src/postgres/interface.c

Remove unused infoArchiveCheckPg() function.

commit   : 1fff60ad2ad56f303080af666ed76e82dbb171b7    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Mar 2019 14:34:45 +0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Mar 2019 14:34:45 +0400    

Click here for diff

The idea was to use this function for archive-push, but in the end archive-push required two very different versions of the function.  

M doc/xml/release.xml
M src/info/infoArchive.c
M src/info/infoArchive.h
M test/src/module/info/infoArchiveTest.c

Add constants for .ok/.error status extensions.

commit   : 8560db5c42fdc07de44b86daf76ad195282f67d7    
  
author   : David Steele <[email protected]>    
date     : Sun, 24 Mar 2019 13:54:05 +0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 24 Mar 2019 13:54:05 +0400    

Click here for diff

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c

Refactor PostgreSQL interface to remove most code duplication.

commit   : 7cf7373761cb83253a502e8d0af4f925c86a7944    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Mar 2019 21:11:36 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Mar 2019 21:11:36 +0400    

Click here for diff

Having a copy per version worked well until it was time to add new features or modify existing functions.  Then it was necessary to modify every version and try to keep them all in sync.  
  
Consolidate all the PostgreSQL types into a single file using #if for type versions.  Many types do not change or change infrequently so this cuts down on duplication.  In addition, it is far easier to see what has changed when a new version is added.  
  
Use macros to write the interface functions.  There is still duplication here since some changes require a new copy of the macro, but it is far less than before.  

M doc/xml/release.xml
M src/Makefile
M src/postgres/interface.c
D src/postgres/interface/v083.auto.c
M src/postgres/interface/v083.c
D src/postgres/interface/v083.h
D src/postgres/interface/v084.auto.c
M src/postgres/interface/v084.c
D src/postgres/interface/v084.h
D src/postgres/interface/v090.auto.c
M src/postgres/interface/v090.c
D src/postgres/interface/v090.h
D src/postgres/interface/v091.auto.c
M src/postgres/interface/v091.c
D src/postgres/interface/v091.h
D src/postgres/interface/v092.auto.c
M src/postgres/interface/v092.c
D src/postgres/interface/v092.h
D src/postgres/interface/v093.auto.c
M src/postgres/interface/v093.c
D src/postgres/interface/v093.h
D src/postgres/interface/v094.auto.c
M src/postgres/interface/v094.c
D src/postgres/interface/v094.h
D src/postgres/interface/v095.auto.c
M src/postgres/interface/v095.c
D src/postgres/interface/v095.h
D src/postgres/interface/v096.auto.c
M src/postgres/interface/v096.c
D src/postgres/interface/v096.h
D src/postgres/interface/v100.auto.c
M src/postgres/interface/v100.c
D src/postgres/interface/v100.h
D src/postgres/interface/v110.auto.c
M src/postgres/interface/v110.c
D src/postgres/interface/v110.h
A src/postgres/interface/version.auto.h
A src/postgres/interface/version.h
A src/postgres/interface/version.intern.h
M src/postgres/version.h
M test/code-count/file-type.yaml
M test/src/module/postgres/interfaceTest.c

Add WAL info to PostgreSQL interface.

commit   : e938a89250bd6a6512d4d6b5217a9750c848ca49    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Mar 2019 19:44:06 +0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Mar 2019 19:44:06 +0400    

Click here for diff

This allows the WAL header to be read for any supported version on PostgreSQL.  

M doc/xml/release.xml
M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/interface/v083.auto.c
M src/postgres/interface/v083.c
M src/postgres/interface/v083.h
M src/postgres/interface/v084.auto.c
M src/postgres/interface/v084.c
M src/postgres/interface/v084.h
M src/postgres/interface/v090.auto.c
M src/postgres/interface/v090.c
M src/postgres/interface/v090.h
M src/postgres/interface/v091.auto.c
M src/postgres/interface/v091.c
M src/postgres/interface/v091.h
M src/postgres/interface/v092.auto.c
M src/postgres/interface/v092.c
M src/postgres/interface/v092.h
M src/postgres/interface/v093.auto.c
M src/postgres/interface/v093.c
M src/postgres/interface/v093.h
M src/postgres/interface/v094.auto.c
M src/postgres/interface/v094.c
M src/postgres/interface/v094.h
M src/postgres/interface/v095.auto.c
M src/postgres/interface/v095.c
M src/postgres/interface/v095.h
M src/postgres/interface/v096.auto.c
M src/postgres/interface/v096.c
M src/postgres/interface/v096.h
M src/postgres/interface/v100.auto.c
M src/postgres/interface/v100.c
M src/postgres/interface/v100.h
M src/postgres/interface/v110.auto.c
M src/postgres/interface/v110.c
M src/postgres/interface/v110.h
M test/define.yaml
M test/src/module/postgres/interfaceTest.c

Add missing static keyword.

commit   : 1c9645d4166165f70bf837b7d053415d78a216f2    
  
author   : David Steele <[email protected]>    
date     : Tue, 19 Mar 2019 19:20:38 +0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 19 Mar 2019 19:20:38 +0400    

Click here for diff

M src/command/archive/get/file.c

Remove redundant documentation from PostgreSQL interface files and clarify ambiguous function names.

commit   : 5c732e992904bc7f3201c119787d84cf05a8997d    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Mar 2019 22:10:25 +0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Mar 2019 22:10:25 +0400    

Click here for diff

Move the documentation to postgres/interface.c so it can be updated without having to update N source files.  
  
The "is" function was not very specific so rename to "controlIs".  

M doc/xml/release.xml
M src/perl/embed.auto.c
M src/postgres/interface.c
M src/postgres/interface/v083.c
M src/postgres/interface/v083.h
M src/postgres/interface/v084.c
M src/postgres/interface/v084.h
M src/postgres/interface/v090.c
M src/postgres/interface/v090.h
M src/postgres/interface/v091.c
M src/postgres/interface/v091.h
M src/postgres/interface/v092.c
M src/postgres/interface/v092.h
M src/postgres/interface/v093.c
M src/postgres/interface/v093.h
M src/postgres/interface/v094.c
M src/postgres/interface/v094.h
M src/postgres/interface/v095.c
M src/postgres/interface/v095.h
M src/postgres/interface/v096.c
M src/postgres/interface/v096.h
M src/postgres/interface/v100.c
M src/postgres/interface/v100.h
M src/postgres/interface/v110.c
M src/postgres/interface/v110.h

Use restore command for remote performances tests.

commit   : e26d510d0c1d8157ef5d9149bed9c2902ec29ba2    
  
author   : David Steele <[email protected]>    
date     : Sun, 17 Mar 2019 22:11:35 +0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 17 Mar 2019 22:11:35 +0400    

Click here for diff

Since archive-push is being moved to C, the Perl remote will no longer work with that command.  
  
Eventually this module will need to be rewritten in C, but for now just use the restore command which is planned to be migrated last.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Module/Performance/PerformanceIoTest.pm

Add file write to the S3 storage driver.

commit   : 856a369b863fb134ec249b7036eea70ba56d89ac    
  
author   : David Steele <[email protected]>    
date     : Sun, 17 Mar 2019 22:00:54 +0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 17 Mar 2019 22:00:54 +0400    

Click here for diff

Now that repositories are writable the storage drivers that don't yet support file writes need to be updated to do so.  
  
Note that the part size for multi-part upload has not been defined as a proper constant.  This will become an option in the near future so it doesn't seem worth creating a constant that we might then forget to remove.  

M doc/xml/release.xml
M src/Makefile
A src/storage/driver/s3/fileWrite.c
A src/storage/driver/s3/fileWrite.h
M src/storage/driver/s3/storage.c
M src/storage/driver/s3/storage.h
M src/storage/helper.c
M test/define.yaml
M test/src/module/storage/s3Test.c

Add document creation to XML objects.

commit   : 7193738288e7c25bba526f5133fad9d4cf70d4c0    
  
author   : David Steele <[email protected]>    
date     : Sun, 17 Mar 2019 21:38:07 +0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 17 Mar 2019 21:38:07 +0400    

Click here for diff

The xml objects only exposed read methods of the underlying libxml2.  
  
This worked for S3 commands that only received data but to send data we need to be able to create XML documents from scratch.  
  
Add the ability to create empty documents and add nodes and contents.  

M doc/xml/release.xml
M src/common/type/xml.c
M src/common/type/xml.h
M test/src/module/common/typeXmlTest.c

Add a note regarding verifying checkpoint against replay position.

commit   : fc974626cd060271c39480ea5ac960b8db903699    
  
author   : David Steele <[email protected]>    
date     : Sun, 17 Mar 2019 08:35:40 +0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 17 Mar 2019 08:35:40 +0400    

Click here for diff

M lib/pgBackRest/Db.pm

Add file write to the remote storage driver.

commit   : 8ebc6d6c34e7ccc7e4d07f1321fd4b1dce1579c7    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 21:50:19 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 21:50:19 +0400    

Click here for diff

Now that repositories are writable the storage drivers that don't yet support file writes need to be updated to do so.  

M doc/xml/release.xml
M src/Makefile
M src/storage/driver/remote/fileRead.c
A src/storage/driver/remote/fileWrite.c
A src/storage/driver/remote/fileWrite.h
M src/storage/driver/remote/protocol.c
M src/storage/driver/remote/protocol.h
M src/storage/driver/remote/storage.c
M test/define.yaml
M test/src/module/storage/remoteTest.c

Move WAL path prefix logic into walPath().

commit   : 2d386cd26672ed6c1ff2ca20465b7c7505cf9de6    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 16:14:10 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 16:14:10 +0400    

Click here for diff

This logic is used by both archive-push and archive-get.  

M doc/xml/release.xml
M src/Makefile
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M test/define.yaml
M test/src/module/command/archiveCommonTest.c

Add storageRepoWrite() to storage helper.

commit   : 12273a10346a5520ce556a19df7b0cd4f9be74be    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 15:48:50 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 15:48:50 +0400    

Click here for diff

Writable repos are allowed with the addition of the CIFS driver, so add a helper function.  

M doc/xml/release.xml
M src/storage/helper.c
M src/storage/helper.h
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Make notion of current PostgreSQL info ID in C align with Perl.

commit   : 66c2f4cd2e43f05d65e06f3c89094fbdf702d60d    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 15:27:38 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 15:27:38 +0400    

Click here for diff

The C code was assuming that the current PostgreSQL version in archive.info/backup.info was the most recent item in the history, but this is not always the case with some stanza-upgrade scenarios.  If a cluster is restored from before the upgrade and stanza-upgrade is run again, it will revert db-id to the original history item.  
  
Instead, load db-id from the db section explicitly as the Perl code does.  
  
This did not affect archive-get since it does a reverse scan through the history versions and does not rely on the current version.  

M doc/xml/release.xml
M src/info/infoArchive.c
M src/info/infoPg.c
M src/info/infoPg.h
M test/src/module/command/archiveGetTest.c

Fix issues with remote/local command logging options.

commit   : b2b2cf0511b326c695e825bee0c5f5faf648224c    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 15:00:02 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 15:00:02 +0400    

Click here for diff

Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that.  
  
Also, make sure that stderr is enabled at error level as it was on Perl.  This helps expose error information for debugging.  
  
For remotes, suppress log and lock paths since these are not applicable on remote hosts.  These options should be set in the local config if they need to be overridden.  

M doc/xml/release.xml
M src/protocol/helper.c
M test/define.yaml
M test/src/module/protocol/protocolTest.c

httpClientRequest() accepts a body parameter.

commit   : d377e926c806faa8d61744e27eacfa7bf610c445    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 13:07:43 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 13:07:43 +0400    

Click here for diff

None of our C HTTP requests have needed to output a body, but they will with the migration of archive-push.  
  
Also, add constants that are useful when POSTing/PUTing data.  

M doc/xml/release.xml
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/storage/driver/s3/storage.c
M test/src/module/common/ioHttpTest.c

Add httpHeaderDup().

commit   : 078df66e2c66ea992c423ffc92a789a4f036b622    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 12:51:37 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 12:51:37 +0400    

Click here for diff

Allow duplication of headers and updating of redactions.  

M doc/xml/release.xml
M src/common/io/http/header.c
M src/common/io/http/header.h
M test/src/module/common/ioHttpTest.c

Add hash size constants and extern hash type constant.

commit   : be670e25450a174f6652850af765805de1139263    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 12:32:54 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 12:32:54 +0400    

Click here for diff

The size constants are convenient for creating data structures of the proper size.  
  
The hash type constant must be extern'd so that results can be pulled from a filter.  

M doc/xml/release.xml
M src/common/crypto/hash.c
M src/common/crypto/hash.h

Add missing const to cryptoHashOne() and cryptoHashOneStr().

commit   : 76560b3cf0959c11b92e3b98c640f5e6070b1e7a    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 12:26:25 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 12:26:25 +0400    

Click here for diff

M doc/xml/release.xml
M src/common/crypto/hash.c
M src/common/crypto/hash.h

Fix incorrect buffer size used in cryptoHashOne().

commit   : 0eb4c2c8291ed0dc5c211594b5e85e1b31a9d84d    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 12:21:16 +0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Mar 2019 12:21:16 +0400    

Click here for diff

This was missing when bufUsed() was introduced.  
  
It is not currently a live issue, but becomes a problem in the new archive-push code where the entire buffer is not always used.  

M doc/xml/release.xml
M src/common/crypto/hash.c

Add command for removing all Docker containers to the Vagrantfile.

commit   : 9c1549585e0d8a4613954bbfc194dd0dc0a537a8    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 22:18:56 +0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 22:18:56 +0400    

Click here for diff

M test/Vagrantfile

Add cvtSSizeToZ() and debug macros.

commit   : b68d1d60a1532e9d1ef2703b043d538643d6f8cc    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 22:16:50 +0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 22:16:50 +0400    

Click here for diff

M doc/xml/release.xml
M src/common/debug.h
M src/common/type/convert.c
M src/common/type/convert.h
M test/src/module/common/typeConvertTest.c

Add missing log param and assert.

commit   : d16bff64cdcb5be6c6ac3b8f31d394c2393b7649    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 10:30:48 +0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 10:30:48 +0400    

Click here for diff

M src/storage/driver/posix/fileWrite.c

Improve comment, remove duplicated header and extraneous braces.

commit   : a6e219bc8bf1f6d93a24ba120fa9614d7b667ce3    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 10:25:24 +0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 10:25:24 +0400    

Click here for diff

M src/command/archive/get/file.c

Make strLstDup() null-tolerant.

commit   : 5554377632807a3e778273b9882a191e8a6edae6    
  
author   : David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 10:20:52 +0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 15 Mar 2019 10:20:52 +0400    

Click here for diff

Duping a NULL StringList without checking if it is NULL is a useful capability.  

M doc/xml/release.xml
M src/common/type/stringList.c
M test/src/module/common/typeStringTest.c

Fix issues when a path option is / terminated.

commit   : 9382283586da4794d3585dbf6680c3a2f3d10a33    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Mar 2019 13:48:33 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Mar 2019 13:48:33 +0400    

Click here for diff

This condition was not being properly checked for in the C code and it caused problems in the info command, at the very least.  
  
Instead of applying a local fix, introduce a new path option type that will rigorously check the format of any incoming paths.  
  
Reported by Marc Cousin.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M src/config/config.c
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.c
M src/perl/config.c
M src/perl/embed.auto.c
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/command/infoTest.c
M test/src/module/config/parseTest.c

Add separate archive-push-async command.

commit   : b8ebea6b1c3cf2a26c44f04c7552c66fd9acb44a    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Mar 2019 13:38:55 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Mar 2019 13:38:55 +0400    

Click here for diff

This command was previously forked off from the archive-push command which required a bit of artificial option and log manipulation.  
  
A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/xml/release.xml
M lib/pgBackRest/Archive/Push/Push.pm
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Main.pm
M src/Makefile
M src/command/archive/push/push.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/define.c
M src/main.c
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Command/CommandArchivePushPerlTest.pm
M test/src/module/command/archivePushTest.c

Add CIFS storage driver.

commit   : 982b47c5ecec475703860f52a0eabe7d501a96e2    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Mar 2019 13:28:33 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Mar 2019 13:28:33 +0400    

Click here for diff

This driver borrows heavily from the Posix driver.  
  
At this point the only difference is that CIFS does not allow explicit directory fsyncs so they need to be suppressed.  At some point the CIFS diver will also omit link support.  
  
With the addition of this driver repository storage is now writable.  

M doc/xml/release.xml
M src/Makefile
A src/storage/driver/cifs/storage.c
A src/storage/driver/cifs/storage.h
M src/storage/helper.c
M test/define.yaml
A test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c

Remove package patch merged to upstream.

commit   : 941dbb47313b25339d438e7a57975bcd88c65168    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Mar 2019 13:02:19 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Mar 2019 13:02:19 +0400    

Click here for diff

D test/patch/debian-package.patch

Begin v2.12 development.

commit   : cf5a5b7b9a97fb5907990812745d42ec4e3de7fb    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Mar 2019 10:43:35 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 11 Mar 2019 10:43:35 +0200    

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h