pgBackRest 2.22 commit log · Life with Open Source

15 Jan 2020, 13:21 by Yogesh Sharma

`v2.21: C Migration Complete`

commit   : 2c0ba0820d3e4adea31ef6b8ee201be653ef9579    
  
author   : David Steele <[email protected]>    
date     : Wed, 15 Jan 2020 13:21:52 -0700    
  
committer: David Steele <[email protected]>    
date     : Wed, 15 Jan 2020 13:21:52 -0700

Click here for diff

Bug Fixes:  
  
* Fix options being ignored by asynchronous commands. The asynchronous archive-get/archive-push processes were not loading options configured in command configuration sections, e.g. [global:archive-get]. (Reviewed by Cynthia Shang. Reported by Urs Kramer.)  
* Fix handling of \ in filenames. \ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading. Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.  
  
Features:  
  
* pgBackRest is now pure C.  
* Add pg-user option. Specifies the database user name when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior. (Contributed by Mike Palmiotto.)  
* Allow path-style URIs in S3 driver.  
  
Improvements:  
  
* The backup command is implemented entirely in C. (Reviewed by Cynthia Shang.)

M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/version.h
M test/code-count/file-type.yaml

`Fix options being ignored by asynchronous commands.`

commit   : 8d3710b2fe94c90783374d19a435bded1903fe1b    
  
author   : David Steele <[email protected]>    
date     : Wed, 15 Jan 2020 12:24:58 -0700    
  
committer: David Steele <[email protected]>    
date     : Wed, 15 Jan 2020 12:24:58 -0700

Click here for diff

The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc.  
  
Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section.  
  
Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/command.c
M src/command/remote/remote.c
M src/command/restore/restore.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/exec.c
M src/config/exec.h
M src/config/load.c
M src/config/parse.auto.c
M src/config/parse.c
M src/main.c
M src/protocol/helper.c
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/common/exitTest.c
M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c
M test/src/module/protocol/protocolTest.c

`Fix function return logging type.`

commit   : c43ec9d38c7cd543d7a832f1e362705658ca5c12    
  
author   : David Steele <[email protected]>    
date     : Wed, 15 Jan 2020 08:19:32 -0700    
  
committer: David Steele <[email protected]>    
date     : Wed, 15 Jan 2020 08:19:32 -0700

Click here for diff

This was working because the stack was being cleaned up by a try block at the call site.

M src/common/crypto/hash.c

`Clarify comment.`

commit   : f8a29c266e1e9b6e207fdd74478e3aca9b728cb9    
  
author   : David Steele <[email protected]>    
date     : Mon, 13 Jan 2020 20:36:42 -0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 13 Jan 2020 20:36:42 -0700

Click here for diff

M src/command/remote/remote.c

`Update comments in command/remote module.`

commit   : a7738ebba3cfd3a732c5de997a113746136133af    
  
author   : David Steele <[email protected]>    
date     : Mon, 13 Jan 2020 13:21:28 -0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 13 Jan 2020 13:21:28 -0700

Click here for diff

M test/src/module/command/remoteTest.c

`Fix misaligned braces.`

commit   : a969a860abea7384572a1244913bcfbd8851545d    
  
author   : David Steele <[email protected]>    
date     : Mon, 13 Jan 2020 11:13:37 -0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 13 Jan 2020 11:13:37 -0700

Click here for diff

M src/command/backup/backup.c

`Allow path-style URIs in S3 driver.`

commit   : fe263e87b1822b051ccea3fc0a8fea64f944dd7b    
  
author   : David Steele <[email protected]>    
date     : Sun, 12 Jan 2020 11:31:06 -0700    
  
committer: David Steele <[email protected]>    
date     : Sun, 12 Jan 2020 11:31:06 -0700

Click here for diff

Although path-style URIs have been deprecated by AWS, they may still be used with products like Minio because no additional DNS configuration is required.  
  
Path-style URIs must be explicitly enabled since it is not clear how they can be auto-detected reliably.  More importantly, faulty detection could cause regressions in current installations.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/storage/helper.c
M src/storage/s3/storage.c
M src/storage/s3/storage.h
M test/src/module/command/helpTest.c
M test/src/module/storage/s3Test.c

`Add string constants.`

commit   : 069345d33959eb7b2ff0f4ae2b2ec65023ac7186    
  
author   : David Steele <[email protected]>    
date     : Sun, 12 Jan 2020 11:00:10 -0700    
  
committer: David Steele <[email protected]>    
date     : Sun, 12 Jan 2020 11:00:10 -0700

Click here for diff

M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringz.h

`Add time to storage ls JSON output.`

commit   : 3f89ecf8d9b848c2ca063581bc75064d9d353255    
  
author   : David Steele <[email protected]>    
date     : Fri, 10 Jan 2020 09:39:33 -0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 10 Jan 2020 09:39:33 -0700

Click here for diff

Time is supported in all drivers with the update to S3 at 61538f93, so it is now possible to add time to the ls command and have it work on all repo types.

M src/command/storage/list.c
M test/src/module/command/storageTest.c

`Improve code that updates/removes pg options passed to a remote.`

commit   : 0fe7bb2ec4d6ba10af8f2a6fd736e2c4cbdb2af2    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Jan 2020 12:20:13 -0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Jan 2020 12:20:13 -0700

Click here for diff

The prior code was updating/removing hard-coded options but new options are added all the time and there was no indication that this code needed to be updated.  For example, dc1e7ca2 added the pg-user option but this code was not updated.  
  
Instead find the options to update/remove dynamically.  The new code uses prefixes, which is not perfect, but the patterns for pg options are pretty well established and this seems safer than the existing code.

M src/protocol/helper.c

`Update inaccurate comments.`

commit   : 4c8653fc8bc6e8b3070c59ba12af4e5f2131c21c    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Jan 2020 11:56:03 -0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Jan 2020 11:56:03 -0700

Click here for diff

M src/protocol/helper.c

`Fix comment typo.`

commit   : 2e11389ed4db29d530ec8e480134c44eab9a5896    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 9 Jan 2020 11:52:51 -0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Jan 2020 11:52:51 -0700

Click here for diff

M src/config/exec.h

`Make quoting in cfgExeParam() optional.`

commit   : 0c5c78e5e16b10f79dcfa023c6276a65ad87481d    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Jan 2020 09:23:15 -0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Jan 2020 09:23:15 -0700

Click here for diff

Parameter lists that are passed directly to exec*() do not need quoting when spaces are present.  Worse, the quotes will not be stripped and the option value will be garbled.  
  
Unfortunately this still does not fix all issues with quoting since we don't know how it might need to be escaped to work with SSH command configuration.  The answer seems to be to pass the options in the protocol layer but that's beyond the scope of this commit.

M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/restore/restore.c
M src/config/exec.c
M src/config/exec.h
M src/protocol/helper.c
M test/src/module/config/execTest.c

`Add internal remote-type option.`

commit   : 7de5ce23ad387602125f478ceec2c834703080d5    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Jan 2020 18:59:02 -0700    
  
committer: David Steele <[email protected]>    
date     : Wed, 8 Jan 2020 18:59:02 -0700

Click here for diff

This option was overloaded on the general type option but it makes sense to split this out since the meaning is pretty different.  
  
Rename the values to conform to current standards, i.e. pg and repo, now that the Perl code won't care anymore.

M build/lib/pgBackRestBuild/Config/Data.pm
M lib/pgBackRest/LibCAuto.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/remote/protocol.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

`Fix test log message to match pg-version parameter name.`

commit   : 7a1871c341c2ae530927bcf018792733530fc417    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Jan 2020 09:54:44 -0700    
  
committer: David Steele <[email protected]>    
date     : Wed, 8 Jan 2020 09:54:44 -0700

Click here for diff

It was confusing that this part of the log message did not match the parameter name, which made reproducing test failures from CI a little harder.

M test/lib/pgBackRestTest/Common/JobTest.pm

`Remove obsolete Perl code missed in f0ef73db.`

commit   : 8e1f78b4c82d3b4f7ecbdb053df1c8fd46bf3c1e    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Jan 2020 12:20:51 -0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Jan 2020 12:20:51 -0700

Click here for diff

M lib/pgBackRest/Db.pm

`Fix comment typos.`

commit   : a84ae6be04fc21c52c04284004836292089fa0e1    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Jan 2020 16:16:48 -0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 6 Jan 2020 16:16:48 -0700

Click here for diff

M src/storage/helper.c

`Parse dates in storageS3InfoList() and storageS3Info().`

commit   : 61538f932c58c3bb0afad5b6c16cf997604e3606    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Jan 2020 15:53:53 -0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 6 Jan 2020 15:53:53 -0700

Click here for diff

Previously dates were not being filled by these functions which was fine since dates were not used.  
  
We plan to use dates for the ls command plus it makes sense for the driver to be complete since it will be used as an example.

M doc/xml/release.xml
M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

`Add httpLastModifiedToTime() to parse HTTP last-modified header.`

commit   : d2fb4f977ceba499f134ad363ea3170f0d9ef5b9    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Jan 2020 15:24:49 -0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 6 Jan 2020 15:24:49 -0700

Click here for diff

M doc/xml/release.xml
M src/Makefile.in
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/common.c
M src/common/io/http/common.h
M test/define.yaml
M test/src/module/common/ioHttpTest.c

`Add basic time management functions.`

commit   : a08298ce1b9ee96a9f84b613848327f7f60eb063    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Jan 2020 15:18:52 -0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 6 Jan 2020 15:18:52 -0700

Click here for diff

These are similar to what mktime() and strptime() do but they ignore the local system timezone which saves having to munge the TZ env variable to do time conversions.

M doc/xml/release.xml
M src/common/time.c
M src/common/time.h
M test/define.yaml
M test/src/module/common/timeTest.c

`Update LICENSE.txt for 2020.`

commit   : fc47907fe32560e8d3086287023d6f727027a0a1    
  
author   : David Steele <[email protected]>    
date     : Fri, 3 Jan 2020 09:32:52 -0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 3 Jan 2020 09:32:52 -0700

Click here for diff

M LICENSE

`Remove command option from Perl db application name.`

commit   : 011bcb48697ddb56fe1396502a36254718f74dfa    
  
author   : David Steele <[email protected]>    
date     : Sun, 29 Dec 2019 21:22:18 -0700    
  
committer: David Steele <[email protected]>    
date     : Sun, 29 Dec 2019 21:22:18 -0700

Click here for diff

The command option will be removed from the C code so it needs to be removed here as well.  
  
This code is now used only for testing so it's not important that it be so precise.

M lib/pgBackRest/Db.pm

`Remove unused LibC code.`

commit   : 33e328abbf5992caccc15ff42ca2dca14502c983    
  
author   : David Steele <[email protected]>    
date     : Sat, 28 Dec 2019 18:30:32 -0700    
  
committer: David Steele <[email protected]>    
date     : Sat, 28 Dec 2019 18:30:32 -0700

Click here for diff

The code was made obsolete by the migration to C.

D lib/pgBackRest/Common/Lock.pm
M lib/pgBackRest/LibCAuto.pm
M libc/LibC.xs
M libc/Makefile.PL
M libc/build/lib/pgBackRestLibC/Build.pm
D libc/xs/common/encode.xs
D libc/xs/common/encode.xsh
D libc/xs/common/lock.xs
M libc/xs/config/config.xs
D libc/xs/crypto/random.xs
M test/lib/pgBackRestTest/Env/ConfigEnvTest.pm

`Add error parameter to cfgCommandId().`

commit   : e72a9dd0d2b6aad58a5d845112631253382e20e2    
  
author   : David Steele <[email protected]>    
date     : Sat, 28 Dec 2019 13:37:03 -0700    
  
committer: David Steele <[email protected]>    
date     : Sat, 28 Dec 2019 13:37:03 -0700

Click here for diff

This allows commands to be checked for validity without generating an error.

M libc/xs/common/lock.xs
M libc/xs/config/define.xs
M src/command/remote/remote.c
M src/config/config.c
M src/config/config.h
M src/config/parse.c
M test/src/module/config/configTest.c
M test/src/module/config/parseTest.c

`Remove obsolete function declaration.`

commit   : d28837a300d4b585792c049dce25fd73387408b3    
  
author   : David Steele <[email protected]>    
date     : Sat, 28 Dec 2019 13:31:59 -0700    
  
committer: David Steele <[email protected]>    
date     : Sat, 28 Dec 2019 13:31:59 -0700

Click here for diff

M src/config/define.h

`Change meaning of TEST_RESULT_STR() macro.`

commit   : d41eea685a732fa80d99d0af0fc52746e13d1152    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Dec 2019 18:08:27 -0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Dec 2019 18:08:27 -0700

Click here for diff

This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping.  
  
TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration.  
  
Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().

M CONTRIBUTING.md
M doc/xml/contributing.xml
M test/src/common/harnessTest.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/storageTest.c
M test/src/module/common/compressGzipTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/debugOffTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/encodeTest.c
M test/src/module/common/execTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/forkTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/logTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/stackTraceTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeMcvTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

`Remove duplicate header and extra linefeed.`

commit   : 02d3918b3233276651fff443b7fcee3f325cf263    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Dec 2019 19:37:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Dec 2019 19:37:55 -0500

Click here for diff

M src/command/storage/list.c

`Remove errant tabs and fix spacing.`

commit   : 74c3842595f49b81be341faf56f2343e9a5c60c8    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Dec 2019 16:25:46 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Dec 2019 16:25:46 -0500

Click here for diff

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/BackRestDoc/Common/DocConfig.pm
M libc/Makefile.PL
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/module/command/backupCommonTest.c
M test/src/module/db/dbTest.c

`Remove errant tab.`

commit   : 7dfea6b71728bc4985021ce8eb993c1376344581    
  
author   : Mike Palmiotto <[email protected]>    
date     : Thu, 19 Dec 2019 16:20:11 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Dec 2019 16:20:11 -0500

Click here for diff

M build/lib/pgBackRestBuild/Config/Data.pm

`Add pg-user option.`

commit   : dc1e7ca22dbdae2556513468433b0591d7cf5a44    
  
author   : Mike Palmiotto <[email protected]>    
date     : Thu, 19 Dec 2019 11:26:38 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Dec 2019 11:26:38 -0500

Click here for diff

Specifies the database user name when connecting to PostgreSQL.  
  
If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/db/helper.c
M src/db/protocol.c
M test/src/module/db/dbTest.c

`Fix misspellings of libpq.`

commit   : 9452084dd174925c554273eb2f8b224c89069123    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 23:32:39 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 23:32:39 -0500

Click here for diff

M test/src/common/harnessPq.h
M test/src/module/postgres/clientTest.c

`Fix misaligned continuation character.`

commit   : 63a855e2f7108f54100b973e23e4bcf3005a1647    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 23:26:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 23:26:52 -0500

Click here for diff

M test/src/module/db/dbTest.c

`Error if option prefix and index total are not both defined.`

commit   : 2f7e139534de54fb19fa46edf7eeb7ecf00d8284    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 22:35:30 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 22:35:30 -0500

Click here for diff

They are both required for indexed options.

M build/lib/pgBackRestBuild/Config/Data.pm

`Add comments about increasing Vagrantfile disk size.`

commit   : d780d084b7b4ad0950bd35b905c778c3a47e8860    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 21:56:02 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 21:56:02 -0500

Click here for diff

M test/Vagrantfile

`Skip vagrant disksize option if no plugin.`

commit   : d89d9f1c52b9fc7bf4827d1b367da6e8520a79d2    
  
author   : Mike Palmiotto <[email protected]>    
date     : Tue, 17 Dec 2019 21:47:19 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 21:47:19 -0500

Click here for diff

Previously, `vagrant up` would bail if no `vagrant-disksize` plugin was  
installed. This option is just a nice-to-have, so skip it rather than  
bailing.

M test/Vagrantfile

`Add note about symlink required for contributing documentation to build.`

commit   : ac35dcac39699b5ecae5f3d178cb837bac1446dc    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 21:28:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 21:28:40 -0500

Click here for diff

M doc/manifest.xml

`Remove unused Perl library missed in f0ef73db.`

commit   : 3394a5017091e57b0adc296b43ac00010a223c3d    
  
author   : Stephen Frost <[email protected]>    
date     : Tue, 17 Dec 2019 21:25:54 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 21:25:54 -0500

Click here for diff

M CONTRIBUTING.md
M doc/xml/contributing.xml

`Remove integration tests that are now covered in the unit tests.`

commit   : 620386f034fd97371c467bda4dc0d90f423bf663    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 20:14:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 20:14:45 -0500

Click here for diff

Most of these tests are just checking that errors are thrown when required.  These are well covered in various unit tests.  
  
The "cannot resume" tests are also well covered in the backup unit tests.  
  
Finally, config warnings are well covered in the config unit tests.  
  
There is more to be done here, but this accounts for the low-hanging fruit.

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

`Integration test improvements for disk and memory efficiency.`

commit   : 977ec2e307c5487df9ed6fadebd486a92c75f269    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 15:23:07 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Dec 2019 15:23:07 -0500

Click here for diff

Set log-level-file=off when more that one test will run.  In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test.  This improves performance pretty dramatically since trace-level logging is expensive.  If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option.  
  
Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel.  
  
Update log replacement rules so that these options can change without affecting expect logs.

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/test.pl
M test/travis.pl

`Increase memory in ramdisk for Travis CI testing.`

commit   : ccea30b8d8970acf9c1826c5c988d819aabf2d0f    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Dec 2019 10:20:23 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Dec 2019 10:20:23 -0500

Click here for diff

The co6 tests were occasionally running out of space so bump up the size of the ramdisk a bit to hopefully prevent this.  
  
A longer term solution would be to disable the trace-level file logs when running on Travis CI since they seem to be using most of the space.

M test/travis.pl

`Don't warn when stop-auto is enabled on PostgreSQL >= 9.6.`

commit   : 6bd280f7bd667305317a7c825d913dac9719c475    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Dec 2019 09:53:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Dec 2019 09:53:50 -0500

Click here for diff

PostgreSQL >= 9.6 uses non-exclusive backup which has implicit stop-auto since the backup will stop when the connection is terminated.  
  
The warning was made more verbose in 1f2ce45e but this now seems like a bad idea since there are likely users with mixed version environments where stop-auto is enabled globally.  There's no reason to fill their logs with warnings over a harmless option.  If anything we should warn when stop-auto is explicitly set to false but this doesn't seem very important either.  
  
Revert to the prior behavior, which is to warn and reset when stop-auto is enabled on PostgreSQL < 9.3.

M src/command/backup/backup.c
M src/db/db.c
M test/src/module/command/backupTest.c
M test/src/module/db/dbTest.c

`Fix handling of \ in filenames.`

commit   : 03849840b813fdfdc2e00874fa8e3bbe9ac95338    
  
author   : David Steele <[email protected]>    
date     : Fri, 13 Dec 2019 21:33:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 13 Dec 2019 21:33:13 -0500

Click here for diff

\ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading.  
  
Use jsonFromStr() to properly quote and escape \.  
  
Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.

M doc/xml/release.xml
M src/info/info.c
M test/src/common/harnessInfo.c
M test/src/module/info/manifestTest.c

`pgBackRest is now pure C.`

commit   : f0ef73db7009cd6e08740d270a6ee7565efc9f8c    
  
author   : David Steele <[email protected]>    
date     : Fri, 13 Dec 2019 17:55:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 13 Dec 2019 17:55:41 -0500

Click here for diff

Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.  
  
Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.  
  
Remove "c" option that allowed the remote to tell if it was being called from C or Perl.  
  
Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.  
  
Update build and installation instructions in the user guide.  
  
Remove all Perl unit tests.  
  
Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.  
  
Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.

M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/Data.pm
D build/lib/pgBackRestBuild/Embed/Build.pm
M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Backup/Info.pm
D lib/pgBackRest/Common/Cipher.pm
D lib/pgBackRest/Common/Io/Process.pm
M lib/pgBackRest/Db.pm
M lib/pgBackRest/LibCAuto.pm
D lib/pgBackRest/Main.pm
M lib/pgBackRest/Manifest.pm
D lib/pgBackRest/Protocol/Base/Master.pm
D lib/pgBackRest/Protocol/Base/Minion.pm
D lib/pgBackRest/Protocol/Command/Master.pm
D lib/pgBackRest/Protocol/Command/Minion.pm
D lib/pgBackRest/Protocol/Helper.pm
D lib/pgBackRest/Protocol/Local/Master.pm
D lib/pgBackRest/Protocol/Local/Process.pm
D lib/pgBackRest/Protocol/Remote/Master.pm
D lib/pgBackRest/Protocol/Remote/Minion.pm
D lib/pgBackRest/Protocol/Storage/File.pm
M lib/pgBackRest/Protocol/Storage/Helper.pm
D lib/pgBackRest/Protocol/Storage/Remote.pm
M libc/LibC.xs
M libc/Makefile.PL
R096 src/perl/config.c libc/xs/config/configTest.xsh
M libc/xs/storage/storage.xs
M src/Makefile.in
M src/build.auto.h.in
M src/command/archive/get/get.c
M src/command/backup/pageChecksum.c
M src/command/info/info.c
M src/common/exit.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/configure
M src/configure.ac
M src/info/infoPg.c
M src/main.c
D src/perl/config.h
D src/perl/embed.auto.c
D src/perl/exec.c
D src/perl/exec.h
D src/perl/libc.auto.c
M src/protocol/helper.c
M src/storage/storage.c
M src/storage/storage.intern.h
M test/define.yaml
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
D test/lib/pgBackRestTest/Module/Common/CommonEncodePerlTest.pm
D test/lib/pgBackRestTest/Module/Common/CommonIniPerlTest.pm
D test/lib/pgBackRestTest/Module/Common/CommonIoBufferedPerlTest.pm
D test/lib/pgBackRestTest/Module/Common/CommonIoHandlePerlTest.pm
D test/lib/pgBackRestTest/Module/Common/CommonIoProcessPerlTest.pm
D test/lib/pgBackRestTest/Module/Common/CommonLogPerlTest.pm
D test/lib/pgBackRestTest/Module/Info/InfoInfoArchivePerlTest.pm
D test/lib/pgBackRestTest/Module/Info/InfoInfoBackupPerlTest.pm
D test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
D test/lib/pgBackRestTest/Module/Protocol/ProtocolCommonMinionPerlTest.pm
D test/lib/pgBackRestTest/Module/Protocol/ProtocolHelperPerlTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
D test/lib/pgBackRestTest/Module/Storage/StorageHelperPerlTest.pm
D test/lib/pgBackRestTest/Module/Storage/StoragePerlTest.pm
A test/patch/debian-package.patch
A test/patch/rhel-package.patch
M test/src/module/common/exitTest.c
D test/src/module/perl/configTest.c
D test/src/module/perl/execTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c
M test/test.pl

`The backup command is implemented entirely in C.`

commit   : 1f2ce45e6b613edfb628ac40fd2369c9455692ba    
  
author   : David Steele <[email protected]>    
date     : Fri, 13 Dec 2019 17:14:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 13 Dec 2019 17:14:26 -0500

Click here for diff

For the most part this is a direct migration of the Perl code into C except as noted below.  
  
A backup can now be initiated from a linked directory.  The link will not be stored in the manifest or recreated on restore.  If a link or directory does not already exist in the restore location then a directory will be created.  
  
The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew.  This has never been an issue in the field that we know of, but we found it in testing.  
  
For online backups all times are fetched from the PostgreSQL primary host (before only copy start was).  This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting.  
  
Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not.  This was a long-standing bug in the Perl code.  
  
Resume will now work even if hardlink settings have been changed.  
  
Reviewed by Cynthia Shang.

M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Archive/Common.pm
D lib/pgBackRest/Backup/Backup.pm
D lib/pgBackRest/Backup/File.pm
M lib/pgBackRest/Db.pm
M lib/pgBackRest/Main.pm
M src/Makefile.in
A src/command/backup/backup.c
A src/command/backup/backup.h
M src/command/backup/protocol.c
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
D test/lib/pgBackRestTest/Module/Backup/BackupFileUnitPerlTest.pm
D test/lib/pgBackRestTest/Module/Backup/BackupUnitPerlTest.pm
D test/lib/pgBackRestTest/Module/Command/CommandArchiveCommonPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/backupTest.c

`Allow end anchor to be excluded in backupRegExp().`

commit   : e206093beb43981f11b532f05897c383f93d8f63    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 18:52:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 18:52:16 -0500

Click here for diff

This is useful for matching files in the backup history directory which have characters after the backup label.

M src/command/backup/common.c
M src/command/backup/common.h
M test/src/module/command/backupCommonTest.c

`Add pgLsnRangeToWalSegmentList() to convert lsn range to wal segments.`

commit   : 8acfb6adf410571464c956ad161ad3df227b73a5    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 16:43:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 16:43:34 -0500

Click here for diff

M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

`Move not found error into walSegmentFind().`

commit   : 81295fd3889ce05829b6443aa29032ccffdfef3e    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 16:28:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 16:28:26 -0500

Click here for diff

This error is also needed in backup so move it here to centralize it.

M src/command/archive/common.c
M src/command/check/check.c
M test/src/module/command/archiveCommonTest.c

`Fix bad arithmetic in pgLsnToWalSegment().`

commit   : 1378d9c58b05f2f45352df155463ab8cd55276a6    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 16:21:51 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 16:21:51 -0500

Click here for diff

/ takes precedence over & but the appropriate parens were not provided.  
  
By some bad luck the tests worked either way, so add a new test that only works the correct way to prevent a regression.

M src/postgres/interface.c
M test/src/module/postgres/interfaceTest.c

`Add pgWalPath() to return version-specific WAL path.`

commit   : 676be2c77390587bb36c9d88bcf1a54d333ee0f3    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 16:11:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 16:11:09 -0500

Click here for diff

Also update the manifest module to use the new function.

M src/info/manifest.c
M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

`Begin v2.21 development.`

commit   : 94cb9540f1c94746d78ae4efb16246b26daf79a0    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 09:05:10 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Dec 2019 09:05:10 -0500

Click here for diff

M doc/RELEASE.md
M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h