pgBackRest 2.20 commit log

v2.19: C Migrations and Bug Fixes

commit   : 2d10293d041cbbc72e4a2bac32401f6f384ecb1a    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Nov 2019 15:51:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Nov 2019 15:51:28 -0500    

Click here for diff

Bug Fixes:  
  
* Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)  
* Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)  
  
Improvements:  
  
* JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang.)  
* The check command is implemented entirely in C. (Contributed by Cynthia Shang.)  
  
Documentation Improvements:  
  
* Document how to contribute to pgBackRest. (Contributed by Cynthia Shang.)  
* Document maximum version for auto-stop option. (Contributed by Brad Nicholson.)  
  
Test Suite Improvements:  
  
* Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)  
* Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)  
* Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)  

M README.md
M doc/RELEASE.md
M doc/release.pl
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Add Strings for STORAGE_REPO_ARCHIVE and STORAGE_REPO_BACKUP.

commit   : a44c5d0315e49c42eb908a4ed0605bee0c509f6d    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Nov 2019 13:12:07 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Nov 2019 13:12:07 -0500    

Click here for diff

These constants are used often enough that they deserve to have String constants rather than repeatedly calling STRDEF().  

M src/command/archive/get/file.c
M src/command/expire/expire.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/info/infoBackup.c
M src/storage/helper.c
M src/storage/helper.h
M test/src/module/info/infoBackupTest.c
M test/src/module/storage/posixTest.c

Fix handling of repeated HTTP headers.

commit   : 10c8eeaf6c0413da265047aeb02a047209ef9d69    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Nov 2019 18:58:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Nov 2019 18:58:45 -0500    

Click here for diff

When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior.  
  
Reported by donicrosby.  

M doc/xml/release.xml
M src/common/io/http/header.c
M test/src/module/common/ioHttpTest.c

Update MinIO to newest release.

commit   : 43171786336733d8e1efe672b801236fd3e2d829    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Nov 2019 17:56:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Nov 2019 17:56:34 -0500    

Click here for diff

We had some problems with newer versions so had held off on updating.  Those problems appear to have been resolved.  
  
In addition, the --compat flag is no longer required.  Prior versions of MinIO required all parts of a multi-part upload (except the last) to be of equal size.  The --compat flag was introduced to restore the default S3 behavior.  Now --compat is only required when ETag is being used for MD5 verification, which we don't do.  

M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm

Add TIME parameter debug type.

commit   : edcc7306a39852ddff507d384555230001442bf4    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Nov 2019 09:46:00 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Nov 2019 09:46:00 -0500    

Click here for diff

Previously we were using int64_t to debug time_t but this may not be right depending on how the compiler represents time_t, e.g. it could be a float.  
  
Since a mismatch would have caused a compiler error we are not worried that this has actually happened, and anyway the worst case is that the debug log would be wonky.  
  
The primary benefit, aside from correctness, is that it makes choosing a parameter debug type for time_t obvious.  

M src/command/restore/file.c
M src/common/debug.h
M src/common/type/convert.c
M src/common/type/convert.h
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/storage.c
M test/define.yaml
M test/src/module/common/typeConvertTest.c

Add building a development environment to contributing documentation.

commit   : eca00e0be00bcd692b10de1b89a700f6525d51a7    
  
author   : David Steele <[email protected]>    
date     : Thu, 7 Nov 2019 13:11:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 7 Nov 2019 13:11:01 -0500    

Click here for diff

This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros.  
  
Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI.  We'll address this in the future when we add contributing documentation to the website.  

M CONTRIBUTING.md
M doc/manifest.xml
M doc/xml/contributing.xml

Allow mock integration tests for all VM types.

commit   : 8b682b75d2ade2e52c75b417fad2eb61150616d1    
  
author   : David Steele <[email protected]>    
date     : Sat, 2 Nov 2019 10:35:48 +0100    
  
committer: David Steele <[email protected]>    
date     : Sat, 2 Nov 2019 10:35:48 +0100    

Click here for diff

Previously the mock integration tests would be skipped for VMs other than the standard four used in CI.  Now VMs outside the standard four will run the same tests as VM4 (currently U18).  

M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm

Add markdown renderer for execute-list block.

commit   : b3e5d88304860fbcb566059d08cbf85e2c3cd95a    
  
author   : David Steele <[email protected]>    
date     : Sat, 2 Nov 2019 09:59:39 +0100    
  
committer: David Steele <[email protected]>    
date     : Sat, 2 Nov 2019 09:59:39 +0100    

Click here for diff

M doc/lib/BackRestDoc/Markdown/DocMarkdownRender.pm

Use getcwd() to construct path when WAL path is relative.

commit   : 7168e0744018d7d27668d59bfe9b6deafe923efa    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Oct 2019 14:55:25 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Oct 2019 14:55:25 +0100    

Click here for diff

Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places.  PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.  

M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c

Error when specified vm is invalid.

commit   : e06db21e35f701b62c625b9f25436459b8fd5c26    
  
author   : David Steele <[email protected]>    
date     : Thu, 17 Oct 2019 14:00:18 +0200    
  
committer: David Steele <[email protected]>    
date     : Thu, 17 Oct 2019 14:00:18 +0200    

Click here for diff

M test/lib/pgBackRestTest/Common/VmTest.pm
M test/test.pl

Disable code generation on dry-run.

commit   : a52faf83a502427a7079ec2bae0593b318c2594c    
  
author   : David Steele <[email protected]>    
date     : Thu, 17 Oct 2019 11:56:45 +0200    
  
committer: David Steele <[email protected]>    
date     : Thu, 17 Oct 2019 11:56:45 +0200    

Click here for diff

M test/test.pl

Update last tests that required sudo.

commit   : fa6a54bb4503efdc8e02a98089a935b6d52a3797    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Oct 2019 17:05:24 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Oct 2019 17:05:24 +0200    

Click here for diff

All tests should now run in a sudo-less environment.  

M .travis.yml
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

C test harness refactor.

commit   : 48bd9e22f124163aa61574027476af8b3737dc39    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Oct 2019 15:48:33 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Oct 2019 15:48:33 +0200    

Click here for diff

Consolidate setting configuration into hrnInit() and rename other functions for consistency.  
  
Split out internal functions into a new header.  

M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
A test/src/common/harnessTest.intern.h
M test/src/test.c

Allow parameters to be passed to travis.pl.

commit   : b4aeb217e665a166cb75577e07f80b01bc0345d1    
  
author   : David Steele <[email protected]>    
date     : Tue, 15 Oct 2019 17:19:42 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 15 Oct 2019 17:19:42 +0200    

Click here for diff

This makes configuring tests easier.  
  
Also add a parameter for tests that require sudo.  This should be retired at some point but some tests still require it.  

M .travis.yml
M test/travis.pl

Remove package build sudo into the container.

commit   : f3b2189659a7f0e8e660c1be42175f0ff82cffb0    
  
author   : David Steele <[email protected]>    
date     : Tue, 15 Oct 2019 13:27:03 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 15 Oct 2019 13:27:03 +0200    

Click here for diff

By running this in the container we no longer need sudo on the host system for package builds.  

M test/test.pl

Run tests in tmpfs.

commit   : 67dde73727d2b30ce3f329054398a1f516a65899    
  
author   : David Steele <[email protected]>    
date     : Mon, 14 Oct 2019 11:51:14 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 14 Oct 2019 11:51:14 +0200    

Click here for diff

This will likely improve performance, but it also makes the filesystem consistent between platforms.  
  
A number of tests were failing on shiftfs, which was the default for arm64 on Travis.  

M test/travis.pl

Add disk/mem info for Travis.

commit   : b1dc5a6d269b858960061e59f3885303d2417051    
  
author   : David Steele <[email protected]>    
date     : Sun, 13 Oct 2019 14:02:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 13 Oct 2019 14:02:52 -0400    

Click here for diff

This is helpful for determining the virtual machine resources.  

M .travis.yml

Update packages required for Travis-CI builds.

commit   : 64c6102a1536ba94f4957fe950038f31525748b5    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 14:45:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 14:45:39 -0400    

Click here for diff

These packages are expected on the arm64 build even though we are using the same os image as amd64.  It appears the arm64 image is slimmer.  

M test/travis.pl

Use a lower user id for posix storage tests.

commit   : 35eef2b8676115036b9d68b14223629dd1e1b773    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 14:16:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 14:16:22 -0400    

Click here for diff

arm64 was not happy with the old user id, so use something smaller.  

M test/src/module/storage/posixTest.c

Use < 0 and > 0 for strCmp() tests.

commit   : 827e95944a2f13301fa14af5a8f11324e17a1cd9    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 13:52:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 13:52:45 -0400    

Click here for diff

Using -1 and 1 was a bit sloppy since the spec only guarantees that the values will be < 0 and > 0.  
  
Found on arm64 where the values were -64 and 64.  

M test/src/module/common/typeStringTest.c

Update container images to PostgreSQL 12 GA.

commit   : a2fa1d04b0cbd6dbedaa2447273371c20bd46e56    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 11:26:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 11:26:13 -0400    

Click here for diff

M doc/xml/user-guide.xml
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm

Add Ubuntu 19.04 container definition.

commit   : 397a41e0f9f82987952912b8127a5e6929e29752    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 11:24:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 11:24:55 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/VmTest.pm

Update lcov to 1.14.

commit   : 93656db186e6da286bd704dc65235fcc209bb19d    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 11:24:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 11:24:21 -0400    

Click here for diff

1.13 is not compatible with gcc 8 which is what ships with newer distributions.  Build from source to get a more recent version.  
  
1.13 is not compatible with gcc 9 so we'll need to address that at a later date.  

M test/Vagrantfile
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Remove pgbackrest test user.

commit   : 11c7c8fabb1e3eb9dadce2231fbb9cb3d76d553f    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 09:45:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 09:45:18 -0400    

Click here for diff

This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories.  The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user.  
  
Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user.  This allows us to remove most sudos, except where they are explicitly needed in tests.  
  
While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.  

M test/Vagrantfile
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Info/InfoInfoArchivePerlTest.pm
M test/lib/pgBackRestTest/Module/Info/InfoInfoBackupPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/src/common/harnessTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c
M test/src/test.c
M test/test.pl
M test/travis.pl

Fix recovery test failing in PostgreSQL 12.0.

commit   : 6f0e7f00af3e83d071713a671a6f155cd1fa3b88    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 09:26:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2019 09:26:19 -0400    

Click here for diff

This test was not creating recovery.signal when testing with --type=preserve.  The preserve recovery type only keeps existing files and does not create any.  
  
RC1 was just ignoring recovery.signal and going right into recovery.  Weirdly, 12.0 used restore_command to do crash recovery which made the problem harder to diagnose, but this has now been fixed in PostgreSQL and should be released in 12.1.  

M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Add a warning about jq rounding numbers to the documentation.

commit   : 59a4a0c1b1726812126a13e57bee29689b0a8fdc    
  
author   : David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 13:14:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 13:14:47 -0400    

Click here for diff

M doc/xml/user-guide.xml

Remove pretty-printing from jsonFromKv() and jsonFromVar().

commit   : db1dc4f275667903f560c140c2c61cce7f9b4020    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 11 Oct 2019 13:03:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 13:03:52 -0400    

Click here for diff

Now that pretty-printing has been removed from the info command it no longer has a purpose, so remove it.  

M libc/xs/postgres/client.xs
M libc/xs/storage/storage.xs
M libc/xs/storage/storage.xsh
M src/command/info/info.c
M src/common/type/json.c
M src/common/type/json.h
M src/info/info.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M src/perl/config.c
M src/perl/libc.auto.c
M src/protocol/command.c
M src/protocol/server.c
M test/src/common/harnessInfo.c
M test/src/common/harnessPq.c
M test/src/module/command/backupCommonTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/postgres/clientTest.c

JSON output from the info command is no longer pretty-printed.

commit   : d90b2724f883686afdae386c1350abfdcfb0d5fa    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 11 Oct 2019 12:56:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 12:56:03 -0400    

Click here for diff

Monitoring systems can more easily ingest the JSON without linefeeds.  
  
External tools such as jq can be used to pretty-print if desired.  

M doc/example/pgsql-pgbackrest-info.sql
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c

Remove info expect tests from mock/all and mock/stanza.

commit   : 29725805668656a920283401ef144a98cee2c5f9    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 11 Oct 2019 12:38:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 12:38:03 -0400    

Click here for diff

These tests are redundant now that we have full coverage in the unit tests are are not worth maintaining anymore.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm

Refactor --pre option in documentation.

commit   : e4408c7dd390b13bb022b29e8f0b151fa0526362    
  
author   : David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 12:28:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 12:28:47 -0400    

Click here for diff

Mark all pre commands as skip so they won't be run again after the container is built.  
  
Ensure that pre commands added to the container are run as the container user if they are not intended to run as root.  

M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/lib/BackRestDoc/Common/DocRender.pm

Don't autogenerate embedded libc code by default.

commit   : 642ce003c8e8f59ec67d3bda4c18decb4aa21ae2    
  
author   : David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 11:32:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Oct 2019 11:32:51 -0400    

Click here for diff

This is only needed when new code is added to the Perl C library, which is becoming rare as the migration progresses.  
  
Also, the code will vary slightly based on the Perl version used for generation so for normal users it is just noise.  
  
Suggested by Stephen Frost.  

M doc/xml/release.xml
M test/test.pl

Make perl/exec test container required.

commit   : bcd3e4953aa1c8e8ccfadd141f38473b2ecbaff6    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 22:09:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 22:09:56 -0400    

Click here for diff

This test fails in some cases when --vm=none but it's not worth investigating since this code will be going away soon.  

M test/define.yaml

Fix mismatched timezone in expect test.

commit   : e3d87ebace36984728ae571a77ca1e11ae383d5b    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 19:43:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 19:43:42 -0400    

Click here for diff

Also run the --vm-none tests in a non-UTC timezone to prevent regressions.  
  
Suggested by Stephen Frost.  

M doc/xml/release.xml
M test/src/module/command/infoTest.c
M test/travis.pl

Allow tests that use ports to run in parallel.

commit   : 6db4e59a6631eb47758fd0dff772e2ca086e924b    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 16:13:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 16:13:43 -0400    

Click here for diff

Set the test index in the C unit test code so it can assign port numbers that won't conflict between tests.  

M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTls.c
M test/src/common/harnessTls.h
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/s3Test.c
M test/src/test.c
M test/travis.pl

Fix container test path being used when --vm=none.

commit   : 13fcbb24e940084710acc13627515522215369dc    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 15:09:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 15:09:11 -0400    

Click here for diff

Suggested by Stephen Frost.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm

Disable docker service for --vm=none test.

commit   : e732720d855cffa0e4bb80997529254323fd7767    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 11:28:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 11:28:37 -0400    

Click here for diff

Starting this service wastes time and this will help catch any regressions where containers are being built when they should not be.  

M .travis.yml

Remove code to generate .travis.yml.

commit   : 9a3ba649e111f3529bb6fe09238d9fcb81454487    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 11:25:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 11:25:59 -0400    

Click here for diff

Most of the logic has been moved to test/travis.pl so there wasn't much purpose to this code anymore.  

M test/code-count/file-type.yaml
D test/lib/pgBackRestTest/Common/CiTest.pm
M test/test.pl
M test/travis.pl

Don't require sudo to run tests with --vm=none.

commit   : 696e6a7c444ba0ec3a28f8762b6a7266751faa0f    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 11:21:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 11:21:09 -0400    

Click here for diff

Run these tests without sudo privileges on Travis to prevent regressions.  

M test/test.pl
M test/travis.pl

Move release note to correct section.

commit   : 50d1d0a4e8f232842a27c4998cb6af3f1d906bf3    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 09:28:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 09:28:50 -0400    

Click here for diff

M doc/xml/release.xml

Fix remote timeout in delta restore.

commit   : 47e0ba2502e5c3e8eb088ffcbd774144741b599b    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 09:27:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 09:27:46 -0400    

Click here for diff

When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout.  
  
Add keep-alives to prevent remote timeout.  
  
Reported by James Sewell, Jens Wilke.  

M doc/xml/release.xml
M src/Makefile.in
M src/protocol/server.c

Fix typo.

commit   : 82baf1e521e2a1ed7b27403478aa4a1dc4a6afcc    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 10 Oct 2019 08:33:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Oct 2019 08:33:40 -0400    

Click here for diff

M CONTRIBUTING.md
M doc/xml/contributing.xml

Add gcc 9 support.

commit   : 7f369006b5c812d0d3e05f4735d03623c3873b45    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Oct 2019 15:03:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Oct 2019 15:03:03 -0400    

Click here for diff

A number of tests have been updated and Fedora 30 has been added to the test suite so the unit tests can run on gcc 9.  
  
Stop running unit tests on co6/7 since we appear to have ample unit test coverage.  

M .travis.yml
M test/container.yaml
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Module/Info/InfoInfoArchivePerlTest.pm
M test/src/common/harnessTest.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/common/regExpTest.c
M test/travis.pl

Remove dependency on aws cli for testing.

commit   : 528f4c4347d89a0c05b0a9521e8ef91432616b0c    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Oct 2019 14:38:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Oct 2019 14:38:24 -0400    

Click here for diff

This tool was only being used it a few places but was a pretty large dependency.  
  
Rework the forceStorageMove() code using our storage layer and replace one aws cli cp with a storage put.  
  
Also, remove the Dockerfile that was once used to build the Scality S3 test container.  

M test/Vagrantfile
D test/docker/s3-server.docker
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/travis.pl

Refactor error logic to make sure Db object gets freed immediately.

commit   : ac870b42de5a51436a4a08c6ad405ac4f1dd0763    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Oct 2019 08:43:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Oct 2019 08:43:30 -0400    

Click here for diff

Because db can be reset to NULL on an error in the try block we need nested try blocks to ensure that db is non-NULL and can be freed on an error after being created.  
  
This is not a production issue because the db will be freed when the temp mem context is freed, but it does affect reproducibility in the tests and is a bit tidier.  

M src/db/helper.c

Be smarter about which packages are loaded for testing.

commit   : 61c4f64895497da4cbe1e9225f274d255dd5133e    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 18:56:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 18:56:55 -0400    

Click here for diff

Now that our tests are more diversified it makes sense to load only the packages that are needed for each test.  
  
Move the package loads from .travis.yaml to test/travis.pl where we have more control over what is loaded.  

M .travis.yml
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/travis.pl

The check command is implemented entirely in C.

commit   : a1c13a50dd0886076a1f32e58901eec271d9ec9a    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 8 Oct 2019 18:04:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 18:04:09 -0400    

Click here for diff

Note that building the manifest on each host has been temporarily removed.  
  
This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.  

M doc/xml/release.xml
M doc/xml/user-guide.xml
D lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Main.pm
M lib/pgBackRest/Protocol/Helper.pm
M lib/pgBackRest/Protocol/Remote/Minion.pm
M src/Makefile.in
M src/command/check/check.c
M src/command/check/common.c
M src/command/check/common.h
M src/command/stanza/common.c
M src/command/stanza/common.h
M src/command/stanza/create.c
M src/command/stanza/upgrade.c
M src/db/db.c
M src/db/db.h
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessPq.h
M test/src/module/command/checkTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/db/dbTest.c

Update expire command to use C backup.info reconstruct.

commit   : ecae5e34e5063838bb55d55b1752c6b2eaaaa75b    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 8 Oct 2019 17:30:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 17:30:33 -0400    

Click here for diff

This was still being done in Perl until the C Manifest object was available.  

M lib/pgBackRest/Main.pm
M src/command/expire/expire.c
M src/main.c
M src/perl/embed.auto.c
M test/src/module/command/expireTest.c

Add infoBackupLoadFileReconstruct() to InfoBackup object.

commit   : 4e4d1f414a153232079f2e2e9ce203ffc5a2362c    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 8 Oct 2019 16:04:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 16:04:27 -0400    

Click here for diff

Check the backup.info file against the backup path.  Add any backups that are missing and remove any backups that no longer exist.  
  
It's important to run this before backup or expire to be sure we are using the most up-to-date list of backups.  

M src/Makefile.in
M src/info/infoBackup.c
M src/info/infoBackup.h
M test/src/module/info/infoBackupTest.c

Add missing header file.

commit   : b2825b82c7b8cf140969dd4e9fd04552e921bbfc    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 15:47:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 15:47:47 -0400    

Click here for diff

M test/src/common/harnessConfig.h

Add pgBackRest version to Info and Manifest objects.

commit   : 6d8d0eeba71fa7cd00af6e40c0adf74638a371bd    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 8 Oct 2019 15:37:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 15:37:08 -0400    

Click here for diff

This was not being exposed previously because it is primarily informational, but now it is needed to reconstruct the backup.info file.  

M src/info/info.c
M src/info/info.h
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/info/infoTest.c
M test/src/module/info/manifestTest.c

Document how to contribute to pgBackRest.

commit   : 38b72eded4b0f295ed1f30f849f0a99f69e396d2    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 8 Oct 2019 15:27:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 15:27:17 -0400    

Click here for diff

There's a lot more to be done here, but this is a good start.  

M CODING.md
A CONTRIBUTING.md
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/manifest.xml
M doc/xml/coding.xml
A doc/xml/contributing.xml
M doc/xml/release.xml
M test/Vagrantfile

Update release instructions with a reminder to install latex.

commit   : a4152a0ea1dc5c28515aedf9690cab22765076cb    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 8 Oct 2019 12:29:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 12:29:42 -0400    

Click here for diff

Latex is no longer installed by default in the Vagrantfile.  

M doc/RELEASE.md

Allow most unit tests to run outside of a container.

commit   : 45881c74aeff4bb25559ec0254fa7fc1960d9cab    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 12:06:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 12:06:30 -0400    

Click here for diff

Three major changes were required to get this working:  
  
1) Provide the path to pgbackrest in the build directory when running outside a container.  Tests in a container will continue to install and run against /usr/bin/pgbackrest.  
  
1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path.  Also set a per-test log-path while we are at it.  
  
2) Use localhost instead of a custom host for TLS test connections.  Tests in containers will continue to update /etc/hosts and use the custom host.  
  
Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing.  
  
Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time.  Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.  

M .travis.yml
M doc/release.pl
M test/Vagrantfile
M test/define.yaml
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/ExecuteTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/common/harnessLog.c
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTls.c
M test/src/common/harnessTls.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/storageTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c
M test/src/test.c
M test/test.pl
M test/travis.pl

Add missing spaces.

commit   : 77b0c6c993a0e6ff45a6a99f343c3709a016d152    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 10:35:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2019 10:35:47 -0400    

Click here for diff

M src/common/type/variant.h

Remove pgPath parameter from pgControlFromFile().

commit   : 5394893e33a844991a913e3b3c8874b435ac88d1    
  
author   : David Steele <[email protected]>    
date     : Thu, 3 Oct 2019 11:14:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 3 Oct 2019 11:14:22 -0400    

Click here for diff

In practice this function is always used with storagePg*() so pgPath is known.  

M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/stanza/common.c
M src/main.c
M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

Document maximum version for auto-stop option.

commit   : 887ec97e0743f758a0a2d2222cd1553b91f0908d    
  
author   : Brad Nicholson <[email protected]>    
date     : Thu, 3 Oct 2019 06:58:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 3 Oct 2019 06:58:52 -0400    

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/define.auto.c

Begin v2.19 development.

commit   : 008ec6b05c631adf08cba312d6bab972f02881ef    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Oct 2019 17:04:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Oct 2019 17:04:26 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h