pgBackRest 2.6 commit log

v2.06: Checksum Delta Backup and PostgreSQL 11 Support

commit   : 904550c97f674d3f576d3113756f2b5c89b55ddf    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Oct 2018 14:56:51 +0100    
  
committer: David Steele <[email protected]>    
date     : Tue, 16 Oct 2018 14:56:51 +0100    

Click here for diff

Bug Fixes:  
  
* Fix missing missing URI encoding in S3 driver. (Reported by Dan Farrell.)  
* Fix incorrect error message for duplicate options in configuration files. (Reported by Jesper St John.)  
* Fix incorrectly reported error return in info logging. A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.  
  
Features:  
  
* Add checksum delta for incremental backups which uses checksums rather than timestamps to determine if files have changed. (Contributed by Cynthia Shang.)  
* PostgreSQL 11 support, including configurable WAL segment size.  
  
Improvements:  
  
* Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL. Previously an error would be generated if other files were present and not owned by the PostgreSQL user.  
* Improve info command to display the stanza cipher type. (Contributed by Cynthia Shang. Suggested by Douglas J Hunley.)  
* Improve support for special characters in filenames.  
* Allow delta option to be specified in the pgBackRest configuration file. (Contributed by Cynthia Shang.)  

M README.md
M doc/resource/exe.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h

PostgreSQL 11 support.

commit   : 2c272c220b39739ff324305593034487df81b466    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 Oct 2018 23:23:49 +0100    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Oct 2018 23:23:49 +0100    

Click here for diff

PostgreSQL 11 RC1 support was tested in 9ae3d8c46 when the u18 container was rebuilt.  Nothing substantive changed after RC1 so pgBackRest is ready for PostgreSQL 11 GA.  

M doc/xml/release.xml

Install nodejs from deb.nodesource.com.

commit   : 9ae3d8c46ac10273ca53c942410f768e27395230    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 Oct 2018 23:13:08 +0100    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Oct 2018 23:13:08 +0100    

Click here for diff

The standard npm packages on Ubuntu 18.04 suddenly required libssl1.0 which broke the pgbackrest package builds. Installing nodejs from deb.nodesource.com seems to work fine with standard libssl.  
  
This package is required by ScalityS3 which is used for local S3 testing.  

M doc/xml/release.xml
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Improve documentation in filter.h and filter.internal.h.

commit   : 98ff8ccc59fa1e344426f3084e0a37498e56ce12    
  
author   : David Steele <[email protected]>    
date     : Tue, 9 Oct 2018 15:08:49 +0100    
  
committer: David Steele <[email protected]>    
date     : Tue, 9 Oct 2018 15:08:49 +0100    

Click here for diff

When the filter interface internals were split out into a new header file the documentation was not moved as it should have been.  Additionally some functions which should have been moved were left behind.  
  
Move the documentation and functions to filter.internal.h and add more documentation.  Filters are a tricky subject so the more documentation the better.  
  
Also add documentation for the user-facing filter functions in filter.h.  

M doc/xml/release.xml
M src/Makefile
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c

Add ioReadLine()/ioWriteLine() to IoRead/IoWrite objects.

commit   : 68110d04b24dc633e5b5b6869924dd36a2307135    
  
author   : David Steele <[email protected]>    
date     : Sun, 7 Oct 2018 17:50:10 +0100    
  
committer: David Steele <[email protected]>    
date     : Sun, 7 Oct 2018 17:50:10 +0100    

Click here for diff

Allow a single linefeed-terminated line to be read or written.  This is useful for various protocol implementations, including HTTP and pgBackRest's protocol.  
  
On read the maximum line size is limited to buffer-size to prevent runaway memory usage in case a linefeed is not found.  This seems fine for HTTP but we may need to revisit this decision when implementing the pgBackRest protocol.  Another option would be to increase the minimum buffer size (currently 16KB).  

M doc/xml/release.xml
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/write.c
M src/common/io/write.h
M src/storage/driver/posix/fileRead.c
M test/src/module/common/ioTest.c

Disable flapping archive/get unit on CentOS 6.

commit   : db8dce7adcf277a021ad8766ab91e86fad913eff    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 Oct 2018 17:54:43 +0100    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 Oct 2018 17:54:43 +0100    

Click here for diff

This test has been flapping since 9b9396c7.  It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs.  It only happens on Travis and is not reproducible in any development environment that we have tried.  
  
For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause.  We are actively developing these tests and hope the issue will be identified during the course of normal development.  
  
A number of improvements were made to the tests while searching for this issue.  While none of them helped, it makes sense to keep the improvements.  

M doc/xml/release.xml
M test/define.yaml
M test/src/module/archive/getTest.c

Add .gitignore to C src directory.

commit   : ed5d7a53de7cd952825e824ae507e275c4648e18    
  
author   : Stephen Frost <[email protected]>    
date     : Sun, 30 Sep 2018 12:27:05 +0100    
  
committer: David Steele <[email protected]>    
date     : Sun, 30 Sep 2018 12:27:05 +0100    

Click here for diff

Suggested by Stephen Frost.  

M doc/xml/release.xml
A src/.gitignore

Avoid calling strlen() twice in bufNewZ().

commit   : 3be9f2dee79c02e728e2339dd047d6990ec215c5    
  
author   : Stephen Frost <[email protected]>    
date     : Sun, 30 Sep 2018 11:45:37 +0100    
  
committer: David Steele <[email protected]>    
date     : Sun, 30 Sep 2018 11:45:37 +0100    

Click here for diff

M doc/xml/release.xml
M src/common/type/buffer.c

Fix incorrect error message for duplicate options in configuration files.

commit   : 5404628148a3039f3933607c7f63cb4620f71761    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Sep 2018 17:48:40 +0100    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Sep 2018 17:48:40 +0100    

Click here for diff

Duplicating a non-multi-value option was not throwing the correct message when the option was a boolean.  
  
The reason was that the option was being validated as a boolean before the multi-value check was being done.  The validation code assumed it was operating on a string but was instead operating on a string list causing an assertion to fail.  
  
Since it's not safe to do the multi-value check so late, move it up to the command-line and configuration file parse phases instead.  
  
Reported by Jesper St John.  

M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c

Add cfgDefOptionMulti() to identify multi-value options.

commit   : be2271f6d312e6497e98df2c792d2455afb461eb    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Sep 2018 17:24:06 +0100    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Sep 2018 17:24:06 +0100    

Click here for diff

Previously this was done in two separate places by checking if an option was type hash or list.  
  
Bad enough that it was in two places, but an upcoming bug fix will add another instance so make it a function.  

M doc/xml/release.xml
M src/config/define.c
M src/config/define.h
M src/config/parse.c
M test/src/module/config/defineTest.c

Add cryptoHmacOne() for HMAC support.

commit   : e66e68e324fb8381e0592a419b3aafdf70703b56    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Sep 2018 09:20:47 +0100    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Sep 2018 09:20:47 +0100    

Click here for diff

There doesn't seem to be any need to implement this as a filter since current use cases (S3 authentication) work on small datasets.  
  
So, use the single function method provided by OpenSSL for simplicity.  

M doc/xml/release.xml
M src/Makefile
M src/crypto/hash.c
M src/crypto/hash.h
M test/define.yaml
M test/src/module/crypto/hashTest.c

Comment fixes.

commit   : 6470f03fa28a72c9603d4d913ce58c7223b6cc0b    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Sep 2018 22:36:42 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Sep 2018 22:36:42 +0100    

Click here for diff

M src/storage/driver/posix/storage.c
M test/src/module/storage/posixTest.c

Add bufHex()to Buffer object.

commit   : bcca625062b4ee8394f933218e890238a75e9419    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Sep 2018 22:33:48 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Sep 2018 22:33:48 +0100    

Click here for diff

A general-purpose function for converting buffers to hex strings.  

M doc/xml/release.xml
M libc/xs/crypto/hash.xs
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/crypto/hash.c
M src/crypto/hash.h
M src/info/info.c
M src/perl/libc.auto.c
M test/define.yaml
M test/src/module/common/typeBufferTest.c
M test/src/module/crypto/hashTest.c

Add bufNewZ() to Buffer object.

commit   : 51484a008f1172b51c2296974de7bfcd37ab24b8    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Sep 2018 18:46:52 +0100    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Sep 2018 18:46:52 +0100    

Click here for diff

This constructor creates a Buffer object directly from a zero-terminated string.  The old way was to create a String object first, then convert that to a Buffer using bufNewStr().  
  
Updated in all places that used the old pattern.  

M doc/xml/release.xml
M src/common/type/buffer.c
M src/common/type/buffer.h
M test/src/module/archive/commonTest.c
M test/src/module/archive/getTest.c
M test/src/module/archive/pushTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/config/parseTest.c
M test/src/module/crypto/hashTest.c
M test/src/module/storage/posixTest.c

Support configurable WAL segment size.

commit   : d038b9a029f0c981e32a94b8b8632c356ffec7ac    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 Sep 2018 10:24:42 +0100    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 Sep 2018 10:24:42 +0100    

Click here for diff

PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB.  
  
There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup.  Both operations require the WAL segment size to properly build a list.  
  
Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database.  
  
The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead.  This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them.  While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL.  
  
Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.  

M doc/xml/release.xml
M lib/pgBackRest/Archive/Common.pm
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Db.pm
M src/Makefile
M src/command/archive/common.h
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/info/infoPg.c
M src/main.c
M src/perl/embed.auto.c
D src/postgres/info.c
A src/postgres/interface.c
R050 src/postgres/info.h src/postgres/interface.h
A src/postgres/interface/v083.auto.c
A src/postgres/interface/v083.c
A src/postgres/interface/v083.h
A src/postgres/interface/v084.auto.c
A src/postgres/interface/v084.c
A src/postgres/interface/v084.h
A src/postgres/interface/v090.auto.c
A src/postgres/interface/v090.c
A src/postgres/interface/v090.h
A src/postgres/interface/v091.auto.c
A src/postgres/interface/v091.c
A src/postgres/interface/v091.h
A src/postgres/interface/v092.auto.c
A src/postgres/interface/v092.c
A src/postgres/interface/v092.h
A src/postgres/interface/v093.auto.c
A src/postgres/interface/v093.c
A src/postgres/interface/v093.h
A src/postgres/interface/v094.auto.c
A src/postgres/interface/v094.c
A src/postgres/interface/v094.h
A src/postgres/interface/v095.auto.c
A src/postgres/interface/v095.c
A src/postgres/interface/v095.h
A src/postgres/interface/v096.auto.c
A src/postgres/interface/v096.c
A src/postgres/interface/v096.h
A src/postgres/interface/v100.auto.c
A src/postgres/interface/v100.c
A src/postgres/interface/v100.h
A src/postgres/interface/v110.auto.c
A src/postgres/interface/v110.c
A src/postgres/interface/v110.h
M src/postgres/pageChecksum.c
D src/postgres/type.h
M src/postgres/version.h
M test/code-count/file-type.yaml
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Archive/ArchiveCommonPerlTest.pm
M test/lib/pgBackRestTest/Module/Backup/BackupFileUnitPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/archive/getTest.c
D test/src/module/postgres/infoTest.c
A test/src/module/postgres/interfaceTest.c

PostgreSQL 11 Beta 4 support.

commit   : c0b0b4e541fc7cc5077e7f6b9fa646e7654ca9a4    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Sep 2018 13:25:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 Sep 2018 13:25:27 -0400    

Click here for diff

Catalog version changed for this release, so update it.  
  
Also update and upload a new container with beta 4 installed.  

M doc/xml/release.xml
M lib/pgBackRest/Db.pm
M src/perl/embed.auto.c
M src/postgres/info.c
M test/container.yaml
M test/src/module/postgres/infoTest.c

Update introduction to be pithy.

commit   : d513a46e6d9d34a28dd4b99636d6ea11d3e0dff4    
  
author   : David Steele <[email protected]>    
date     : Wed, 19 Sep 2018 18:26:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 19 Sep 2018 18:26:12 -0400    

Click here for diff

M README.md
M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/xml/index.xml
M doc/xml/release.xml

Use command in authorized_hosts to improve SSH security.

commit   : 50266cc26c2c76fd155779e85c82dbed58cc2d84    
  
author   : David Steele <[email protected]>    
date     : Wed, 19 Sep 2018 16:41:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 19 Sep 2018 16:41:20 -0400    

Click here for diff

Unsecured, passwordless SSH can be a scary thing. If an attacker gains access to one system they can easily hop to other systems.  
  
Add documentation on how to use the command parameter in authorized_keys to limit ssh to running a single command, pgbackrest. There is more that could be done for security but this likely addresses most needs.  
  
Also change references to "trusted ssh" to "passwordless ssh" since this seems more correct.  
  
Suggested by Stephen Frost, Magnus Hagander.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Add checksum delta for incremental backups.

commit   : 880fbb5e578c387e52f83c8eca1040b42f4e21a8    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 19 Sep 2018 11:12:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 19 Sep 2018 11:12:45 -0400    

Click here for diff

Use checksums rather than timestamps to determine if files have changed.  This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby.  
  
If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full.  Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well.  
  
Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time.  A future commit will address automatically enabling the feature in cases where it seems likely to be useful.  
  
Contributed by Cynthia Shang.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Backup/File.pm
M lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Manifest.pm
M src/config/define.auto.c
M src/perl/embed.auto.c
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
A test/lib/pgBackRestTest/Module/Backup/BackupFileUnitPerlTest.pm
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/help/helpTest.c

Allow delta option to be specified in the pgBackRest configuration file.

commit   : bf0691576a56df0d662488cee19d21e924a3cecb    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 18 Sep 2018 13:29:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Sep 2018 13:29:05 -0400    

Click here for diff

This option was previously allowed on the command-line only for no particular reason that we could determine.  
  
Being able to specify it in the config file seems like a good idea and won't change current usage.  
  
Contributed by Cynthia Shang.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/define.auto.c
M src/config/parse.auto.c

Allow hashSize() to run on remote storage.

commit   : b6b2c915b24fdcfb64e1fe6a0c8a796da50c9fe4    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 18 Sep 2018 11:39:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Sep 2018 11:39:48 -0400    

Click here for diff

Apparently we never needed to run this function remotely.  
  
It will be needed by the backup checksum delta feature, so implement it now.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M lib/pgBackRest/Protocol/Helper.pm
M lib/pgBackRest/Protocol/Remote/Minion.pm
M lib/pgBackRest/Protocol/Storage/Remote.pm
M lib/pgBackRest/Storage/Local.pm
M src/perl/embed.auto.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
M test/lib/pgBackRestTest/Module/Storage/StorageLocalPerlTest.pm

Restore bIgnoreMissing flag in backupFile() lost in storage refactor.

commit   : 052e483057e4ee55d29c229c39b92c90a9cee0af    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 18 Sep 2018 10:18:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Sep 2018 10:18:39 -0400    

Click here for diff

The test to make sure that some files (e.g. pg_control) do not get removed during the backup was lost during the storage refactor committed at de7fc37f.  
  
This did not impact the integrity of the backups, but bring it back since it is a nice sanity check.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M lib/pgBackRest/Backup/File.pm
M src/perl/embed.auto.c

Merge all posix storage tests into a single unit.

commit   : 03003562d86f7fdbab21cad9f7a7ee56c3070f93    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 Sep 2018 11:45:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 17 Sep 2018 11:45:41 -0400    

Click here for diff

As we add storage drivers it's important to keep the tests for each completely separate.  Rather than have three tests for each driver, standardize on having a single test unit for each driver.  

M doc/xml/release.xml
M test/define.yaml
D test/src/module/storage/fileTest.c
D test/src/module/storage/helperTest.c
R060 test/src/module/storage/storageTest.c test/src/module/storage/posixTest.c

Add -ftree-coalesce-vars option to unit test compilation.

commit   : e55d73304102145c46c32f25dbe103cd4c750d37    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 Sep 2018 11:38:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 17 Sep 2018 11:38:10 -0400    

Click here for diff

This is a workaround for inefficient handling of many setjmps in gcc >= 4.9.  Setjmp is used in all error handling, but in the unit tests each test macro contains an error handling block so they add up pretty quickly for large unit tests.  
  
Enabling -ftree-coalesce-vars in affected versions reduces build time and memory requirements by nearly an order of magnitude.  Even so, compiles are much slower than gcc <= 4.8.  
  
We submitted a bug for this at: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87316  
Which was marked as a duplicate of: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63155  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm

Add CIFS driver to storage helper for read-only repositories.

commit   : b5f749b21cbf51e3ba9f3bc2129c337cb6ebec92    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 18:41:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 18:41:30 -0400    

Click here for diff

For read-only repositories the Posix and CIFS drivers behave exactly the same.  Since that's all we support in C right now it's valid to treat them as the same thing.  An assertion has been added to remind us to add the CIFS driver before allowing the repository to be writable.  
  
Mostly we want to make sure that the C code does not blow up when the repository type is CIFS.  

M doc/xml/release.xml
M src/storage/helper.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/src/module/storage/helperTest.c

Clear test directory between test runs.

commit   : a6c346cb04d8977d37e52072488d89a7fb3a89b6    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 17:26:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 17:26:04 -0400    

Click here for diff

Previously it was the responsibility of the individual tests to clean up after themselves.  Now the test harness now does the cleanup automatically.  
  
This means that some paths/files need to be recreated with each run but that doesn't happen very often.  
  
An attempt has been made to remove all redundant cleanup code but it's hard to know if everything has been caught.  No issues will be caused by anything that was missed, but they will continue to chew up time in the tests.  

M doc/xml/release.xml
M test/src/common/harnessTest.c
M test/src/module/archive/commonTest.c
M test/src/module/archive/getTest.c
M test/src/module/storage/fileTest.c
M test/src/module/storage/storageTest.c

Move test expect log out of the regular test directory.

commit   : 4119ce208d1d6de2b096d0465721e93dcb981d3f    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 15:58:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 15:58:46 -0400    

Click here for diff

Storing the expect log (created by common/harnessLog) in the regular test directory was not ideal.  It showed up in tests and made it difficult to clear the test directory between each run.  
  
Move the expect log to a purpose-built directory one level up so it does not interfere with regular testing.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessLog.c
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/module/storage/storageTest.c
M test/src/test.c

Fix missing test caused by a misplaced YAML tag.

commit   : 8852622fa241836b0a6c5c08d755f2c3d990427d    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 15:53:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 15:53:19 -0400    

Click here for diff

M doc/xml/release.xml
M test/define.yaml

Make comment blocks consistent across all tests.

commit   : c47fc28c8b9cba4fba60f206ec029773f60171af    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 14:15:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 14:15:21 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/common/assertOffTest.c
M test/src/module/common/assertOnTest.c
M test/src/module/common/encodeTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/typecTest.c
M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/parseTest.c
M test/src/module/crypto/cipherBlockTest.c
M test/src/module/crypto/cryptoTest.c
M test/src/module/crypto/randomTest.c
M test/src/module/perl/configTest.c
M test/src/module/perl/execTest.c
M test/src/module/postgres/infoTest.c
M test/src/module/postgres/pageChecksumTest.c

Merge protocol storage helper into storage helper.

commit   : 84ab787b1a7f8ca42d9b2be72a730ace852667a1    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 14:12:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 14:12:53 -0400    

Click here for diff

These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it.  
  
So, merge the storage helpers. This also has the benefit of making the code a bit simpler.  
  
Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.  

M doc/xml/release.xml
M src/Makefile
M src/command/archive/common.c
M src/command/archive/get/file.c
M src/command/archive/get/get.c
D src/protocol/storage/helper.c
D src/protocol/storage/helper.h
M src/storage/helper.c
M src/storage/helper.h
M test/define.yaml
M test/src/module/archive/commonTest.c
M test/src/module/archive/getTest.c
M test/src/module/archive/pushTest.c
D test/src/module/protocol/storageHelperTest.c
M test/src/module/storage/helperTest.c

Fix buffer underrun in configuration test harness.

commit   : 3014b05dabaa66fbb3ccfba89c79cca49905a695    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 13:41:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Sep 2018 13:41:49 -0400    

Click here for diff

If the total bytes read from the expect log file was 0 then the last byte of whatever was in memory before harnessLogBuffer would be set to 0.  
  
On 32-bit systems this expressed as the high order byte of a pointer being cleared and wackiness (in the form of segfaults) ensued.  

M doc/xml/release.xml
M test/src/common/harnessLog.c

Update all interfaces to use variable parameter constructors.

commit   : 0c02481d6e5cb7c85cd7caf770d06820137f27b4    
  
author   : David Steele <[email protected]>    
date     : Sat, 15 Sep 2018 21:07:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 15 Sep 2018 21:07:00 -0400    

Click here for diff

Fixed parameter constructors made adding new interface functions a burden, so we switched to using structs to define interfaces in the storage module at c49eaec7.  
  
While propagating this pattern to the IO interfaces it became obvious that the existing variable parameter function pattern (begun in the storage module) was more succinct and consistent with the existing code.  
  
So, use variable parameter functions to define all interfaces. This assumes that the non-interface parameters will be fixed, which seems reasonable for low-level code.  

M doc/xml/release.xml
M src/Makefile
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
A src/common/io/filter/filter.intern.h
M src/common/io/filter/size.c
M src/common/io/read.c
M src/common/io/read.h
A src/common/io/read.intern.h
M src/common/io/write.c
M src/common/io/write.h
A src/common/io/write.intern.h
M src/compress/gzipCompress.c
M src/compress/gzipDecompress.c
M src/crypto/hash.c
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/storage.c
M src/storage/fileRead.c
M src/storage/fileRead.intern.h
M src/storage/fileWrite.c
M src/storage/fileWrite.intern.h
M src/storage/storage.c
M src/storage/storage.intern.h
M test/src/module/common/ioTest.c

Allow C or Perl coverage to run on more than one VM.

commit   : f0ed89f21fe2ae6278e64395f4191519807e529b    
  
author   : David Steele <[email protected]>    
date     : Sat, 15 Sep 2018 13:27:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 15 Sep 2018 13:27:06 -0400    

Click here for diff

C or Perl coverage tests can now be run on any VM provided a recent enough version of Devel::Cover or lcov is available.  
  
For now, leave u18 as the only VM to run coverage tests due to some issues with older versions of lcov.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/test.pl

Remove compiler warnings that are not valid for u16.

commit   : 31cdd9d20b114f5c71c585a6effc4e71fae0d9c2    
  
author   : David Steele <[email protected]>    
date     : Sat, 15 Sep 2018 08:23:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 15 Sep 2018 08:23:55 -0400    

Click here for diff

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm

Full abstraction of storage driver interface.

commit   : c49eaec776014806af194e7a2d9e88691f361f99    
  
author   : David Steele <[email protected]>    
date     : Fri, 14 Sep 2018 16:08:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 14 Sep 2018 16:08:33 -0400    

Click here for diff

The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly.  
  
Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC).  
  
Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.  

M doc/xml/release.xml
M libc/xs/storage/storage.xs
M src/Makefile
M src/command/archive/get/get.c
M src/common/debug.h
M src/common/lock.c
M src/perl/libc.auto.c
M src/protocol/storage/helper.c
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileRead.h
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/fileWrite.h
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h
M src/storage/fileRead.c
M src/storage/fileRead.h
A src/storage/fileRead.intern.h
M src/storage/fileWrite.c
M src/storage/fileWrite.h
A src/storage/fileWrite.intern.h
M src/storage/helper.c
M src/storage/info.h
M src/storage/storage.c
M src/storage/storage.h
A src/storage/storage.intern.h
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/src/module/archive/commonTest.c
M test/src/module/archive/getTest.c
M test/src/module/archive/pushTest.c
M test/src/module/command/controlTest.c
M test/src/module/common/lockTest.c
M test/src/module/help/helpTest.c
M test/src/module/info/infoTest.c
M test/src/module/postgres/infoTest.c
M test/src/module/storage/fileTest.c
M test/src/module/storage/storageTest.c

Don't perform valgrind when requested.

commit   : aeb1fa3dfbf4a4b7384f8f2bc5a18a23e741396e    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 Sep 2018 19:12:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 Sep 2018 19:12:40 -0400    

Click here for diff

The --no-valgrind flag was not being honored.  It's not clear if this flag ever worked, but it does now.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm

Rename posix driver files/functions for consistency.

commit   : fd14ceb3995fca731141f09f7c940819138c63a2    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 Sep 2018 18:58:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 Sep 2018 18:58:22 -0400    

Click here for diff

The posix driver was developed over time and the naming is not very consistent.  
  
Rename the files and functions to work well with other drivers and generally favor longer names since the driver functions are seldom (eventually never) used outside the driver itself.  

M doc/xml/release.xml
M libc/LibC.xs
M libc/Makefile.PL
M src/Makefile
M src/common/lock.c
M src/perl/libc.auto.c
R093 src/storage/driver/posix/driverFile.c src/storage/driver/posix/common.c
R060 src/storage/driver/posix/driverFile.h src/storage/driver/posix/common.h
R078 src/storage/driver/posix/driverRead.c src/storage/driver/posix/fileRead.c
R065 src/storage/driver/posix/driverRead.h src/storage/driver/posix/fileRead.h
R079 src/storage/driver/posix/driverWrite.c src/storage/driver/posix/fileWrite.c
R059 src/storage/driver/posix/driverWrite.h src/storage/driver/posix/fileWrite.h
R092 src/storage/driver/posix/driver.c src/storage/driver/posix/storage.c
R080 src/storage/driver/posix/driver.h src/storage/driver/posix/storage.h
M src/storage/fileRead.c
M src/storage/fileRead.h
M src/storage/fileWrite.c
M src/storage/fileWrite.h
M src/storage/storage.c
M src/storage/storage.h
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/module/storage/fileTest.c

Don't use negations in objects below Storage.

commit   : ab1762663cdcc3564113c5524dc384991228c0a4    
  
author   : David Steele <[email protected]>    
date     : Thu, 13 Sep 2018 17:53:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 Sep 2018 17:53:48 -0400    

Click here for diff

The Storage object represents some some optional parameters as negated if the default is true.  This allows sensible defaults without having to specify most optional parameters.  
  
However, there's no need to propagate this down to functions that require all parameters to be passed -- it makes the code and logging more confusing.  Rename the parameters and update logic to remove negations.  

M doc/xml/release.xml
M src/storage/driver/posix/driverWrite.c
M src/storage/driver/posix/driverWrite.h
M src/storage/storage.c
M test/src/module/storage/fileTest.c

Update comments missed in 5aa458ff.

commit   : 1fb9fe7026f355f86c44fac375a370458c10fc99    
  
author   : David Steele <[email protected]>    
date     : Wed, 12 Sep 2018 18:14:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 12 Sep 2018 18:14:23 -0400    

Click here for diff

M src/common/io/filter/buffer.c
M src/common/io/filter/group.c
M src/common/io/filter/size.c
M src/common/type/buffer.c
M src/common/type/list.c
M src/common/type/string.c
M src/common/type/stringList.c
M src/compress/gzipCompress.c
M src/compress/gzipDecompress.c
M src/info/infoPg.c
M src/storage/storage.c

Simplify debug logging by allowing log functions to return String objects.

commit   : 5aa458ffaecc42cbf6ee37ee4f985d255e31443e    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 18:32:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 18:32:56 -0400    

Click here for diff

Previously, debug log functions had to handle NULLs and truncate output to the available buffer size.  This was verbose for both coding and testing.  
  
Instead, create a function/macro combination that allows log functions to return a simple String object.  The wrapper function takes care of the memory context, handles NULLs, and truncates the log string based on the available buffer size.  

M doc/xml/release.xml
M src/common/io/filter/buffer.c
M src/common/io/filter/buffer.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/size.c
M src/common/io/filter/size.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/compress/gzipCompress.c
M src/compress/gzipCompress.h
M src/compress/gzipDecompress.c
M src/compress/gzipDecompress.h
M src/info/infoPg.c
M src/info/infoPg.h
M src/storage/storage.c
M src/storage/storage.h
M test/define.yaml
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeStringListTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/compress/gzipTest.c
M test/src/module/info/infoPgTest.c

Migrate local, unencrypted, non-S3 archive-get command to C.

commit   : 9b9396c7b7745cc4c765285e24ed131e7f72f343    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 15:42:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 15:42:31 -0400    

Click here for diff

The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs.  Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C.  
  
This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible.  It should not be possible to determine if the C version is running unless debug-level logging is enabled.  

M doc/xml/release.xml
M src/Makefile
A src/command/archive/get/file.c
A src/command/archive/get/file.h
M src/command/archive/get/get.c
M src/compress/gzip.h
M test/define.yaml
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/src/module/archive/getTest.c

Change locking around async process forking to be more test friendly.

commit   : 787e7c295ff64b7f862693784fa901d8b046a3d4    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 14:43:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 14:43:16 -0400    

Click here for diff

The lock is now released before the fork and reacquired after the fork so the parent process no longer needs to worry about clearing the lock.  
  
This is the same locking mechanism that will be used once archive-get-async is exec'd as a separate command, so introduce it now to simplify testing.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M test/src/module/archive/getTest.c

Make archive-get info messages consistent between C and Perl implementations.

commit   : 9e574a37dc4a5c03155a00673a944c2da0a51528    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 12:30:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 12:30:48 -0400    

Click here for diff

The info messages were spread around and logged differently based on the execution path and in some cases logged nothing at all.  
  
Temporarily track the async server status with a flag so that info messages are not output in the async process.  The async process will be refactored as a separate command to be exec'd in a future commit.  

M doc/xml/release.xml
M lib/pgBackRest/Archive/Get/Get.pm
M src/command/archive/get/get.c
M src/perl/embed.auto.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
M test/src/module/archive/getTest.c

Command-line help update missed in cabb3f857.

commit   : 6c1d48b0186f4df40787349d37b35d822e043289    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 10:55:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Sep 2018 10:55:55 -0400    

Click here for diff

M src/config/define.auto.c
M test/src/module/help/helpTest.c

List allowable values for the buffer-size option in the configuration reference.

commit   : cabb3f8577d6de2a6fec7b6e78e68671fc39a65b    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Sep 2018 13:33:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Sep 2018 13:33:39 -0400    

Click here for diff

Contributed by Cynthia Shang.  
Suggested by Stéphane Schildknecht.  

M doc/xml/reference.xml
M doc/xml/release.xml

Improve info command to display the stanza cipher type.

commit   : e351b8c67ca16f4210449c9d51916db1b3ad2416    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Sep 2018 13:09:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Sep 2018 13:09:45 -0400    

Click here for diff

Contributed by Cynthia Shang.  
Suggested by Douglas J Hunley.  

M doc/xml/release.xml
M lib/pgBackRest/Info.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
M test/lib/pgBackRestTest/Module/Info/InfoUnitPerlTest.pm

Improve support for special characters in filenames.

commit   : c688bc86276fcc43bf53747ebbf3259bb8d5a62b    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Sep 2018 10:54:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Sep 2018 10:54:34 -0400    

Click here for diff

% characters caused issues in backup/restore due to filenames being appended directly into a format string.  
  
Reserved XML characters (<>&') caused issues in the S3 driver due to improper escaping.  
  
Add a file with all common special characters to regression testing.  

M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Common/Xml.pm
M lib/pgBackRest/Restore.pm
M lib/pgBackRest/Storage/S3/Driver.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageS3PerlTest.pm

Fix missing missing URI encoding in S3 driver.

commit   : 80ef6fce7529c5f9b8309ac0b8451ddce2a08e2b    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Sep 2018 10:47:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Sep 2018 10:47:00 -0400    

Click here for diff

File names with uncommon characters (e.g. @) caused authentication failures due to S3 encoding them correctly while the S3 driver did not.  
  
Reported by Dan Farrell.  

M doc/xml/release.xml
M lib/pgBackRest/Common/Http/Client.pm
M lib/pgBackRest/Common/Http/Common.pm
M lib/pgBackRest/Storage/S3/Auth.pm
M src/perl/embed.auto.c

Make Valgrind return an error even when a non-fatal issue is detected.

commit   : f7fc8422f780d18d045f42124dc40aa0920fe416    
  
author   : David Steele <[email protected]>    
date     : Fri, 7 Sep 2018 16:50:01 -0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 7 Sep 2018 16:50:01 -0700    

Click here for diff

By default Valgrind does not exit with an error code when a non-fatal error is detected, e.g. unfreed memory.  Use the --error-exitcode option to enabled this behavior.  
  
Update some minor issues discovered in the tests as a result.  Luckily, no issues were missed in the core code.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/module/archive/getTest.c
M test/src/module/common/logTest.c
M test/src/module/compress/gzipTest.c
M test/src/module/postgres/infoTest.c
M test/src/valgrind.suppress

Fix typo in unit test error messages, EXECTED => EXPECTED.

commit   : faaa9a91fda78f8e38d0a479675f08a5b17a7302    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Sep 2018 15:12:22 -0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Sep 2018 15:12:22 -0700    

Click here for diff

M doc/xml/release.xml
M test/src/common/harnessTest.h

Migrate control functions to detect stop files to C from Perl.

commit   : 6e9b6fdca919b9f6fe2bfdb88447ee1f4f7abea7    
  
author   : David Steele <[email protected]>    
date     : Fri, 7 Sep 2018 08:03:05 -0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 7 Sep 2018 08:03:05 -0700    

Click here for diff

Basic functions to detect the presence of stanza or all stop files and error when they are present.  
  
The functionality to detect stop files without error was not migrated. This functionality is only used by stanza-delete and will be migrated with that command.  

M doc/xml/release.xml
M src/Makefile
A src/command/control/control.c
A src/command/control/control.h
M test/define.yaml
A test/src/module/command/controlTest.c

Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C.

commit   : 5bdaa35fa5fea30bf7cdf10a5077cb3b661aa3db    
  
author   : David Steele <[email protected]>    
date     : Fri, 7 Sep 2018 08:00:18 -0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 7 Sep 2018 08:00:18 -0700    

Click here for diff

Also refactor regular expression defines to make them more reusable.  

M doc/xml/release.xml
M src/Makefile
M src/command/archive/common.c
M src/command/archive/common.h
M test/define.yaml
M test/src/module/archive/commonTest.c

Add helper for repository storage.

commit   : 96600760930d1713645ffab211d98d2314818432    
  
author   : David Steele <[email protected]>    
date     : Fri, 7 Sep 2018 07:58:08 -0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 7 Sep 2018 07:58:08 -0700    

Click here for diff

Implement rules for generating paths within the archive part of the repository. Add a helper function, storageRepo(), to create the repository storage based on configuration settings.  
  
The repository storage helper is located in the protocol module because it will support remote file systems in the future, just as the Perl version does.  
  
Also, improve the existing helper functions a bit using string functions that were not available when they were written.  

M doc/xml/release.xml
M src/Makefile
A src/protocol/storage/helper.c
A src/protocol/storage/helper.h
M src/storage/helper.c
M test/define.yaml
A test/src/module/protocol/storageHelperTest.c
M test/src/module/storage/helperTest.c

Info objects now parse JSON and use specified storage.

commit   : 960ad73298fa9c3b9ea60b24428a10ede9a58828    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Sep 2018 10:12:14 -0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Sep 2018 10:12:14 -0700    

Click here for diff

Use JSON code now that it is available and remove temporary hacks used to get things working initially.  
  
Use passed storage objects rather than using storageLocal().  All storage objects in C are still local but this won't always be the case.  
  
Also, move Postgres version conversion functions to postgres/info.c since they have no dependency on the info objects and will likely be useful elsewhere.  

M doc/xml/release.xml
M src/Makefile
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoPg.c
M src/info/infoPg.h
M src/postgres/info.c
M src/postgres/info.h
M test/define.yaml
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/postgres/infoTest.c

Move encryption in mock/archive tests to remote tests.

commit   : de1b74da0cf9c4f182e3a4cde52d0b5163111a44    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Sep 2018 09:35:34 -0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Sep 2018 09:35:34 -0700    

Click here for diff

The new archive-get C code can't run (yet) when encryption is enabled.  Therefore move the encryption tests so we can test the new C code.  We'll move it back when encryption is enabled in C.  
  
Also, push one WAL segment with compression to test decompression in the C code.  

M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm

Fix incorrectly reported error return in info logging.

commit   : 6361a061812b20efeb89db110ebd592a0d52404f    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Sep 2018 21:46:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 Sep 2018 21:46:41 -0400    

Click here for diff

A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.  
  
Also improve info messages when an archive segment is or is not found.  

M doc/xml/release.xml
M lib/pgBackRest/Archive/Get/File.pm
M lib/pgBackRest/Archive/Get/Get.pm
M src/command/command.c
M src/common/exit.c
M src/perl/embed.auto.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/src/module/common/exitTest.c

Posix file functions now differentiate between open and missing errors.

commit   : 800afeef70464da57af023614b1dfeecce5e112c    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Sep 2018 17:47:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 Sep 2018 17:47:23 -0400    

Click here for diff

The Perl functions do so and the integration tests rely on checking for these errors.  This has been exposed as more functionality is moved into C.  
  
Passing the errors types is now a bit complicated so instead use a flag to determine which errors to throw.  

M doc/xml/release.xml
M src/storage/driver/posix/driver.c
M src/storage/driver/posix/driverFile.c
M src/storage/driver/posix/driverFile.h
M src/storage/driver/posix/driverRead.c
M src/storage/driver/posix/driverWrite.c
M test/src/module/config/parseTest.c
M test/src/module/storage/fileTest.c
M test/src/module/storage/storageTest.c

Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL.

commit   : 375ff9f9d283842da3620f22c9c5bdb61249ae9e    
  
author   : David Steele <[email protected]>    
date     : Fri, 31 Aug 2018 16:06:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 31 Aug 2018 16:06:40 -0400    

Click here for diff

Previously an error would be generated if other files were present and not owned by the PostgreSQL user.  This hasn't been a big deal in practice but it could cause issues.  
  
Also add tests to make sure the same logic applies with links to files, i.e. all other files in the directory should be ignored.  This was actually working correctly, but there were no tests for it before.  

M doc/xml/release.xml
M lib/pgBackRest/Manifest.pm
M lib/pgBackRest/Storage/Local.pm
M lib/pgBackRest/Storage/Posix/Driver.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm

Begin v2.06 development.

commit   : 41746b53cd91c0d7203d855f9186c685ea59af47    
  
author   : David Steele <[email protected]>    
date     : Fri, 31 Aug 2018 14:24:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 31 Aug 2018 14:24:36 -0400    

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h