pgBackRest 2.15 (upcoming) commit log

Fix archive retention expiring too aggressively.

commit   : 62715ebf2d8b0585c35cd2ee14d6aabf9cc0f1f8    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 19 Jun 2019 17:49:38 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 19 Jun 2019 17:49:38 -0400    

Click here for diff

The problem expressed when repo1-archive-retention-type was set to diff.  In this case repo1-archive-retention ended up being effectively equal to one, which meant PITR recovery was only possible from the last backup.  WAL required for consistency was still preserved for all backups.  
  
This issue is not present in the C migration committed at 434cd832, which was written before this bug was reported.  Even so, we wanted to note this issue in the release notes in case any other users have been affected.  
  
Fixed by Cynthia Shang.  
Reported by Mohamad El-Rifai.  

M doc/xml/release.xml

Add FAQ on where to find old Debian/Ubuntu packages.

commit   : a7d64bab7abe56132ad1c83eb6bfcca1166e0000    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 19:02:09 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 19:02:09 -0400    

Click here for diff

M doc/xml/faq.xml

Add FAQ to the documentation.

commit   : e2d791394a3577aac53a21e3141e4e337ac8edf0    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Tue, 18 Jun 2019 18:42:47 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 18:42:47 -0400    

Click here for diff

Contributed by Cynthia Shang.  

M doc/manifest.xml
A doc/xml/faq.xml
M doc/xml/release.xml

The expire command is implemented entirely in C.

commit   : 434cd832855e4e189403962753fd8771e29880a4    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 15:19:20 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 15:19:20 -0400    

Click here for diff

This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests.  
  
Along the way at least one bug was fixed, see issue #748.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
D lib/pgBackRest/Expire.pm
M lib/pgBackRest/Main.pm
M src/Makefile.in
A src/command/expire/expire.c
A src/command/expire/expire.h
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
M test/expect/mock-all-002.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
A test/src/module/command/expireTest.c

TLS/HTTP statistics log replacements.

commit   : f88bee7b3321a2c79f0317913ba1e83a56d19c7d    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 10:13:28 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 10:13:28 -0400    

Click here for diff

These statistics can change with any code update so they cause a lot of churn in the expect logs.  

M test/lib/pgBackRestTest/Common/LogTest.pm

Remove hard-coded PostgreSQL user so $PGUSER works.

commit   : 0efdf2576f02b1768fb8805b27571e403ac3cb52    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 07:35:34 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 07:35:34 -0400    

Click here for diff

The PostgreSQL user was hard-coded to the OS user which libpq will automatically use if $PGUSER is not set, so this code was redundant and prevented $PGUSER from working when set.  
  
Suggested by Julian Zhang, Janis Puris.  

M doc/xml/release.xml
M lib/pgBackRest/Db.pm
M src/perl/embed.auto.c

Add S3 info type missed in 7f2f5354.

commit   : 593446718a8997bfd674cf5e54230290cb5bcdea    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 06:45:47 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 18 Jun 2019 06:45:47 -0400    

Click here for diff

File is the only type possible in S3.  

M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Add storage layer for tests and documentation.

commit   : 0a96a2895d7a1e4c5eec402d3742f1d1e25cc126    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 17 Jun 2019 09:16:44 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 17 Jun 2019 09:16:44 -0400    

Click here for diff

The tests and documentation have been using the core storage layer but soon that will depend entirely on the C library, creating a bootstrap problem (i.e. the storage layer will be needed to build the C library).  
  
Create a simplified Posix storage layer to be used by documentation and the parts of the test code that build and execute the actual tests.  The actual tests will still use the core storage driver so they can interact with any type of storage.  

M build/lib/pgBackRestBuild/Build.pm
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M doc/doc.pl
M doc/lib/BackRestDoc/Custom/DocCustomRelease.pm
M doc/release.pl
A test/lib/pgBackRestTest/Common/Storage.pm
A test/lib/pgBackRestTest/Common/StoragePosix.pm
A test/lib/pgBackRestTest/Common/StoragePosixRead.pm
A test/lib/pgBackRestTest/Common/StoragePosixWrite.pm
M test/test.pl

Migrate page checksum filter to C.

commit   : ceafd8e19d416106f50c4ff440bf1cf7fcb5553f    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 17 Jun 2019 07:52:03 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 17 Jun 2019 07:52:03 -0400    

Click here for diff

This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement.  
  
The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.  

M src/Makefile.in
A src/command/backup/pageChecksum.c
A src/command/backup/pageChecksum.h
M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/pageChecksum.c
M src/postgres/pageChecksum.h
M test/define.yaml
M test/src/module/command/backupCommonTest.c
M test/src/module/postgres/pageChecksumTest.c

Add new types to JSON render.

commit   : 892f35a7283462fae0bc3077aeb23c9624a50dea    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 17 Jun 2019 07:42:12 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 17 Jun 2019 07:42:12 -0400    

Click here for diff

Still not a complete list, but we are getting there.  

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Add backup management functions to InfoBackup.

commit   : c64c9c05905d435cd650195ed9470355a4d2be2f    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Mon, 17 Jun 2019 06:59:06 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 17 Jun 2019 06:59:06 -0400    

Click here for diff

Allow current backups to be listed and deleted.  
  
Also expose some constants required by expire and stanza-* commands.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/Makefile.in
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoManifest.h
M test/src/module/info/infoBackupTest.c

Rename info*New() functions to info*NewLoad().

commit   : 44bafc127d9d4e38478aa44d0adc0f34695c6640    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Mon, 17 Jun 2019 06:47:15 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 17 Jun 2019 06:47:15 -0400    

Click here for diff

These names more accurately reflect what the functions do and follow the convention started in Info and InfoPg.  
  
Also remove the ignoreMissing parameter since it was never used.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/command/archive/get/file.c
M src/command/archive/push/push.c
M src/command/info/info.c
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c

Fix filters not processing when there is no input.

commit   : f05fbc54a8f00a56937439fb88eafe8e239781e8    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 14 Jun 2019 08:04:28 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 14 Jun 2019 08:04:28 -0400    

Click here for diff

Some filters (e.g. encryption and compression) produce output even if there is no input.  Since the filter group was marked as "done" initially, processing would not run when there was zero input and that resulted in zero output.  
  
All filters start not done so start the filter group the same way.  

M src/common/io/filter/group.c
M test/src/module/common/ioTest.c

Use retries to wait for test S3 server to start.

commit   : 9ba95e993ba3dbf73f8967a875608980e6380633    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 13 Jun 2019 17:58:33 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 13 Jun 2019 17:58:33 -0400    

Click here for diff

The prior method of tailing the docker log no longer seems reliable.  Instead, keep retrying the make bucket command until it works and show the error if it times out.  

M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm

Create filter group in read/write object’s memory context.

commit   : b9233f7412e44c8daf41db05bd9699fa088ece5d    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 13 Jun 2019 12:44:40 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 13 Jun 2019 12:44:40 -0400    

Click here for diff

The filter group was being created in the parent context so it would go away if the parent context was freed.  

M src/common/io/read.c
M src/common/io/write.c

Remove extra spaces.

commit   : 20be5c9bf39d606b8586ffcf575a7443c6375479    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 13 Jun 2019 12:35:13 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 13 Jun 2019 12:35:13 -0400    

Click here for diff

M src/common/object.h

Integrate S3 storage driver with HTTP client cache.

commit   : fdd375b63d3962845efbb38a7020d852143b97fd    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 11 Jun 2019 16:26:32 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 11 Jun 2019 16:26:32 -0400    

Click here for diff

This allows copying from one S3 object to another.  We generally try to avoid doing this but there are a few cases where it is needed and the tests do it quite a bit.  
  
One thing to look out for here is that reads require the http client to be explicitly released by calling httpClientDone().  This means than clients could grow if they are not released properly.  The http statistics will hopefully alert us if this is happening.  

M src/Makefile.in
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/storage.intern.h
M src/storage/s3/write.c
M test/src/module/storage/s3Test.c

Add HTTP client cache.

commit   : ced42d6511e9b5735a2281dd0e1faec41870fd37    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 11 Jun 2019 10:48:22 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 11 Jun 2019 10:48:22 -0400    

Click here for diff

This cache manages multiple http clients and returns one to the caller that is not busy.  It is the responsibility of the caller to indicate when they are done with a client. If returnContent is set then the client will automatically be marked done.  
  
Also add special handing for HEAD requests to recognize that content-length is informational only and no content is expected.  

M src/Makefile.in
A src/common/io/http/cache.c
A src/common/io/http/cache.h
M src/common/io/http/client.c
M src/common/io/http/client.h
M test/define.yaml
M test/src/module/common/ioHttpTest.c

Add tag to specify minio version to use for documentation build.

commit   : 6e809e578fbd24269205de25cc6e9c84cefc5647    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 11 Jun 2019 10:34:42 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 11 Jun 2019 10:34:42 -0400    

Click here for diff

The new minio major release broke the build.  We'll need to figure that out but for now use the last major version, which is known to work.  

M doc/xml/user-guide.xml

Add info() and infoList() to S3 driver.

commit   : 7f2f535460e3499c1811b5beca9130f84b78c982    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 10 Jun 2019 16:09:38 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 10 Jun 2019 16:09:38 -0400    

Click here for diff

These should be the last functions required to complete the implementation of the S3 driver.  

M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Revert removal of aws cli in 3e1b06ac.

commit   : 9d1b03781f9fe88b868cdfaf4808d84ad1f072fb    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 10 Jun 2019 15:52:51 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 10 Jun 2019 15:52:51 -0400    

Click here for diff

This is required for integration tests to create buckets.  

M test/Vagrantfile

Create general-purpose S3 list function.

commit   : 456391f67c220647e130fdc7a7fc805b075cfd87    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 10 Jun 2019 12:23:04 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 10 Jun 2019 12:23:04 -0400    

Click here for diff

There are already two functions which need file lists and more on the way, so create a general-purpose function to reduce code duplication.  

M src/storage/helper.c
M src/storage/s3/storage.c
M src/storage/s3/storage.h
M test/src/module/storage/s3Test.c

Add missing const qualifiers.

commit   : fb8bd095ca93e2bf77eaec0f7573f961f691c3ab    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 10 Jun 2019 10:37:20 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 10 Jun 2019 10:37:20 -0400    

Click here for diff

M src/common/type/xml.c
M src/common/type/xml.h

Return IoRead/IoWrite from io*FilterGroupSet()

commit   : 78b333790c69aca991f740a9ea6b1982fb70c3d6    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 7 Jun 2019 10:35:44 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 7 Jun 2019 10:35:44 -0400    

Click here for diff

This makes is possible to chain functions more effectively.  

M src/common/io/read.c
M src/common/io/read.h
M src/common/io/write.c
M src/common/io/write.h

Expose storage type.

commit   : e701e8fd846539ec20f2a9be90b0590ceb6a7be6    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 5 Jun 2019 11:50:49 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 5 Jun 2019 11:50:49 -0400    

Click here for diff

Code calling the storage should be driver agnostic but knowing the storage type can be useful for debugging and error reporting.  

M src/storage/storage.c
M src/storage/storage.h
M test/src/module/storage/posixTest.c

Enforce requiring repo-cipher-pass at config parse time.

commit   : 6ff3325c7744a8f6f0e1e965fa673040a2f40a71    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 5 Jun 2019 11:43:17 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 5 Jun 2019 11:43:17 -0400    

Click here for diff

This was not enforced at parse time because repo1-cipher-type could be passed on the command-line even in cases where encryption was not needed by the subprocess.  
  
Filter repo-cipher-type so it is never passed on the command line.  If the subprocess does not have access to the passphrase then knowing the encryption type is useless anyway.  

M build/lib/pgBackRestBuild/Config/Data.pm
M lib/pgBackRest/Config/Config.pm
M src/config/define.auto.c
M src/config/exec.c
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Storage/StorageHelperPerlTest.pm
M test/src/module/config/execTest.c

Use wal_level=replica in the documentation for PostgreSQL >= 9.6.

commit   : d7bd0c58cdd9a434aa5893db16e6bcf8425e26b9    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 5 Jun 2019 07:27:24 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 5 Jun 2019 07:27:24 -0400    

Click here for diff

The documentation was using wal_level=hot_standby which is a deprecated setting.  
  
Also remove the reference to wal_level=archive since it is no longer supported and is not recommended for older versions.  
  
Suggested by Patrick McLaughlin.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Add String to types supported by JSON list render.

commit   : aca11b2fa19337f934fc095e33f014d5e6df71f3    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 17:40:27 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 17:40:27 -0400    

Click here for diff

All types should be supported eventually but for now this answers the need.  

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Fix incorrect error type on missing path.

commit   : 0ab6f3bb87005d5a5d102156730b1e3099894189    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 13:38:05 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 13:38:05 -0400    

Click here for diff

M src/storage/storage.c
M test/src/module/command/archiveGetTest.c
M test/src/module/config/parseTest.c
M test/src/module/storage/posixTest.c

Make working with filter groups less restrictive.

commit   : 4b91259de8d7feec6d33d154032169aa32c879e3    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 12:56:04 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 12:56:04 -0400    

Click here for diff

Filter groups could not be manipulated once they had been assigned to an IO object.  Now they can be freely manipulated up to the time the IO object is opened.  
  
Also, move the filter group into the IO object's context so they don't need to be tracked separately.  

M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/write.c
M src/common/io/write.h
M test/src/module/common/ioTest.c

Remove per-stanza repo cache clear during testing.

commit   : 92e04ea9f4e0eb3f6cd88a72497a026cbd348280    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 10:34:19 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 10:34:19 -0400    

Click here for diff

This was not being used and is not supported by the equivalent C code.  

M lib/pgBackRest/Protocol/Storage/Helper.pm
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Command/CommandArchiveGetPerlTest.pm
M test/lib/pgBackRestTest/Module/Info/InfoInfoArchivePerlTest.pm
M test/lib/pgBackRestTest/Module/Info/InfoInfoBackupPerlTest.pm
M test/lib/pgBackRestTest/Module/Stanza/StanzaAllPerlTest.pm

Use HEAD to check if a file exists on S3.

commit   : 44eb21ea935fdaa4e96d4637c7b62cb5a73d3e77    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 10:05:27 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 10:05:27 -0400    

Click here for diff

The previous implementation searched for the file in a list which worked but was not optimal.  For arbitrary bucket structures it would also produce a false negative if a match was not found in the first 1000 entries.  This was not an issue for our repo structure since the max hits on exists calls is two but it seems worth fixing to avoid future complications.  

M src/common/io/http/client.c
M src/common/io/http/client.h
M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Make C S3 requests use the same host logic as Perl.

commit   : 15b8e3b6af7327179d8bdd2e0c6833ed28005b0b    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 09:39:08 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 09:39:08 -0400    

Click here for diff

The C code was passing the host (if specified) with the request which could force the server into path-style URLs, which are not supported.  
  
Instead, use the Perl logic of always passing bucket.endpoint in the request no matter what host is used for the HTTPS connection.  
  
It's an open question whether we should support path-style URLs but since we don't it's useless to tell the server otherwise.  Note that Amazon S3 has deprecated path-style URLs and they are no longer supported on newly created buckets.  

M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Fix erroneous parameter name.

commit   : 5f92c36b30072edb71b12f79163c37c67d9806bc    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 08:04:33 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 4 Jun 2019 08:04:33 -0400    

Click here for diff

The exists() function only checks for files so the parameter name should reflect that.  

M src/storage/posix/storage.c
M src/storage/remote/storage.c
M src/storage/s3/storage.c
M src/storage/storage.intern.h

Bring back PATH_PAX for platforms that don’t define it.

commit   : 0945d9666b5af1a5893ba0b4537e9cd1f72be478    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 1 Jun 2019 11:09:01 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 1 Jun 2019 11:09:01 -0400    

Click here for diff

This define was replaced in 8c712d89 with limits.h but that caused an issue with the hurd-i386 build for Debian which apparently does not define this value.  

M src/storage/posix/storage.c

Fix mismatched log/test macros.

commit   : a207af79215d05539825c484dbfa9aa2a26cf7d0    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 1 Jun 2019 09:44:40 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 1 Jun 2019 09:44:40 -0400    

Click here for diff

M src/storage/storage.c

Add CPPFLAGS to compile rules.

commit   : 12bca3c43eb2ee0df80ef5e10850657b52fb680b    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 1 Jun 2019 09:28:31 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 1 Jun 2019 09:28:31 -0400    

Click here for diff

This should silence the last of the Debian package warnings.  

M src/Makefile.in
M test/lib/pgBackRestTest/Common/BuildTest.pm

Fix build.flags being removed on each build.

commit   : 388ba0458c37ab4e8b82df0c0fa2b6a4d7462ecb    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 31 May 2019 18:37:31 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 31 May 2019 18:37:31 -0400    

Click here for diff

This was being removed by rsync which forced a full build even when a partial should have been fine.  Rewrite the file after the rsync so it is preserved.  

M test/test.pl

Remove Debian package patch merged to upstream.

commit   : 6cba50c3f23a7a7822ae0fd03ff3b5a3f6d8a32a    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 31 May 2019 18:32:40 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 31 May 2019 18:32:40 -0400    

Click here for diff

D test/patch/debian-package.patch

Fix typo in macro name.

commit   : 3a5fd7b28a79c1564ddd2b5bad36d4d4be358453    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 30 May 2019 11:19:59 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 30 May 2019 11:19:59 -0400    

Click here for diff

M libc/LibC.h
M libc/xs/crypto/cipherBlock.xs
M libc/xs/crypto/hash.xs
M src/perl/libc.auto.c

Add code classification exclusion missed in 3e1b06ac.

commit   : a2ec1253e97805e2a3b990d8bc23c98ab8013d09    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 30 May 2019 10:44:35 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 30 May 2019 10:44:35 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Build all docs with S3 using –var=s3-all=y

commit   : 64260b2e9878944116dc78812a3a21633c5f3d15    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 29 May 2019 08:38:45 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 29 May 2019 08:38:45 -0400    

Click here for diff

Force repo-type=s3 for all tests.  This is not currently the default for any OS builds.  

M doc/lib/BackRestDoc/Common/DocRender.pm
M doc/xml/dtd/doc.dtd
M doc/xml/user-guide.xml

Add internal flag for commands.

commit   : 404284b90ff0f67fe29fa5a7b2831ebf51d459aa    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 28 May 2019 12:18:05 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 28 May 2019 12:18:05 -0400    

Click here for diff

Allow commands to be skipped by default in the command help but still work if help is requested for the command directly.  There may be other uses for the flag in the future.  
  
Update help for ls now that it is exposed.  

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/xml/reference.xml
M src/command/help/help.c
M src/config/config.auto.c
M src/config/config.c
M src/config/config.h
M src/config/define.auto.c
M test/src/module/config/configTest.c

Add ls command.

commit   : 20e5b92f366848ec1464e2272c6af2c29ac7b36d    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 28 May 2019 10:03:48 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 28 May 2019 10:03:48 -0400    

Click here for diff

Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging.  
  
This command is internal-only so the interface may change at any time without notice.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/xml/reference.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/LibCAuto.pm
M src/Makefile.in
A src/command/storage/list.c
A src/command/storage/list.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/define.c
M src/config/parse.auto.c
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
A test/src/module/command/storageTest.c

Move tls/http statistics output to command/command.

commit   : 3b3327eae6dd14f67f1002f094bd33ab7785f765    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 28 May 2019 09:50:59 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 28 May 2019 09:50:59 -0400    

Click here for diff

This module already has the filtering required to keep these messages from being displayed by default for commands that output to stdout (e.g. info).  

M src/Makefile.in
M src/command/command.c
M src/common/exit.c
M test/src/module/command/commandTest.c
M test/src/module/common/exitTest.c

Use minio as local S3 emulator in documentation.

commit   : 3e1b06acaa84399abcfaa8c684f437b63aa38de5    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 27 May 2019 07:37:20 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 27 May 2019 07:37:20 -0400    

Click here for diff

The documentation was relying on a ScalityS3 container built for testing which wasn't very transparent.  Instead, use the stock minio container and configure it in the documentation.  
  
Also, install certificates and CA so that TLS verification can be enabled.  

M doc/lib/BackRestDoc/Common/DocExecute.pm
A doc/resource/fake-cert/.gitignore
A doc/resource/fake-cert/README.md
A doc/resource/fake-cert/ca.crt
A doc/resource/fake-cert/ca.key
A doc/resource/fake-cert/s3-server.crt
A doc/resource/fake-cert/s3-server.key
A doc/resource/fake-cert/s3.cnf
M doc/xml/dtd/doc.dtd
M doc/xml/user-guide.xml
M test/Vagrantfile
M test/lib/pgBackRestTest/Common/HostGroupTest.pm
M test/lib/pgBackRestTest/Common/HostTest.pm

Refactoring path support in the storage module.

commit   : a474ba54c5c9c7bdba6ffa3e92671bb9565889f6    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 26 May 2019 12:41:15 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 26 May 2019 12:41:15 -0400    

Click here for diff

Not all storage types support paths as a physical thing that must be created/destroyed.  Add a feature to determine which drivers use paths and simplify the driver API as much as possible given that knowledge and by implementing as much path logic as possible in the Storage object.  
  
Remove the ignoreMissing parameter from pathSync() since it is not used and makes little sense.  
  
Create a standard list of error messages for the drivers to use and apply them where the code was modified -- there is plenty of work still to be done here.  

M libc/Makefile.PL
M src/Makefile.in
M src/command/restore/file.c
D src/storage/posix/common.c
D src/storage/posix/common.h
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/storage.intern.h
M src/storage/posix/write.c
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/define.yaml
M test/expect/mock-archive-001.log
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/infoTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Log TLS and HTTP statistics on exit.

commit   : 38f28bd52081405321939fec66046bd9ada35c23    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 26 May 2019 12:32:49 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 26 May 2019 12:32:49 -0400    

Click here for diff

These stats measure how efficiently TLS and HTTP are reusing connections (i.e. pipelining).  

M src/Makefile.in
M src/common/exit.c
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M test/src/module/common/exitTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c

Set log level of protocol processing functions to debug.

commit   : 819eda0818669ea9cc45bcd65c57e3b4c7d29129    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 26 May 2019 12:29:29 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 26 May 2019 12:29:29 -0400    

Click here for diff

Setting these to trace effectively made debug level useless in local/remote processes since all debug messages were demoted to trace when called from these functions.  

M src/command/archive/get/protocol.c
M src/command/archive/push/protocol.c
M src/command/restore/protocol.c
M src/config/protocol.c
M src/protocol/server.c
M src/storage/remote/protocol.c

Make info(), pathCreate() and pathSync() optional for storage drivers.

commit   : d12d94c53c466eb808459b98669a877c19b5716b    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 18:09:16 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 18:09:16 -0400    

Click here for diff

These functions are not required for repository storage so make them optional and error if they are not implemented for non-repository storage, .e.g. pg or spool.  
  
The goal is to simplify the drivers (e.g. S3) that are intended only for repository storage.  

M src/storage/s3/storage.c
M src/storage/storage.c
M test/src/module/storage/s3Test.c

Add pathRemove() and remove() to S3 storage.

commit   : 39645fc1a99540063399a7a3b86177a330bd743c    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 14:33:47 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 14:33:47 -0400    

Click here for diff

These functions will be needed for the expire command.  

M src/common/io/http/client.c
M src/common/io/http/client.h
M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Add function to determine if HTTP response code is OK.

commit   : 6c385dfa7a7f489159c06c5a31fd3cd7c77c39a8    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 13:56:38 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 13:56:38 -0400    

Click here for diff

OK responses are a range rather than a single value, so simplify the check with a function.  

M src/common/io/http/client.c
M src/common/io/http/client.h
M src/storage/s3/read.c
M src/storage/s3/storage.c

storageList() returns an empty list by default for missing paths.

commit   : 96770c529be4df408b9d1fc39fc5157e90555375    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 13:12:56 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 13:12:56 -0400    

Click here for diff

The prior behavior was to return NULL so the caller would know the path was missing, but this is rarely useful, complicates the calling code, and increases the chance of segfaults.  
  
The .nullOnMissing param has been added to enable the prior behavior.  

M src/command/archive/common.c
M src/command/archive/get/get.c
M src/command/info/info.c
M src/config/parse.c
M src/storage/storage.c
M src/storage/storage.h
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Notify compiler that cryptoErrorCode() does not return.

commit   : 707e4a29f04a733cff868f37b90f8bf83a02394c    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 11:09:43 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 11:09:43 -0400    

Click here for diff

M src/common/crypto/common.h

Add missing const qualifier.

commit   : 01f3024b5c0a83819d58ee6ccc2edb5109277aaa    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 11:06:11 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 11:06:11 -0400    

Click here for diff

M src/common/type/xml.c
M src/common/type/xml.h

Add missing menus to the new user guides.

commit   : 39cb6248314e21530924886e6e669ac395daeeb1    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 07:45:03 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 07:45:03 -0400    

Click here for diff

Since the CentOS 6/7 user guides were generated as a single page they did not get menus.  Generate the entire site for each user guide so menus are included.  

M doc/release.pl

Use git url for website clone.

commit   : 04f8b4ea52f89d1542b25d2fb0ba28a43ddaba6d    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 07:41:55 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 24 May 2019 07:41:55 -0400    

Click here for diff

M doc/RELEASE.md

Use the git log to ease release note management.

commit   : ec9622cde883c649c1346cbc0b9057e7f3fcb787    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 22 May 2019 18:54:49 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 22 May 2019 18:54:49 -0400    

Click here for diff

The release notes are generally a direct reflection of the git log.  So, ease the burden of maintaining the release notes by using the git log to determine what needs to be added.  
  
Currently only non-dev items are required to be matched to a git commit but the goal is to account for all commits.  
  
The git history cache is generated from the git log but can be modified to correct typos and match the release notes as they evolve.  The commit hash is used to identify commits that have already been added to the cache.  
  
There's plenty more to do here.  For instance, links to the commits for each release item should be added to the release notes.  

M doc/lib/BackRestDoc/Common/DocRender.pm
M doc/lib/BackRestDoc/Custom/DocCustomRelease.pm
M doc/release.pl
A doc/resource/git-history.cache
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml
M test/travis.pl

Reduce log level for all expect tests to detail.

commit   : 86482c7db943375d48fcee0e243ca96bdf50d35c    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 22 May 2019 18:23:44 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 22 May 2019 18:23:44 -0400    

Click here for diff

The C code is designed to be efficient rather than deterministic at the debug log level.  As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level.  
  
Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level.  Change the logging defaults in the config file and remove as many references to log-level-console as possible.  

M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm

Remove duplicated hint.

commit   : e4cc008b982d47ac526962e310c18626e2aefbc2    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 22 May 2019 11:31:38 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 22 May 2019 11:31:38 -0400    

Click here for diff

At some point this hint got added to the underlying code so it is no longer needed here.  

M lib/pgBackRest/Stanza.pm
M src/perl/embed.auto.c
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log

Save cipher-pass key/value missed in f492f057.

commit   : ff9c1bed5de380a6e7df1cbc6870b972d93b3dc4    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 22 May 2019 11:24:18 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 22 May 2019 11:24:18 -0400    

Click here for diff

This value is required when encryption is enabled.  
  
In passing simplify the expression used to skip the checksum when calculating the checksum.  

M src/info/info.c
M test/src/module/info/infoTest.c

Don’t escape / when rendering JSON.

commit   : 3da60f4b5ee8ed663fc5e8d8968709dbaadee1f6    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 22 May 2019 11:10:43 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 22 May 2019 11:10:43 -0400    

Click here for diff

/ is escaped in the spec but the Perl renderer we use does not escape it which leads to checksum mismatches between the two sets of code.  
  
This particular escape seems to be a more recent addition to the spec and is targeted toward embedding JSON in JavaScript.  
  
\/ is still allowed when parsing JSON.  

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c
M test/src/module/perl/configTest.c

Remove RHEL package patch merged to upstream.

commit   : 664054fea94186374dcae84e31e0cd7329c279d0    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 22 May 2019 06:55:31 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 22 May 2019 06:55:31 -0400    

Click here for diff

D test/patch/rhel-package.patch

Remove extraneous parens.

commit   : e1cce9c6147446c026f528cc4afe518de0bcb87f    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 21 May 2019 14:07:35 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 21 May 2019 14:07:35 -0400    

Click here for diff

M src/storage/posix/storage.c
M src/storage/s3/storage.c

Allow separate paragraphs in release items.

commit   : 936b8a289c4884dd22f7ff0d3d624d6e70980512    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 21 May 2019 10:37:30 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 21 May 2019 10:37:30 -0400    

Click here for diff

The first paragraph should match the first line of the commit message as closely as possible.  The following paragraphs add more information.  
  
Release items have been updated back to 2.01.  

M doc/lib/BackRestDoc/Custom/DocCustomRelease.pm
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml

Rename repo-s3-verify-ssl option to repo-s3-verify-tls.

commit   : e3fe3434b4428398ffbee5359f0e0cdec8e55bcb    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 21 May 2019 10:14:41 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 21 May 2019 10:14:41 -0400    

Click here for diff

The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).  
  
The old name will continue to be accepted.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Protocol/Storage/Helper.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/perl/embed.auto.c
M src/storage/helper.c
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageS3CertPerlTest.pm
M test/src/module/command/helpTest.c
M test/src/module/config/parseTest.c

The local command for restore is implemented entirely in C.

commit   : 1bc84c647490d6cf78975b5dde5b1937016d24c6    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 20 May 2019 17:07:37 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 20 May 2019 17:07:37 -0400    

Click here for diff

This is just the part of restore run by the local helper processes, not the entire command.  
  
Even so, various optimizations in the code (like pipelining and optimizations for zero-length files) should make the restore command faster on object stores.  

M doc/xml/release.xml
M lib/pgBackRest/Protocol/Base/Master.pm
M lib/pgBackRest/Protocol/Local/Minion.pm
M lib/pgBackRest/Protocol/Local/Process.pm
M lib/pgBackRest/RestoreFile.pm
M src/Makefile.in
M src/command/local/local.c
A src/command/restore/file.c
A src/command/restore/file.h
A src/command/restore/protocol.c
A src/command/restore/protocol.h
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
A test/src/module/command/restoreTest.c

Add most unimplemented functions to the remote storage driver.

commit   : a839830333490555800377ea095d756765f476e4    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Mon, 20 May 2019 16:19:14 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 20 May 2019 16:19:14 -0400    

Click here for diff

Add pathCreate(), pathRemove(), pathSync(), and remove() to the driver.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M test/define.yaml
M test/src/module/storage/remoteTest.c

Remove extra linefeed.

commit   : bbf2e0d5b0876d3663cc69d10925c6e0d08cd729    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 20 May 2019 12:24:43 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 20 May 2019 12:24:43 -0400    

Click here for diff

M src/common/type/string.c

Begin v2.15 development.

commit   : a9f589b87a09bd37ad903b5c37bcd3000d47840d    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 20 May 2019 12:23:40 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 20 May 2019 12:23:40 -0400    

Click here for diff

M doc/RELEASE.md
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h