pgBackRest 2.8 commit log

v2.08: Minor Improvements and Bug Fixes

commit   : db24ff8df4b2c2de133461df3ab4f28167516dbf    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Jan 2019 22:04:47 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Jan 2019 22:04:47 +0200    

Click here for diff

Bug Fixes:  
  
* Remove request for S3 object info directly after putting it. (Reported by Matt Kunkel.)  
* Correct archive-get-queue-max to be size type. (Reported by Ronan Dunklau.)  
* Add error message when current user uid/gid does not map to a name. (Reported by Camilo Aguilar.)  
* Error when --target-action=shutdown specified for PostgreSQL < 9.5.  
  
Improvements:  
  
* Set TCP keepalives on S3 connections. (Suggested by Ronan Dunklau.)  
* Reorder info command text output so most recent backup is output last. (Contributed by Cynthia Shang. Suggested by Ryan Lambert.)  
* Change file ownership only when required.  
* Redact authentication header when throwing S3 errors. (Suggested by Brad Nicholson.)  

M README.md
M doc/resource/exe.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Update LICENSE.txt for 2019.

commit   : 98ba7ece23637ead3b6a458372f32517aef0c20f    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Jan 2019 18:46:27 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Jan 2019 18:46:27 +0200    

Click here for diff

M LICENSE

Merge common/typeVariantListTest module into common/typeVariantTest.

commit   : 26c888873e3c0956ec1e4a160715433ae3dd9eaf    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Jan 2019 18:14:43 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Jan 2019 18:14:43 +0200    

Click here for diff

These modules are closely related so it makes sense for them to be merged.  

M doc/xml/release.xml
M test/define.yaml
D test/src/module/common/typeVariantListTest.c
M test/src/module/common/typeVariantTest.c

Merge common/typeStringListTest module into common/typeStringTest.

commit   : 07b9176f257148d02c479207180c6af536ad43e1    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Jan 2019 18:05:13 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Jan 2019 18:05:13 +0200    

Click here for diff

These modules are closely related so it makes sense for them to be merged.  

M doc/xml/release.xml
M test/code-count/file-type.yaml
M test/define.yaml
D test/src/module/common/typeStringListTest.c
M test/src/module/common/typeStringTest.c

Add strLstInsert() and strLstInsertZ() to StringList object.

commit   : 493fb78787b0929831899d333484140bfc636a89    
  
author   : David Steele <[email protected]>    
date     : Tue, 1 Jan 2019 17:31:23 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 1 Jan 2019 17:31:23 +0200    

Click here for diff

Use lstInsert() to implement insertions into any position in a StringList.  

M doc/xml/release.xml
M src/common/type/stringList.c
M src/common/type/stringList.h
M test/src/module/common/typeStringListTest.c

Modify general document elements to allow any child element.

commit   : cc6376fdb6bb3bff8ce0b432e23b1fa4ed13a210    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 31 Dec 2018 09:19:38 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 31 Dec 2018 09:19:38 +0200    

Click here for diff

This allows for nesting of elements such as <b> and <i>.  
  
Contributed by Cynthia Shang.  

M doc/xml/dtd/doc.dtd
M doc/xml/release.xml

Allow custom logo for PDF documentation.

commit   : 2b7440ddf313e6dbf8b53443a26cdf211161ad26    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 31 Dec 2018 08:51:00 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 31 Dec 2018 08:51:00 +0200    

Click here for diff

Contributed by Cynthia Shang.  

M doc/lib/BackRestDoc/Latex/DocLatex.pm
M doc/manifest.xml
A doc/resource/latex/blank.eps
M doc/resource/latex/preamble.tex
M doc/xml/release.xml

Add admonitions to documentation renderers.

commit   : 72865ca33ba383fba54da8db0bb8595890a12a3e    
  
author   : Cynthia Shang <[email protected]>    
date     : Sun, 30 Dec 2018 16:40:20 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 30 Dec 2018 16:40:20 +0200    

Click here for diff

Admonitions call out places where the user should take special care.  
  
Support added for HTML, PDF, Markdown and help text renderers.  XML files have been updated accordingly.  
  
Contributed by Cynthia Shang.  

M doc/lib/BackRestDoc/Common/Doc.pm
M doc/lib/BackRestDoc/Common/DocRender.pm
M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/lib/BackRestDoc/Latex/DocLatexSection.pm
M doc/lib/BackRestDoc/Markdown/DocMarkdownRender.pm
M doc/resource/html/default.css
M doc/resource/latex/preamble.tex
M doc/xml/dtd/doc.dtd
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/test.xml
M doc/xml/user-guide.xml
M src/config/define.auto.c
M test/README.md
M test/src/module/help/helpTest.c

Add optional table captions.

commit   : 3dc327fd05882b2dc27e77a7922f5c275a7e2b25    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 20 Dec 2018 23:20:54 +0200    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 Dec 2018 23:20:54 +0200    

Click here for diff

All renderers now support table captions, when requested.  
  
Contributed by Cynthia Shang.  

M doc/lib/BackRestDoc/Html/DocHtmlElement.pm
M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/lib/BackRestDoc/Latex/DocLatexSection.pm
M doc/lib/BackRestDoc/Markdown/DocMarkdownRender.pm
M doc/resource/html/default.css
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml

Escape special characters in latex when not in a code block.

commit   : 8cf8d29f124723d717fa809168c6ea587bd5d592    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 19 Dec 2018 19:13:26 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 19 Dec 2018 19:13:26 +0200    

Click here for diff

A number of common characters are not allowed in latex without being escaped.  
  
Also convert some HTML-specific codes that are used in the documentation.  
  
Contributed by Cynthia Shang.  

M doc/lib/BackRestDoc/Common/DocRender.pm
M doc/xml/release.xml

Set TCP keepalives on S3 connections.

commit   : 23b583336f3095b3e64d6068199b5c449dc776ac    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 Dec 2018 22:12:59 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Dec 2018 22:12:59 +0200    

Click here for diff

Keepalives may help in situations where RST packets are being blocked by a firewall or otherwise do not arrive.  
  
The C code uses select on all reads so it should never block, but add keepalives just in case.  
  
Suggested by Ronan Dunklau.  

M doc/xml/release.xml
M lib/pgBackRest/Common/Http/Client.pm
M src/common/io/tls/client.c
M src/perl/embed.auto.c

Add HTML table rendering and update PDF/Markdown renderers to support header-less tables.

commit   : 96028073cb45fffc5df0d510f3dd143ca10d7d62    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 18 Dec 2018 21:56:39 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Dec 2018 21:56:39 +0200    

Click here for diff

Contributed by Cynthia Shang.  

M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/lib/BackRestDoc/Latex/DocLatexSection.pm
M doc/lib/BackRestDoc/Markdown/DocMarkdownRender.pm
M doc/resource/html/default.css
M doc/resource/latex/preamble.tex
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml

Base menu ordering on natural ordering in the manifest.

commit   : b85e51d6d545df61cf22501c1db09371219a83da    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 14 Dec 2018 18:46:12 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 14 Dec 2018 18:46:12 -0500    

Click here for diff

Menu ordering was alphabetical which is not ideal for a user interface.  
  
Contributed by Cynthia Shang.  

M doc/lib/BackRestDoc/Common/DocManifest.pm
M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/lib/BackRestDoc/Latex/DocLatex.pm
M doc/manifest.xml
M doc/xml/release.xml

Reorder info command text output so most recent backup is output last.

commit   : 35bbb5bd6881e3f405cbd99c3992c32aa5c6d69f    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 14 Dec 2018 18:25:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 14 Dec 2018 18:25:31 -0500    

Click here for diff

After a stanza-upgrade backups for the old cluster are displayed until they expire.  Cluster info was output newest to oldest which meant after an upgrade the most recent backup would no longer be output last.  
  
Update the text output ordering so the most recent backup is always output last.  
  
Contributed by Cynthia Shang.  
Suggested by Ryan Lambert.  

M doc/xml/release.xml
M lib/pgBackRest/Info.pm
M src/command/info/info.c
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Info/InfoUnitPerlTest.pm
M test/src/module/command/infoTest.c

Migrate local info command to C.

commit   : 205525b60780587366b9786750535a55977f0221    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 13 Dec 2018 16:22:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 Dec 2018 16:22:34 -0500    

Click here for diff

The info command will only be executed in C if the repository is local, i.e. not located on a remote repository host.  S3 is considered "local" in this case.  
  
This is a direct migration from Perl to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/Makefile
M src/command/archive/common.c
M src/command/archive/common.h
A src/command/info/info.c
A src/command/info/info.h
M test/define.yaml
A test/src/module/command/infoTest.c

Add infoBackup object to encapsulate the backup.info file.

commit   : e6ef40e8a327d94e5111d21dd056eefbe5d19a86    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 13 Dec 2018 15:46:18 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 Dec 2018 15:46:18 -0500    

Click here for diff

The infoBackup object is the counterpart to the infoArchive object which encapsulates the archive.info file.  
  
Currently the object is read-only, i.e. it is not possible to create a new or modify an existing backup.info file.  
  
There a number of constants that will also be used in the infoManifest object so go ahead and create a module to contain them so they don't need to be moved later.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/Makefile
M src/info/info.c
M src/info/info.h
A src/info/infoBackup.c
A src/info/infoBackup.h
A src/info/infoManifest.c
A src/info/infoManifest.h
M src/info/infoPg.c
M src/info/infoPg.h
M test/define.yaml
A test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c

Add null support to jsonToKv().

commit   : 56d466ce4b39173c777e70e16914a0a76a400dfb    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 13 Dec 2018 15:20:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 13 Dec 2018 15:20:57 -0500    

Click here for diff

Contributed by Cynthia Shang.  

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Add zlib1g-dev to Debian builds.

commit   : d6dfbfd2f3cc2c838d6c1d28ba96076efa9364a6    
  
author   : David Steele <[email protected]>    
date     : Wed, 12 Dec 2018 14:35:44 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 12 Dec 2018 14:35:44 -0500    

Click here for diff

Not all debian-based distros were getting this installed.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Add documentation for building the documentation.

commit   : df947cfcb22928ac4487c74483a49075e35ea15d    
  
author   : David Steele <[email protected]>    
date     : Wed, 12 Dec 2018 13:52:23 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 12 Dec 2018 13:52:23 -0500    

Click here for diff

A basic primer for building the documentation.  Lots that could be added, but it's a start.  

A doc/README.md
M doc/manifest.xml
A doc/xml/documentation.xml
M doc/xml/release.xml

Use absolute paths so that ./doc.pl runs.

commit   : fdc76742c8e98e630fa4219ba8e9aad79c882e40    
  
author   : David Steele <[email protected]>    
date     : Wed, 12 Dec 2018 13:48:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 12 Dec 2018 13:48:31 -0500    

Click here for diff

The relative paths used would not correctly find required libraries and DTD files.  

M doc/doc.pl
M doc/lib/BackRestDoc/Common/Doc.pm
M doc/xml/release.xml

Fix Centos/RHEL 7 documentation builds.

commit   : ee04ebe3142b0f170ef6662db3b6be3e2ba2e32e    
  
author   : David Steele <[email protected]>    
date     : Wed, 12 Dec 2018 11:15:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 12 Dec 2018 11:15:09 -0500    

Click here for diff

This was caused by a new container version that was released around December 5th.  The new version explicitly denies user logons by leaving /var/run/nologin in place after boot.  
  
The solution is to enable the service that is responsible for removing this file on a successful boot.  

M doc/release.pl
M doc/xml/user-guide.xml

Add infoArchiveIdHistoryMatch() to the InfoArchive object.

commit   : 2f15a90d18e0cdc0d91e9915b400acfc78ccaf26    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 18:45:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 18:45:57 -0500    

Click here for diff

Match a PostgreSQL system identifier and version to a pgBackRest archive id.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/info/infoArchive.c
M src/info/infoArchive.h
M test/define.yaml
M test/src/module/info/infoArchiveTest.c

Use cast to make for loop more readable in InfoPg module.

commit   : f0417ee524d1759b1aa5949abcae4769741ce550    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 18:31:49 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 18:31:49 -0500    

Click here for diff

The previous way worked but was a head-scratcher when reading the code.  This cast hopefully makes it a bit more obvious what is going on.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/info/infoPg.c

Remove #define statements in the InfoPg module to conform with newly-adopted coding standards.

commit   : 2514d08d0dc6d28a59bd884a751b734c70e2c0c0    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 18:15:06 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 18:15:06 -0500    

Click here for diff

Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/info/infoPg.c

Rename constants in Info module for consistency.

commit   : fe152b8f29ff794cd6108c8295e35c44fe37c847    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 18:08:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 18:08:26 -0500    

Click here for diff

INFO is generally used as the prefix for info file constants so rename these accordingly.  
  
Also follow newly-adopted coding standards for when #define is required for a static String constant.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/info/info.c

Add path expression for repository backup to the storage helper.

commit   : e3435ec386c19fc7a6f08b9d08d4cee4bf8e36f4    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 17:35:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 17:35:31 -0500    

Click here for diff

This is the counterpart to the archive path expression and constructs paths into the backup part of the repository.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/storage/helper.c
M src/storage/helper.h
M test/src/module/storage/posixTest.c

Allow NULL stanza in storage helper.

commit   : 4f539db8d9c666fa5a9c4e020e387a3b26886b83    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 17:01:33 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 17:01:33 -0500    

Click here for diff

Some commands (e.g. info) do not take a stanza or the stanza is optional.  In that case it is the job of the command to construct the repository path with a stanza as needed.  
  
Update helper functions to omit the stanza from the constructed path when it is NULL.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/storage/helper.c
M test/src/module/storage/posixTest.c

Improve info error messages introduced in 74b72df9.

commit   : cbf514e191783937de63b5a276fcefc426e6f430    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 16:32:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 16:32:41 -0500    

Click here for diff

- Add detail to errors when info files are loaded with incorrect encryption settings.  
- Throw FileMissingError rather than FileOpenError when both copies of the info file are missing.  
- If one file is present (but errors) and the other is missing, then return the error for the file that was present.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/info/info.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-003.log
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/src/module/archive/getTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoTest.c

Add strSizeFormat() to String object.

commit   : 80a3e215213090878e1519f4f4866de2ff6ee67a    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 16:11:51 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 16:11:51 -0500    

Click here for diff

Converts sizes in bytes to a more human-readable form, .e.g. 1KB, 1.1GB.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/common/type/string.c
M src/common/type/string.h
M test/define.yaml
M test/src/module/common/typeStringTest.c

Fix comment typo.

commit   : 947fa6fc2b086f0c2fad602b9921861c0bd18fd3    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 10 Dec 2018 15:43:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Dec 2018 15:43:52 -0500    

Click here for diff

Contributed by Cynthia Shang.  

M src/storage/storage.c

Add configuration to the standby so it works as a primary when promoted.

commit   : 1c5f8f45b68ad87553c2219ffb8e0da0314ad920    
  
author   : David Steele <[email protected]>    
date     : Fri, 7 Dec 2018 12:32:10 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 7 Dec 2018 12:32:10 -0500    

Click here for diff

This code was generated during testing and it seemed a good idea to keep it.  It is only a partial solution since the primary also needs additional configuration to be able to fail back and forth.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Pick pg_switch_wal()/pg_switch_xlog() based on PostgreSQL version.

commit   : 495391c7430ee8f4aafc006f6cea829e6a3ca981    
  
author   : David Steele <[email protected]>    
date     : Fri, 7 Dec 2018 12:19:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 7 Dec 2018 12:19:28 -0500    

Click here for diff

One step in making the documentation build for all versions that we support.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Add error message when current user uid/gid does not map to a name.

commit   : e6abdfb5b864b6526e0e5cb147911bd8ff5c8e15    
  
author   : David Steele <[email protected]>    
date     : Fri, 7 Dec 2018 07:41:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 7 Dec 2018 07:41:26 -0500    

Click here for diff

This condition resulted in a nasty stack trace dump when the undefined value was used later on.  
  
Reported by Camilo Aguilar.  

M doc/xml/release.xml
M lib/pgBackRest/Restore.pm
M src/perl/embed.auto.c

Clarify that storageExists() is only valid for files.

commit   : ac84ffb90156585c3f7a84ed757bc2603da13f31    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 6 Dec 2018 11:57:30 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Dec 2018 11:57:30 -0500    

Click here for diff

Contributed by Cynthia Shang.  

M src/storage/storage.c

Add comments on String and Variant usage regarding memory contexts.

commit   : a72759eb5bad455fc1cca5e1e073628643d6dc0b    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 6 Dec 2018 11:53:38 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Dec 2018 11:53:38 -0500    

Click here for diff

Contributed by Cynthia Shang.  

M src/common/type/string.h
M src/common/type/variant.h

Disable Centos/RHEL 7 documentation builds.

commit   : 11181e69b818452a0b01d388ca5ce0879e6c3a3d    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Dec 2018 09:04:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Dec 2018 09:04:01 -0500    

Click here for diff

These were introduced in 33fa2ede and ran for a day or so before they started failing consistently on CI.  Local builds work fine.  
  
Disable them to free the pipeline for further commits while we determine the issue.  

M doc/release.pl

Change file ownership only when required.

commit   : e73416e9e39118a3eb7a10d4d3f434ef7cc1c4ba    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 17:56:47 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 17:56:47 -0500    

Click here for diff

Previously chown() would be called even when no ownership changes were required.  
  
In most cases changes are not required and it seems better to perform an extra stat() rather than an extra chown().  
  
Also add unit tests for owner() since there weren't any.  

M doc/xml/release.xml
M lib/pgBackRest/Storage/Posix/Driver.pm
M src/perl/embed.auto.c
M test/define.yaml
M test/lib/pgBackRestTest/Module/Storage/StoragePosixPerlTest.pm

Error when --target-action=shutdown specified for PostgreSQL < 9.5.

commit   : e96986a4e1e7aa5302686f559ca2848e1518669f    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 16:21:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 16:21:45 -0500    

Click here for diff

This equaled "promote" on unsupported versions which qualifies as a surprising behavior.  

M doc/xml/release.xml
M lib/pgBackRest/Restore.pm
M src/perl/embed.auto.c

Clarify when target-action is effective and PostgreSQL version support.

commit   : 5b52f020212d75b4f1001727e1a0d8acbae0fe60    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 15:58:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 15:58:45 -0500    

Click here for diff

Suggested by Keith Fiske.  

M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/define.auto.c

Fix documentation typo.

commit   : a0ecf53ead32070045119e8a129d0d07da3662fa    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 13:07:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 13:07:55 -0500    

Click here for diff

M doc/xml/release.xml

Redact authentication header when throwing S3 errors.

commit   : bf873be4aa2688416ec63c8920144a847df142f7    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 12:51:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 12:51:13 -0500    

Click here for diff

The authentication header contains the access key (not the secret key) so don't include it in errors that can be seen at any log level.  
  
Suggested by Brad Nicholson.  

M doc/xml/release.xml
M lib/pgBackRest/Storage/S3/Request.pm
M src/perl/embed.auto.c

Fix test binary name for gprof.

commit   : cc6447356ef436582fdd3501311cb8da0f4f0342    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 09:15:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 09:15:45 -0500    

Click here for diff

This got missed in 1f8931f7 when the test binary was renamed.  
  
Also output call graph along with the flat report.  The flat report is generally most useful but it doesn't hurt to have both.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm

Add Centos/RHEL 7 option to documentation build.

commit   : 33fa2ede7d106db6bf95712747bd0a8ed52f1be5    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 07:12:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Dec 2018 07:12:28 -0500    

Click here for diff

Centos/RHEL 7 builds can be specified with --var=os-type=centos7.  
  
Also added to the documentation release and CI builds.  

M doc/release.pl
M doc/xml/release.xml
M doc/xml/user-guide.xml

Create common if expressions for testing os-type.

commit   : baeff9e4f04c7f86f9cbe84efaf4dc66b3a4add4    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Dec 2018 17:33:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 Dec 2018 17:33:56 -0500    

Click here for diff

These expressions simplify os-type testing.  This will be especially true as more OS types are added.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Documentation may be built with user-specified packages.

commit   : 9e217d02564d30bcdcb865f87323221aa2969169    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Dec 2018 13:17:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 Dec 2018 13:17:55 -0500    

Click here for diff

By default the documentation builds pgBackRest from source, but the documentation is also a good way to smoke-test packages.  
  
Allow a package file to be specified by passing --var=package=/path/to/package.ext.  This works for Debian and CentOS 6 builds.  

M doc/xml/dtd/doc.dtd
M doc/xml/release.xml
M doc/xml/user-guide.xml

Add coding standards for String constants.

commit   : 0db030fa63b74fb6e0a8ad58f1417e39cd1eac9e    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Dec 2018 11:46:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 Dec 2018 11:46:25 -0500    

Click here for diff

M CODING.md
M doc/xml/coding.xml
M doc/xml/release.xml

Update URL for Docker install.

commit   : 14190f9e6c290383676a4332ff5c71e9cc83c66a    
  
author   : David Steele <[email protected]>    
date     : Mon, 3 Dec 2018 12:41:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 3 Dec 2018 12:41:53 -0500    

Click here for diff

As usual the old URL started providing a broken version of Docker rather than producing a clear error message.  This happens once a year or so.  

M doc/xml/release.xml
M test/Vagrantfile

Replace keywords with more flexible if statements.

commit   : 17e611cb883b6086a802c2adfbda384817feaa09    
  
author   : David Steele <[email protected]>    
date     : Sat, 1 Dec 2018 12:40:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 1 Dec 2018 12:40:01 -0500    

Click here for diff

Keywords were extremely limited and prevented us from generating multi-version documentation and other improvements.  
  
Replace keywords with an if statement that can evaluate a Perl expression with variable replacement.  
  
Since keywords were used to generate cache keys, add a --key-var parameter to identify which variables should make up the key.  

M doc/doc.pl
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/lib/BackRestDoc/Common/DocManifest.pm
M doc/lib/BackRestDoc/Common/DocRender.pm
M doc/lib/BackRestDoc/Custom/DocCustomRelease.pm
M doc/manifest.xml
M doc/release.pl
M doc/xml/dtd/doc.dtd
M doc/xml/dtd/manifest.dtd
M doc/xml/release.xml
M doc/xml/user-guide.xml

Fix comment typo.

commit   : 47490ba9600c25f75a9d833d9a42da3633257751    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 30 Nov 2018 18:47:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Nov 2018 18:47:42 -0500    

Click here for diff

Contributed by Cynthia Shang.  

M src/crypto/cipherBlock.c

Clarify that region/endpoint must be configured correctly for the bucket.

commit   : 977506b44615cc9d9d9c581ffd0fd407b7d174d8    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Nov 2018 11:15:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Nov 2018 11:15:52 -0500    

Click here for diff

Suggested by Pritam Barhate.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Correct archive-get-queue-max to be size type.

commit   : 64b97fd7ca3410cd6bca3c01bc438bb104191f78    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Nov 2018 10:55:29 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Nov 2018 10:55:29 -0500    

Click here for diff

This somehow was not configured as a size option when it was added.  It worked, but queue sizes could not be specified in shorthand, e.g. 128GB.  
  
This is not a breaking change because currently configured integer values will be read as bytes.  
  
Reported by Ronan Dunklau.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/config/define.auto.c

Remove request for S3 object info directly after putting it.

commit   : 1ad67644dade53595e2f210d0f75cc247b3ea2a5    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Nov 2018 10:38:02 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Nov 2018 10:38:02 -0500    

Click here for diff

After a file is copied during backup the size is requested from the storage in case it differs from what was written so that repo-size can be reported accurately. This is useful for situations where compression is being done by the filesystem (e.g. ZFS) and what is stored can differ in size from what was written.  
  
In S3 the reported size will always be exactly what was written so there is no need to check the size and doing so immediately can cause problems because the new file might not appear in list commands. This has not been observed on S3 (though it seems to be possible) but it has been reported on the Swift S3 gateway.  
  
Add a driver capability to determine if size needs to be called after a file is written and if not then simply use the number of bytes written for repo-size.  
  
Reported by Matt Kunkel.  

M doc/xml/release.xml
M lib/pgBackRest/Backup/File.pm
M lib/pgBackRest/Storage/Base.pm
M lib/pgBackRest/Storage/Cifs/Driver.pm
M lib/pgBackRest/Storage/S3/FileWrite.pm
M src/perl/embed.auto.c

Pre-build containers for any execute elements marked pre.

commit   : 5d3c8e47f14c9f497866a899bed7d6e25f0fb67f    
  
author   : David Steele <[email protected]>    
date     : Thu, 29 Nov 2018 14:45:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 29 Nov 2018 14:45:15 -0500    

Click here for diff

This allows the documentation to be built more quickly and offline during development when --pre is specified on the command line.  
  
Each host gets a pre-built container with all the execute elements marked pre. As long as the pre elements do not change the container will not need to be rebuilt.  
  
The feature should not be used for CI builds as it may hide errors in the documentation.  

M doc/doc.pl
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/lib/BackRestDoc/Common/DocManifest.pm
M doc/lib/BackRestDoc/Common/DocRender.pm
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml
M doc/xml/user-guide.xml

Improve error message when info files are missing/corrupt.

commit   : 74b72df9dbfbfb00d5b729cb83bef35e77fd73c6    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 18:41:21 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 18:41:21 -0500    

Click here for diff

The previous error message only showed the last error.  In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked.  
  
Show both errors below a generic "unable to load" error.  Details are now given explaining exactly why the primary and copy failed.  
  
Previously if one file could not be loaded a warning would be output.  This has been removed because it is not clear what the user should do in this case.  Should they do a stanza-create --force?  Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.  

M doc/xml/release.xml
M src/info/info.c
M src/info/infoArchive.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-003.log
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/src/module/archive/getTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoTest.c

Add xmlNodeAttribute() to XmlNode object.

commit   : 47687dd13a24a49409f07b99141b508739228791    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 18:10:54 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 18:10:54 -0500    

Click here for diff

Retrieves a node attribute unless it is the root node.  

M src/common/type/xml.c
M src/common/type/xml.h
M test/src/module/common/typeXmlTest.c

Enable encryption for archive-get command in C.

commit   : 7c2fcb63e4ab62e880ebca935fb67e25fa03d5b3    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 14:56:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 14:56:26 -0500    

Click here for diff

The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info.  The rest was plumbing.  
  
The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.  

M doc/xml/release.xml
M src/Makefile
M src/command/archive/get/file.c
M src/command/archive/get/file.h
M src/command/archive/get/get.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/src/module/archive/getTest.c

Add encryption capability to Info* objects.

commit   : 6c23830991558d74ecade1d510fbd50acf5b0f18    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 14:41:10 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 14:41:10 -0500    

Click here for diff

Info objects can be stored encrypted and loaded with a cipher type and passphrase.  

M doc/xml/release.xml
M src/Makefile
M src/command/archive/get/file.c
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoPg.c
M src/info/infoPg.h
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c

Allow arbitary InOut filters to be chained in IoFilterGroup.

commit   : 410a04a58ee59de3055483a2f971c2e42015267e    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 14:20:12 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 14:20:12 -0500    

Click here for diff

If InOut filters were placed next to each other then the second filter would never get a NULL input signaling it to flush.  This arrangement only worked if the second filter had some other indication that it should flush, such as a decompression filter where the flush is indicated in the input stream.  
  
This is not a live issue because currently no InOut filters are chained together.  

M doc/xml/release.xml
M src/common/io/filter/group.c
M test/src/module/common/ioTest.c

Allow arbitrary multiplier and flush character in IoTestFilterMultiply.

commit   : 838cfa44b76ed885f030ebf75649df7eaace6593    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 14:02:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 14:02:14 -0500    

Click here for diff

This allows for more complex test filter chains.  
  
Rename from IoTestFilterDouble to reflect the new functionality.  

M doc/xml/release.xml
M test/src/module/common/ioTest.c

Add IoFilter interface to CipherBlock object.

commit   : 3e254f4cff349f2de95b557b77e3bb0baa1450c9    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 12:42:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 12:42:36 -0500    

Click here for diff

This allows CipherBlock to be used as a filter in an IoFilterGroup.  The C-style functions used by Perl are now deprecated and should not be used for any new code.  
  
Also add functions to convert between cipher names and CipherType.  

M doc/xml/release.xml
M src/Makefile
M src/crypto/cipherBlock.c
M src/crypto/cipherBlock.h
M src/crypto/crypto.c
M src/crypto/crypto.h
M test/define.yaml
M test/src/module/crypto/cipherBlockTest.c
M test/src/module/crypto/cryptoTest.c

Merge cipher.h into crypto.h.

commit   : c3a84ccae08cf7506f26a000344ea1ba2517f9a8    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 12:03:27 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Nov 2018 12:03:27 -0500    

Click here for diff

There is not enough in cipher.h to make it worth having a separate header.  

M doc/xml/release.xml
M src/Makefile
D src/crypto/cipher.h
M src/crypto/cipherBlock.h
M src/crypto/crypto.h
M test/code-count/file-type.yaml

Rename cipherBlock*() functions to cipherBlock*C().

commit   : 7df7cf6e6d1e04fef41ba37cb13020ada6bf4b22    
  
author   : David Steele <[email protected]>    
date     : Tue, 27 Nov 2018 22:49:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 27 Nov 2018 22:49:24 -0500    

Click here for diff

Some of the old names conflict with the new functions that must be created to implement the filter. Rename these to cipherBlock*C() to indicate that they take C-style parameters.  
  
These functions are only used by the Perl LibC code and will be removed or refactored eventually.  

M libc/xs/crypto/cipherBlock.xs
M src/crypto/cipherBlock.c
M src/crypto/cipherBlock.h
M src/perl/libc.auto.c
M test/src/module/crypto/cipherBlockTest.c

Return IoFilterGroup * from ioFilterGroupAdd().

commit   : fea27dbd7eb4ea72a8c792628bba6ed849f68c08    
  
author   : David Steele <[email protected]>    
date     : Tue, 27 Nov 2018 22:02:08 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 27 Nov 2018 22:02:08 -0500    

Click here for diff

This allows filters adds to be chained.  

M doc/xml/release.xml
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/src/module/common/ioTest.c

Explicitly compile with Posix 2001 standard.

commit   : 56ce98b2f0710b77e9df720722d200add670238e    
  
author   : David Steele <[email protected]>    
date     : Sun, 25 Nov 2018 10:06:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 25 Nov 2018 10:06:31 -0500    

Click here for diff

This standard was being selectively applied in modules that needed it.  
  
Instead, apply the standard to all compilation for consistency.  

M doc/xml/release.xml
M libc/Makefile.PL
M src/Makefile
M src/storage/driver/posix/common.c
M src/storage/driver/posix/storage.c
M test/lib/pgBackRestTest/Common/JobTest.pm

Conditional compilation of Perl logic in exit.c.

commit   : 315aa2c4512c2789889d68d1253366c5c75d405b    
  
author   : David Steele <[email protected]>    
date     : Sun, 25 Nov 2018 08:39:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 25 Nov 2018 08:39:41 -0500    

Click here for diff

This file is the only one to contain Perl logic outside of the perl module.  Make the Perl logic conditional to improve reusability.  

M doc/xml/release.xml
M src/Makefile
M src/common/exit.c
M test/lib/pgBackRestTest/Common/JobTest.pm

Remove extraneous use/include statements.

commit   : 78fe642eaeae4f7f73796fdc45b5c2a81e9f3e9f    
  
author   : David Steele <[email protected]>    
date     : Sat, 24 Nov 2018 20:31:35 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 24 Nov 2018 20:31:35 -0500    

Click here for diff

Use conditional loading to make docs work in the absence of LibC.  
  
Somehow this also required a use statement to be added. Perl, go figure.  

M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/lib/BackRestDoc/Latex/DocLatexSection.pm
M doc/xml/release.xml
M src/common/ini.c
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/test.pl

Rename PGBACKREST/BACKREST constants to PROJECT.

commit   : 801e2a5a2ce14f6ef447526c646f0c641c94efa2    
  
author   : David Steele <[email protected]>    
date     : Sat, 24 Nov 2018 19:05:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 24 Nov 2018 19:05:03 -0500    

Click here for diff

This brings consistency between the C and Perl constants and allows for easier code reuse.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/doc.pl
M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/manifest.xml
M doc/release.pl
M doc/xml/release.xml
M lib/pgBackRest/Archive/Base.pm
M lib/pgBackRest/Archive/Get/Async.pm
M lib/pgBackRest/Archive/Push/Async.pm
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Common/Ini.pm
M lib/pgBackRest/Config/Config.pm
M lib/pgBackRest/Db.pm
M lib/pgBackRest/Protocol/Base/Master.pm
M lib/pgBackRest/Protocol/Base/Minion.pm
M lib/pgBackRest/Protocol/Local/Process.pm
M lib/pgBackRest/Storage/Helper.pm
M lib/pgBackRest/Version.pm
M libc/Makefile.PL
M libc/build/lib/pgBackRestLibC/Build.pm
M src/command/command.c
M src/command/help/help.c
M src/common/lock.c
M src/config/parse.c
M src/info/info.c
M src/main.c
M src/perl/embed.auto.c
M src/perl/exec.c
M src/storage/fileWrite.intern.h
M src/version.h
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Env/ConfigEnvTest.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Common/CommonIniPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Protocol/ProtocolCommonMinionPerlTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/commandTest.c
M test/src/module/help/helpTest.c
M test/src/module/info/infoTest.c
M test/test.pl

Add EOF detection to content read in HttpClient.

commit   : 7a1a68883546bbfa0b24fc65054b0a7bf4f09b66    
  
author   : David Steele <[email protected]>    
date     : Sat, 24 Nov 2018 09:12:44 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 24 Nov 2018 09:12:44 -0500    

Click here for diff

If the connection closed before all content was sent httpClientRead() would get stuck in an infinite loop waiting for it to arrive.  
  
EOF should never be reached during content read so immediately error if EOF is detected.  

M src/common/io/http/client.c
M test/src/module/common/ioHttpTest.c

Improve JSON to Variant conversion and add Variant to JSON conversion.

commit   : f4a1751abc9b42d03e0d041fa386a879beb3ee5b    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 23 Nov 2018 16:02:33 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Nov 2018 16:02:33 -0500    

Click here for diff

Add boolean and one-dimensional list types to jsonToKv().  
  
Add varToJson() and kvToJson() to convert Variants and KeyValues to JSON.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/common/type/json.c
M src/common/type/json.h
M test/define.yaml
M test/src/module/common/typeJsonTest.c

Fix release note typo.

commit   : e641c130d30706755775ee328637e3be9d6dba55    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Nov 2018 12:19:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Nov 2018 12:19:56 -0500    

Click here for diff

M doc/xml/release.xml

Enable S3 storage for archive-get command in C.

commit   : beae37533041e63a71b33757e4df933e197e575f    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Nov 2018 12:18:07 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Nov 2018 12:18:07 -0500    

Click here for diff

The only change required was to remove the filter that prevented S3 storage from being used.  The archive-get command did not require any modification which demonstrates that the storage interface is working as intended.  
  
The mock/archive/3 integration test was modified to run S3 storage locally to provide coverage for the new code paths while mock/stanza/3 was modified to run S3 storage remotely to provide coverage for the existing code paths.  This caused some churn in the expect logs but there was no change in behavior.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M test/expect/mock-archive-003.log
M test/expect/mock-stanza-003.log
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm

Allow I/O read interface to explicitly request blocking reads.

commit   : b5690e21a4ad453f004087adf434adf13d109a8c    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Nov 2018 12:01:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Nov 2018 12:01:36 -0500    

Click here for diff

TlsClient introduced a non-blocking read which is required to read protocol messages that are linefeed-terminated rather than a known size. However, in many cases the expected number of bytes is known in advance so in that case it is more efficient to have tlsClientRead() block until all the bytes are read.  
  
Add block parameter to all read functions and use it when a blocking read is required. For most read functions this is a noop, i.e. if the read function never blocks then it can ignore the parameter.  
  
In passing, set the log level of storageNew*() functions to debug to expose more high-level I/O operations.  

M doc/xml/release.xml
M src/common/io/bufferRead.c
M src/common/io/bufferRead.h
M src/common/io/http/client.c
M src/common/io/read.c
M src/common/io/read.intern.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileRead.h
M src/storage/driver/s3/fileRead.c
M src/storage/driver/s3/fileRead.h
M src/storage/storage.c
M test/.gitignore
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/posixTest.c

Add S3 storage driver.

commit   : 256b727a3d13b680ab2a4c7b3669649c857d1daf    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 19:32:49 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 19:32:49 -0500    

Click here for diff

Only the storageNewRead() and storageList() functions are currently implemented, but this is enough to enable S3 for the archive-get command.  

M doc/xml/release.xml
M src/Makefile
A src/storage/driver/s3/fileRead.c
A src/storage/driver/s3/fileRead.h
A src/storage/driver/s3/storage.c
A src/storage/driver/s3/storage.h
M src/storage/helper.c
M src/storage/storage.c
M test/define.yaml
A test/src/module/storage/s3Test.c

Add HttpClient object.

commit   : 72252ed2a102bf9c43d45500de1a29834d36721c    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 19:11:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 19:11:45 -0500    

Click here for diff

A robust HTTP client with pipelining support and automatic retries.  
  
Using a single object to make multiple requests is more efficient because requests are pipelined whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried.  
  
Only the HTTPS protocol is currently supported.  

M doc/xml/release.xml
M src/Makefile
A src/common/io/http/client.c
A src/common/io/http/client.h
A src/common/io/http/common.c
A src/common/io/http/common.h
A src/common/io/http/header.c
A src/common/io/http/header.h
A src/common/io/http/query.c
A src/common/io/http/query.h
M test/define.yaml
A test/src/module/common/ioHttpTest.c

Add TlsClient object.

commit   : 1dd06a6e460049ff888e152f33697b1e49046c30    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 18:43:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 18:43:25 -0500    

Click here for diff

A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed.  
  
This object is intended to be used for multiple TLS connections against a service so tlsClientOpen() can be called each time a new connection is needed. By default, an open connection will be reused for pipelining so the user must be prepared to retry their transaction on a read/write error if the server closes the connection before it can be reused. If this behavior is not desirable then tlsClientClose() may be used to ensure that the next call to tlsClientOpen() will create a new TLS session.  
  
Note that tlsClientRead() is non-blocking unless there are *zero* bytes to be read from the session in which case it will raise an error after the defined timeout. In any case the tlsClientRead()/tlsClientWrite()/tlsClientEof() functions should not generally be called directly. Instead use the read/write interfaces available from tlsClientIoRead()/tlsClientIoWrite().  

M doc/xml/release.xml
M libc/Makefile.PL
M src/Makefile
A src/common/io/tls/client.c
A src/common/io/tls/client.h
M src/crypto/crypto.c
M src/crypto/crypto.h
M test/define.yaml
A test/src/common/harnessTls.c
A test/src/common/harnessTls.h
A test/src/module/common/ioTlsTest.c
M test/src/module/crypto/cryptoTest.c
M test/src/valgrind.suppress

New test containers with static test certificates.

commit   : ac426bc456425d68e33a7facc8393f26b9d17bb1    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 18:13:37 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 18:13:37 -0500    

Click here for diff

Test certificates were generated dynamically but there are advantages to using static certificates.  For example, it possible to use the same certificate between container versions.  Mostly, it is easier to document the certificates if they are not buried deep in the container code.  
  
The new test certificates are initially intended to be used with the C unit tests but they will eventually be used for integration tests as well.  
  
Two new certificates have been defined. See test/certificate/README.md for details.  
  
The old dynamic certificates will be retained until they are replaced.  

M doc/xml/release.xml
A test/certificate/.gitignore
A test/certificate/README.md
A test/certificate/pgbackrest-test-alt-name.cnf
A test/certificate/pgbackrest-test-alt-name.crt
A test/certificate/pgbackrest-test-ca.crt
A test/certificate/pgbackrest-test-ca.key
A test/certificate/pgbackrest-test.cnf
A test/certificate/pgbackrest-test.crt
A test/certificate/pgbackrest-test.key
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Remove embedded semicolon from String constant macros.

commit   : 53e3651ccaaaa1652f53903534974377a2808aec    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 08:30:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Nov 2018 08:30:14 -0500    

Click here for diff

The embedded semicolon led to inconsistent semicolons when using the macro and is not our general convention.  
  
Remove embedded semicolons from the macros and add semicolons in usage where they were not present.  

M doc/xml/release.xml
M src/common/io/filter/size.c
M src/common/type/string.h
M src/config/define.c
M src/config/define.h
M src/config/parse.c
M src/crypto/hash.c
M src/crypto/hash.h
M src/info/info.c
M src/info/infoPg.c
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h

Add interface objects for libxml2.

commit   : bc25db5667b3b05ba47c547591717ee9627f650c    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Nov 2018 20:40:11 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Nov 2018 20:40:11 -0500    

Click here for diff

Add XmlDocument, XmlNode, and XmlNodeList objects as a thin interface layer on libxml2.  
  
This interface is not intended to be comprehensive. Only a few libxml2 capabilities are exposed but more can be added as needed.  

M doc/xml/release.xml
M doc/xml/user-guide.xml
M libc/Makefile.PL
M src/Makefile
A src/common/type/xml.c
A src/common/type/xml.h
M test/define.yaml
M test/lib/pgBackRestTest/Common/JobTest.pm
A test/src/module/common/typeXmlTest.c

Require S3 key options except for local/remote commands.

commit   : 6680130c6f5553e6d0c587dccc57b4a8508c4ab1    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Nov 2018 19:24:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Nov 2018 19:24:53 -0500    

Click here for diff

S3 key options (repo1-s3-key/repo1-s3-key-secret) were not required which meant that users got an ugly assertion when they were missing rather than a tidy configuration error.  
  
Only the local/remote commands need them to be optional.  This is because local/remote commands get all their options from the command line but secrets cannot be passed on the command line.  Instead, secrets are passed to the local/remote commands via the protocol for any operation that needs them.  
  
The configuration system allows required to be set per command so use that to improve the error messages while not breaking the local/remote commands.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/config/define.auto.c

Add testRepoPath() to let C unit tests know where the code repository is located.

commit   : f743d4e92418d73717225599e3dbd03a4af7dec9    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Nov 2018 15:48:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Nov 2018 15:48:56 -0500    

Click here for diff

This allows a C unit test to access data in the code repository that might be useful for testing.  
  
Add testRepoPathSet() to set the repository path.  
  
In passing remove extra whitespace in the TEST_RESULT_VOID() macro.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/test.c

Add ServiceError for errors from a service that can be retried.

commit   : b0659278cc7ab45519d791c10edbb494024f5148    
  
author   : David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 17:22:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 17:22:22 -0500    

Click here for diff

An example is HTTP 5xx errors which should mostly be retried.  

M build/error.yaml
M doc/xml/release.xml
M lib/pgBackRest/Common/ExceptionAuto.pm
M src/common/error.auto.c
M src/common/error.auto.h
M src/perl/embed.auto.c

Add missing LOG_DEBUG() macro.

commit   : 3aa4fda573d95839d617c65880a240d4093951b7    
  
author   : David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 16:17:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 16:17:24 -0500    

Click here for diff

M doc/xml/release.xml
M src/common/log.h

Reduce debug level for infoIni() to test.

commit   : d0e553e4c2a024be44a92dfa5eb0ce77330bcef7    
  
author   : David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 16:05:06 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 16:05:06 -0500    

Click here for diff

Getters should generally not be logging at debug or trace level since it clutters the log.  
  
In passing move the destructor to the end of the file.  

M doc/xml/release.xml
M src/info/info.c
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log

Begin v2.08 development.

commit   : 6532912d51050df4587ec54eee567e37fcb8b74e    
  
author   : David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 10:04:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 10:04:14 -0500    

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h