pgBackRest 2.14 commit log

v2.14: Bug Fix and Improvements

commit   : 0f685a33a02adde45792711b957a75391fc5f34e    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 20 May 2019 11:51:58 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 20 May 2019 11:51:58 -0400    

Click here for diff

Bug Fixes:  
  
* Fix segfault when process-max > 8 for archive-push/archive-get. (Reported by Jens Wilke.)  
  
Improvements:  
  
* Bypass database checks when stanza-delete issued with force. (Contributed by Cynthia Shang. Suggested by hatifnatt.)  
* Add configure script for improved multi-platform support.  
  
Documentation Features:  
  
* Add user guides for CentOS/RHEL 6/7.  

M README.md
M doc/RELEASE.md
M doc/resource/exe.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml

Update contributor name.

commit   : 51d67ce1ba932476b2349dbe611f77ab4f15213d    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 09:59:34 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 09:59:34 -0400    

Click here for diff

M doc/xml/release.xml

Add linefeeds for clarity.

commit   : 2b75f57232e3d2c983e0a703b6807dadd0b4c0e6    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Thu, 16 May 2019 09:56:48 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 09:56:48 -0400    

Click here for diff

M test/src/module/command/backupCommonTest.c

Update mock/expire module test matrix so expect tests output.

commit   : 19d8358cba0064392cd519e7bce66bb3dcce3dc9    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Thu, 16 May 2019 09:53:55 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 09:53:55 -0400    

Click here for diff

Also add an error message to prevent regression.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm

Add user guides for CentOS/RHEL 67.

commit   : c51274d1b6a85aeb8ecf1dfffcdc68a503ac3de9    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 08:32:02 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 08:32:02 -0400    

Click here for diff

It would be better if the documentation could be generated on multiple operating systems all in one go, but the doc system currently does not allow vars to be changed once they are set.  
  
The solution is to run the docs for each required OS and stitch the documentation together.  It's not pretty but it works and the automation in release.pl should at least make it easy to use.  

M doc/manifest.xml
M doc/release.pl
M doc/xml/index.xml
M doc/xml/metric.xml
M doc/xml/release.xml
A doc/xml/user-guide-index.xml
M doc/xml/user-guide.xml

Restore index menu url default lost in b85e51d6.

commit   : bc7b42e71811e1b3231c1386496efbdf8338e086    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 07:35:45 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 07:35:45 -0400    

Click here for diff

The url for the menu item referring to the index (i.e. site root page) should use {[project-url-root]}.  
  
This allows the url to be set to different values depending on the location of the index.  

M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/xml/release.xml

Add –out-preserve to preserve contents of output path.

commit   : a4561dc0879fd1fb383ecbeafa6936729a515190    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 07:29:17 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 07:29:17 -0400    

Click here for diff

Useful for building a single set of documentation over multiple doc.pl runs.  

M doc/doc.pl
M doc/lib/BackRestDoc/Html/DocHtmlSite.pm
M doc/lib/BackRestDoc/Latex/DocLatex.pm
M doc/lib/BackRestDoc/Markdown/DocMarkdown.pm
M doc/xml/release.xml

Fix typo in doc.pl command-line help.

commit   : bc30027d730e4ce7090919406eb0d7f5c7f7e51e    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 07:04:28 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 16 May 2019 07:04:28 -0400    

Click here for diff

M doc/doc.pl

Bypass database checks when stanza-delete issued with force.

commit   : 18d4cb574180d279dbbdabd8b87b9dab43115659    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 15 May 2019 13:14:58 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 15 May 2019 13:14:58 -0400    

Click here for diff

Previously it was not possible to delete a stanza if the PostgreSQL server could not be contacted.  
  
Contributed by Cynthia Shang.  
Suggested by Roman.  

M doc/xml/release.xml
M lib/pgBackRest/Stanza.pm
M src/perl/embed.auto.c
M test/expect/real-all-004.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Automate coverage summary report generation.

commit   : 5c1d4bcd0d41590e722f2d184cf64a62b740625c    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 15 May 2019 13:04:56 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 15 May 2019 13:04:56 -0400    

Click here for diff

This report replaces the lcov report that was generated manually for each release.  
  
The lcov report was overly verbose just to say that we have virtually 100% coverage.  

M doc/RELEASE.md
M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/manifest.xml
M doc/release.pl
M doc/resource/html/default.css
A doc/xml/auto/metric-coverage-report.auto.xml
A doc/xml/metric.xml
M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
D test/src/lcov.conf
M test/test.pl
M test/travis.pl

Remove -Wswitch-enum compiler option.

commit   : 5bba72b87445f3edd2c7a2e636ffcb46cac5d9dd    
  
author   : David Steele <david@crunchydata.com>    
date     : Wed, 15 May 2019 12:55:08 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Wed, 15 May 2019 12:55:08 -0400    

Click here for diff

The -Wswitch option included in -Wall provides the same level of coverage and allows enum options to be grouped into default.  

M doc/xml/release.xml
M src/common/type/variant.c
M src/configure
M src/configure.ac
M test/lib/pgBackRestTest/Common/JobTest.pm

Improve coverage in config/parse module.

commit   : 53f6232264f27a029d3f6261c75a16585ade1951    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 14 May 2019 10:35:39 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 14 May 2019 10:35:39 -0400    

Click here for diff

M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c

Improve coverage in config/config module.

commit   : 1e5ab69cada2862cbc458ae322bdc166c244f0f8    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 13 May 2019 19:10:24 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 13 May 2019 19:10:24 -0400    

Click here for diff

M doc/xml/release.xml
M src/config/config.c
M test/src/module/config/configTest.c

Add MAX_ASSIGN() macro.

commit   : 57281d479296627e30cccd00c089f3fb40a076ca    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 13 May 2019 18:05:54 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 13 May 2019 18:05:54 -0400    

Click here for diff

This improves coverage in cases where the compared values may be always ascending or descending.  

M src/common/macro.h
M src/protocol/parallel.c

Error on multiple option alternate names and simplify help command.

commit   : 15a33bf74be84bda822e96c91e37479419b451a9    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 13 May 2019 17:10:41 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 13 May 2019 17:10:41 -0400    

Click here for diff

There are currently no options with multiple alternate (deprecated) names so the code to render them in the help command could not be covered.  
  
Remove the uncovered code and add an error when multiple alternate names are configured.  It's not clear that the current code was handling this correctly, so it will need to be reviewed if it comes up again.  

M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/xml/release.xml
M src/command/help/help.c

Improve coverage in perl/exec module.

commit   : 2d2bec842a2c424d5a36c98dc3a465da1696caf4    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 13 May 2019 13:36:24 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 13 May 2019 13:36:24 -0400    

Click here for diff

M doc/xml/release.xml
M src/perl/exec.c
M test/define.yaml
M test/src/module/perl/execTest.c

Add testUser() and testGroup().

commit   : 431532574b1e2e59979bd6cf3c333979d0817ad2    
  
author   : David Steele <david@crunchydata.com>    
date     : Mon, 13 May 2019 07:51:11 -0400    
  
committer: David Steele <david@crunchydata.com>    
date     : Mon, 13 May 2019 07:51:11 -0400    

Click here for diff

Retrieve the name of the current user/group used for testing.  

M doc/xml/release.xml
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/module/storage/posixTest.c

Improve log performance, simplify macros, rename logWill() to logAny().

commit   : 31d0fe5f50a1a061c3ffbf7d194fafffee6a34b7    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 11 May 2019 18:20:57 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 11 May 2019 18:20:57 -0400    

Click here for diff

Pre-calculate the value used by logAny() to improve performance and make it more likely to be inlined.  
  
Move IF_LOG_ANY() into LOG_INTERNAL() to simplify the macros and improve performance of LOG() and LOG_PID().  If the message has no chance of being logged there's no reason to call logInternal().  
  
Rename logWill() to logAny() because it seems more intuitive.  

M doc/xml/release.xml
M src/command/command.c
M src/common/debug.h
M src/common/exit.c
M src/common/log.c
M src/common/log.h
M src/protocol/client.c
M test/src/common/harnessLog.c
M test/src/module/command/commandTest.c
M test/src/module/common/logTest.c
M test/src/module/config/parseTest.c

Improve macros and coverage rules that were hiding missing coverage.

commit   : 87f36e814ea95696870711018c050725e8e7269f    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 11 May 2019 14:51:51 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 11 May 2019 14:51:51 -0400    

Click here for diff

The branch coverage exclusion rules were overly broad and included functions that ended in a capital letter, which disabled all coverage for the statement.  Improve matching so that all characters in the name must be upper-case for a match.  
  
Some macros with internal branches accepted parameters that might contain conditionals.  This made it impossible to tell which branches belonged to which, and in any case an overzealous exclusion rule was ignoring all branches in such cases.  Add the DEBUG_COVERAGE flag to build a modified version of the macros without any internal branches to be used for coverage testing.  In most cases, the branches were optimizations (like checking logWill()) that improve production performance but are not needed for testing.  In other cases, a parameter needed to be added to the underlying function to handle the branch during coverage testing.  
  
Also tweak the coverage rules so that macros without conditionals are automatically excluded from branch coverage as long as they are not themselves a parameter.  
  
Finally, update tests and code where missing coverage was exposed by these changes.  Some code was updated to remove existing coverage exclusions when it was a simple change.  

M doc/RELEASE.md
M doc/xml/release.xml
M src/command/help/help.c
M src/common/debug.h
M src/common/error.c
M src/common/error.h
M src/common/ini.c
M src/common/io/filter/filter.c
M src/common/io/tls/client.c
M src/common/log.h
M src/common/regExp.c
M src/common/stackTrace.c
M src/common/stackTrace.h
M src/common/type/json.c
M src/common/type/variant.h
M src/config/load.c
M src/config/parse.c
M src/perl/exec.c
M src/protocol/client.c
M src/protocol/parallel.c
M src/storage/posix/storage.c
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessDebug.h
M test/src/lcov.conf
M test/src/module/command/archiveCommonTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/logTest.c
M test/src/module/common/stackTraceTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c

Improve efficiency of FUNCTION_LOG*() macros.

commit   : f819a32cdf94bc799a4902ab6aaa559cd11d4ef8    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 11 May 2019 07:57:49 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 11 May 2019 07:57:49 -0400    

Click here for diff

Call stackTraceTestStop()/stackTraceTestStart() once per block instead of with every param call.  This was done to be cautious but is not necessary and slows down development.  
  
These functions were never built into production so had no impact there.  

M doc/xml/release.xml
M src/common/debug.h

Clarify that sort order must be ascending.

commit   : 7e2f6a6a4365b48fc89672b5d7de4c1d8937caa6    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 12:28:39 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 12:28:39 -0400    

Click here for diff

M src/common/type/stringList.c

Improve filter’s notion of “done” to optimize filter processing.

commit   : f0f105ddeca7b3434d9c3bb78e25247b3c6b7584    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 12:10:46 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 12:10:46 -0400    

Click here for diff

Filters had different ideas about what "done" meant and this added complication to the group filter processing.  For example, gzip decompression would detect end of stream and mark the filter as done before it had been flushed.  
  
Improve the IoFilter interface to give a consistent definition of done across all filters, i.e. no filter can be done until it has started flushing no matter what the underlying driver reports.  This removes quite a bit of tricky logic in the processing loop which tried to determine when a filter was "really" done.  
  
Also improve management of the input buffers by pointing directly to the prior output buffer (or the caller's input) to eliminate loops that set/cleared these buffers.  

M doc/xml/release.xml
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/group.c
M src/common/io/filter/size.c
M src/common/io/filter/size.h
M test/src/module/common/ioTest.c

Improve zero-length content handling in HttpClient object.

commit   : d5fac35fe3efe79f2b39a0e1b5ad3bb1cb1dd173    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 09:53:24 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 09:53:24 -0400    

Click here for diff

If content was zero-length then the IO object was not created.  This put the burden on the caller to test that the IO object existed before checking eof.  
  
Instead, create an IO object even if it will immediately return eof.  This has little cost and makes the calling code simpler.  
  
Also add an explicit test for zero-length files in S3 and a few assertions.  

M doc/xml/release.xml
M src/common/io/http/client.c
M src/storage/s3/read.c
M test/src/module/common/ioHttpTest.c
M test/src/module/storage/s3Test.c

Add –c option to request a C remote.

commit   : 15531151d7b3c07f51c442cb8951a5781bcc39d4    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 08:55:48 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 08:55:48 -0400    

Click here for diff

The rules for when a C remote is required are getting complicated and will get worse when restoreFile() is migrated.  
  
Instead, set the --c option when a C remote is required.  This option will be removed when the remote is entirely implemented in C.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/main.c
M src/perl/embed.auto.c
M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Add pathExists() to Storage object.

commit   : c99c7c458b0a04d1f1a637be70bb4f8e8011feb0    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 08:28:58 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 9 May 2019 08:28:58 -0400    

Click here for diff

The S3 driver did not get an implementation since S3 has a weak notion of paths, and it is not currently required.  

M doc/xml/release.xml
M src/storage/posix/storage.c
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/define.yaml
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Remove dead code missed in 1b486847.

commit   : cb00030ee38148a99df9d5ddcaef64252b0b50c9    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 8 May 2019 18:58:07 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 8 May 2019 18:58:07 -0400    

Click here for diff

This commit removed all Perl references to spool storage but some stuff was left behind.  

M lib/pgBackRest/Protocol/Helper.pm
M lib/pgBackRest/Storage/Helper.pm
M src/perl/embed.auto.c
M test/define.yaml
M test/lib/pgBackRestTest/Module/Storage/StorageHelperPerlTest.pm

Add macros for object free functions.

commit   : f1eea2312104ce7fa87119d6ef50a554b154e954    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 3 May 2019 18:52:54 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 3 May 2019 18:52:54 -0400    

Click here for diff

Most of the *Free() functions are pretty generic so add macros to make creating them as easy as possible.  
  
Create a distinction between *Free() functions that the caller uses to free memory and callbacks that free third-party resources.  There are a number of cases where a driver needs to free resources but does not need a normal *Free() because it is handled by the interface.  
  
Add common/object.h for macros that make object maintenance easier.  This pattern can also be used for many more object functions.  

M doc/xml/release.xml
M src/Makefile.in
M src/common/compress/gzip/compress.c
M src/common/compress/gzip/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/exec.c
M src/common/exec.h
M src/common/ini.c
M src/common/ini.h
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/header.c
M src/common/io/http/header.h
M src/common/io/http/query.c
M src/common/io/http/query.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M src/common/io/write.c
M src/common/io/write.h
M src/common/memContext.c
M src/common/memContext.h
M src/common/object.h
M src/common/regExp.c
M src/common/regExp.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/keyValue.c
M src/common/type/keyValue.h
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/xml.c
M src/common/type/xml.h
M src/common/wait.c
M src/common/wait.h
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoPg.c
M src/info/infoPg.h
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.c
M src/protocol/command.h
M src/protocol/parallel.c
M src/protocol/parallel.h
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M src/protocol/server.c
M src/protocol/server.h
M src/storage/posix/read.c
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/read.h
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/write.c
M src/storage/write.h
M test/define.yaml
M test/src/module/common/compressGzipTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/execTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
A test/src/module/common/objectTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/common/waitTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Various MemContext callback improvements.

commit   : 7ae96949f18af90a9b07f1dd712f939ca7a1ec41    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 3 May 2019 18:09:58 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 3 May 2019 18:09:58 -0400    

Click here for diff

Rename memContextCallback() to memContextCallbackSet() to be more consistent with other parts of the code.  
  
Free all context memory when an exception is thrown from a callback.  Previously only the child contexts would be freed and this resulted in some allocations being lost.  In practice this is probably not a big deal since the process will likely terminate shortly, but there may well be cases where that is not true.  

M doc/xml/release.xml
M src/common/compress/gzip/compress.c
M src/common/compress/gzip/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/exec.c
M src/common/io/tls/client.c
M src/common/memContext.c
M src/common/memContext.h
M src/common/regExp.c
M src/common/type/xml.c
M src/protocol/client.c
M src/storage/posix/read.c
M src/storage/posix/write.c
M src/storage/remote/write.c
M test/src/module/common/memContextTest.c

Add common/macro.h for general-purpose macros.

commit   : 4a20d44c6b118143f1c1607bacac0d4e268f6ffe    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 3 May 2019 17:49:57 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 3 May 2019 17:49:57 -0400    

Click here for diff

Add GLUE() macro which is useful for creating identifiers.  
  
Move MACRO_TO_STR() here and rename it STRINGIFY().  This appears to be the standard name for this type of macro and it is also an awesome name.  

M doc/xml/release.xml
M src/Makefile.in
M src/common/debug.h
A src/common/macro.h
M src/common/type/string.c
M test/src/module/common/debugOffTest.c
M test/src/module/common/debugOnTest.c

Simplify storage object names.

commit   : 32ca27a20b13941cfab27672d6f383a0ba44fa20    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 3 May 2019 15:46:15 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 3 May 2019 15:46:15 -0400    

Click here for diff

Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent.  
  
Also remove the "driver" directory so storage implementations are visible directly under "storage".  

M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Storage/Cifs/Driver.pm
M lib/pgBackRest/Storage/Posix/Driver.pm
M lib/pgBackRest/Storage/S3/Driver.pm
M libc/LibC.xs
M libc/Makefile.PL
M libc/build/lib/pgBackRestLibC/Build.pm
M libc/xs/storage/storage.xs
M src/Makefile.in
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/push/file.c
M src/command/remote/remote.c
M src/common/exec.c
M src/common/io/read.c
M src/common/io/write.c
M src/info/info.c
M src/perl/embed.auto.c
M src/perl/libc.auto.c
R077 src/storage/driver/cifs/storage.c src/storage/cifs/storage.c
R079 src/storage/driver/cifs/storage.h src/storage/cifs/storage.h
M src/storage/helper.c
R091 src/storage/driver/posix/common.c src/storage/posix/common.c
R062 src/storage/driver/posix/common.h src/storage/posix/common.h
R069 src/storage/driver/posix/fileRead.c src/storage/posix/read.c
R065 src/storage/driver/s3/fileRead.h src/storage/posix/read.h
R078 src/storage/driver/posix/storage.c src/storage/posix/storage.c
R079 src/storage/driver/posix/storage.h src/storage/posix/storage.h
R066 src/storage/driver/posix/storage.intern.h src/storage/posix/storage.intern.h
R075 src/storage/driver/posix/fileWrite.c src/storage/posix/write.c
R061 src/storage/driver/posix/fileWrite.h src/storage/posix/write.h
R078 src/storage/fileRead.c src/storage/read.c
R067 src/storage/fileRead.h src/storage/read.h
R073 src/storage/fileRead.intern.h src/storage/read.intern.h
R090 src/storage/driver/remote/protocol.c src/storage/remote/protocol.c
R086 src/storage/driver/remote/protocol.h src/storage/remote/protocol.h
R072 src/storage/driver/remote/fileRead.c src/storage/remote/read.c
R063 src/storage/driver/remote/fileRead.h src/storage/remote/read.h
R075 src/storage/driver/remote/storage.c src/storage/remote/storage.c
R077 src/storage/driver/remote/storage.h src/storage/remote/storage.h
R064 src/storage/driver/remote/storage.intern.h src/storage/remote/storage.intern.h
R075 src/storage/driver/remote/fileWrite.c src/storage/remote/write.c
R064 src/storage/driver/remote/fileWrite.h src/storage/remote/write.h
R067 src/storage/driver/s3/fileRead.c src/storage/s3/read.c
R065 src/storage/driver/s3/fileWrite.h src/storage/s3/read.h
R089 src/storage/driver/s3/storage.c src/storage/s3/storage.c
R078 src/storage/driver/s3/storage.h src/storage/s3/storage.h
R060 src/storage/driver/s3/storage.intern.h src/storage/s3/storage.intern.h
R079 src/storage/driver/s3/fileWrite.c src/storage/s3/write.c
R064 src/storage/driver/posix/fileRead.h src/storage/s3/write.h
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
R076 src/storage/fileWrite.c src/storage/write.c
R060 src/storage/fileWrite.h src/storage/write.h
R083 src/storage/fileWrite.intern.h src/storage/write.intern.h
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/remoteTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/lockTest.c
M test/src/module/info/infoTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Improve type safety of interfaces and drivers.

commit   : 8c712d89ebe10e3d318952a0b4db8cd09f8a580c    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 2 May 2019 17:52:24 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 2 May 2019 17:52:24 -0400    

Click here for diff

The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations.  Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been.  
  
Use void pointers instead so that no casts are required.  Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer.  
  
Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type.  This cuts down on accessor methods and the vast majority of those functions were not being used.  Move functions that are still required to .intern.h.  
  
Remove the special "C" crypto functions that were used in libc and instead use the standard interface.  

M doc/xml/release.xml
M libc/xs/crypto/cipherBlock.xs
M libc/xs/crypto/cipherBlock.xsh
M libc/xs/crypto/hash.xs
M libc/xs/crypto/hash.xsh
M libc/xs/storage/storage.xs
M src/Makefile.in
M src/command/archive/get/file.c
M src/command/archive/push/file.c
M src/command/local/local.c
M src/command/remote/remote.c
M src/common/compress/gzip/compress.c
M src/common/compress/gzip/compress.h
M src/common/compress/gzip/decompress.c
M src/common/compress/gzip/decompress.h
M src/common/crypto/cipherBlock.c
M src/common/crypto/cipherBlock.h
M src/common/crypto/hash.c
M src/common/crypto/hash.h
M src/common/exec.c
M src/common/exec.h
M src/common/io/bufferRead.c
M src/common/io/bufferRead.h
M src/common/io/bufferWrite.c
M src/common/io/bufferWrite.h
M src/common/io/filter/buffer.c
M src/common/io/filter/buffer.h
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/size.c
M src/common/io/filter/size.h
M src/common/io/handleRead.c
M src/common/io/handleRead.h
M src/common/io/handleWrite.c
M src/common/io/handleWrite.h
M src/common/io/http/client.c
M src/common/io/read.c
M src/common/io/read.intern.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M src/common/io/write.c
M src/common/io/write.h
M src/common/io/write.intern.h
A src/common/object.h
M src/info/info.c
M src/perl/libc.auto.c
M src/storage/driver/cifs/storage.c
M src/storage/driver/cifs/storage.h
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileRead.h
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/fileWrite.h
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h
A src/storage/driver/posix/storage.intern.h
M src/storage/driver/remote/fileRead.c
M src/storage/driver/remote/fileRead.h
M src/storage/driver/remote/fileWrite.c
M src/storage/driver/remote/fileWrite.h
M src/storage/driver/remote/storage.c
M src/storage/driver/remote/storage.h
A src/storage/driver/remote/storage.intern.h
M src/storage/driver/s3/fileRead.c
M src/storage/driver/s3/fileRead.h
M src/storage/driver/s3/fileWrite.c
M src/storage/driver/s3/fileWrite.h
M src/storage/driver/s3/storage.c
M src/storage/driver/s3/storage.h
A src/storage/driver/s3/storage.intern.h
M src/storage/fileRead.c
M src/storage/fileRead.h
M src/storage/fileRead.intern.h
M src/storage/fileWrite.c
M src/storage/fileWrite.h
M src/storage/fileWrite.intern.h
M src/storage/helper.c
M src/storage/storage.c
M src/storage/storage.intern.h
M test/define.yaml
M test/lib/pgBackRestTest/Module/Storage/StorageFilterCipherBlockPerlTest.pm
M test/src/common/harnessInfo.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/common/compressGzipTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/protocolTest.c
M test/src/module/info/infoTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Update code count rules missed in 027c2638.

commit   : 28359eea83206f6eeddd3fbc91a52cd42399e529    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 2 May 2019 16:33:23 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 2 May 2019 16:33:23 -0400    

Click here for diff

M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Update comments for storageExists() and driver functions.

commit   : 0c816f94814767dd96040aa2cb299dfa169d59e5    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 2 May 2019 15:19:26 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 2 May 2019 15:19:26 -0400    

Click here for diff

M src/storage/driver/posix/storage.c
M src/storage/driver/remote/storage.c
M src/storage/driver/s3/storage.c
M src/storage/storage.c

Various Buffer improvements.

commit   : 498017bcf02560e101703e89b1119d0b48944835    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 2 May 2019 12:43:09 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 2 May 2019 12:43:09 -0400    

Click here for diff

Add bufDup() and bufNewUsedC().  
  
Arrange bufNewC() params to match bufNewUsedC() since they have always seemed backward.  
  
Fix bufHex() to only render the used portion of the buffer and fix some places where used was not being set correctly.  
  
Use a union to make macro assignments for all legal values without casting.  This is much more likely to catch bad assignments.  

M doc/xml/release.xml
M src/common/crypto/hash.c
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/xml.c
M test/src/module/common/compressGzipTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeXmlTest.c

Use THROW_ON_SYS_ERROR*() to improve code coverage.

commit   : 59234f249e73a4038ee1ba5d2e80f8acc11640ce    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 18:36:57 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 18:36:57 -0400    

Click here for diff

There is only one instance in the core code where this helps. It is mostly helpful in the tests.  
  
There is an argument to be made that only THROW_SYS_ERROR*() variants should be used in the core code to improve test coverage.  If so, that will be the subject of a future commit.  

M doc/xml/release.xml
M src/storage/driver/s3/storage.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/logTest.c

Don’t append strerror() to error message when errno is 0.

commit   : 683b096e187605c5cf16a727cd2c413540e4d150    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 18:03:32 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 18:03:32 -0400    

Click here for diff

Some functions (e.g. getpwnam()/getgrnam()) will return an error but not set errno.  In this case there's no use in appending strerror(), which will be "Success".  This is confusing since an error has just been reported.  
  
At least in the examples above, an error with no errno set just means "missing" and our current error message already conveys that.  

M doc/xml/release.xml
M src/common/error.c
M test/src/module/common/errorTest.c
M test/src/module/storage/posixTest.c

Merge duplicated contributor.

commit   : 6ad44db9a0a30ad933a6ca8012de52558b7b9c02    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 17:30:06 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 17:30:06 -0400    

Click here for diff

M doc/xml/release.xml

Add missing assert.

commit   : 518bba1ed950633746f33474cbfa90f76324e345    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 17:22:01 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 17:22:01 -0400    

Click here for diff

M src/storage/driver/remote/storage.c

Allow StorageFileWrite to set user, group, and modification time.

commit   : af33126cb68915db9644ca941fc2490d09107b0c    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 17:17:14 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 17:17:14 -0400    

Click here for diff

The S3 driver will throw an error if these options are set since they are not supported.  

M doc/xml/release.xml
M src/storage/driver/cifs/storage.c
M src/storage/driver/cifs/storage.h
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/fileWrite.h
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h
M src/storage/driver/remote/fileWrite.c
M src/storage/driver/remote/fileWrite.h
M src/storage/driver/remote/protocol.c
M src/storage/driver/remote/storage.c
M src/storage/driver/remote/storage.h
M src/storage/driver/s3/storage.c
M src/storage/driver/s3/storage.h
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Fix segfault when process-max > 8 for archive-push/archive-get.

commit   : d0c296bd5b67966fd35fd9f70c25788dd6de0c34    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 16:10:27 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 16:10:27 -0400    

Click here for diff

The remote list was at most 9 (based on pg[1-8]-* max index) so anything over 8 wrote into unallocated memory.  
  
The remote for the main process is (currently) stored in position zero so do the same for remotes started from locals, since there should only be one.  The main process will need to start more remotes in the future which is why there is extra space.  
  
Reported by Jens Wilke.  

M doc/xml/release.xml
M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Add missing httpUriEncode() in S3 request.

commit   : c935b1c9e83d6f14644de3aa5f36011234e69423    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 15:20:54 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 15:20:54 -0400    

Click here for diff

This was not a live bug since the files currently handled by C can't have special characters in them.  

M doc/xml/release.xml
M src/storage/driver/s3/storage.c
M test/src/module/storage/s3Test.c

commit   : 62b48d8a5a9bb7aa962e96566073dd58cfdb4436    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 15:14:10 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 15:14:10 -0400    

Click here for diff

This allows info to be retrieved from linked files/paths in a single step.  

M doc/xml/release.xml
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h
M src/storage/driver/remote/storage.c
M src/storage/driver/remote/storage.h
M src/storage/driver/s3/storage.c
M src/storage/driver/s3/storage.h
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/src/module/storage/posixTest.c

Expose handle (file descriptor) from IoWrite when applicable.

commit   : 60edadf71dbbe6c82c5a83bcebee36a16b42ed88    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 14:54:49 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 14:54:49 -0400    

Click here for diff

This is a followup to dee90d3e which exposed file handles for IoRead.  
  
Also expose handle for StorageDriverPosixFileRead missed in dee90d3e.  

M doc/xml/release.xml
M src/common/io/handleWrite.c
M src/common/io/handleWrite.h
M src/common/io/write.c
M src/common/io/write.h
M src/common/io/write.intern.h
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileRead.h
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/fileWrite.h
M test/src/module/common/ioTest.c
M test/src/module/storage/posixTest.c

Add missing space.

commit   : e2141cc05faa4032f7da55afacc57bf17ce11838    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 14:23:37 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Apr 2019 14:23:37 -0400    

Click here for diff

M src/protocol/helper.c

Add configure script for improved multi-platform support.

commit   : 027c2638719dffa9ba99250085c403e89a2a8a9a    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 26 Apr 2019 08:08:23 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Apr 2019 08:08:23 -0400    

Click here for diff

Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile.  
  
Each C file must include "build.auto.conf" before all other includes and defines.  This is enforced by test.pl for includes, but it won't detect incorrect define ordering.  
  
Update packages to call configure and use standard flags to pass options.  

M .travis.yml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M libc/LibC.xs
M src/.gitignore
D src/Makefile
A src/Makefile.in
A src/build.auto.h.in
M src/command/archive/common.c
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/archive/push/file.c
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M src/command/backup/common.c
M src/command/command.c
M src/command/control/control.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/local/local.c
M src/command/remote/remote.c
M src/common/compress/gzip/common.c
M src/common/compress/gzip/compress.c
M src/common/compress/gzip/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/common.c
M src/common/crypto/hash.c
M src/common/debug.c
M src/common/encode.c
M src/common/encode/base64.c
M src/common/error.c
M src/common/exec.c
M src/common/exit.c
M src/common/fork.c
M src/common/ini.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/group.c
M src/common/io/filter/size.c
M src/common/io/handleRead.c
M src/common/io/handleWrite.c
M src/common/io/http/client.c
M src/common/io/http/common.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/io.c
M src/common/io/read.c
M src/common/io/tls/client.c
M src/common/io/write.c
M src/common/lock.c
M src/common/log.c
M src/common/memContext.c
M src/common/regExp.c
M src/common/stackTrace.c
M src/common/time.c
M src/common/type/buffer.c
M src/common/type/convert.c
M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/list.c
M src/common/type/string.c
M src/common/type/stringList.c
M src/common/type/variant.c
M src/common/type/variantList.c
M src/common/type/xml.c
M src/common/wait.c
M src/config/config.c
M src/config/define.c
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M src/config/protocol.c
A src/configure
A src/configure.ac
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoManifest.c
M src/info/infoPg.c
M src/main.c
M src/perl/config.c
M src/perl/exec.c
M src/perl/libc.auto.c
M src/postgres/interface.c
M src/postgres/interface/v083.c
M src/postgres/interface/v084.c
M src/postgres/interface/v090.c
M src/postgres/interface/v091.c
M src/postgres/interface/v092.c
M src/postgres/interface/v093.c
M src/postgres/interface/v094.c
M src/postgres/interface/v095.c
M src/postgres/interface/v096.c
M src/postgres/interface/v100.c
M src/postgres/interface/v110.c
M src/postgres/pageChecksum.c
M src/protocol/client.c
M src/protocol/command.c
M src/protocol/helper.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/server.c
M src/storage/driver/cifs/storage.c
M src/storage/driver/posix/common.c
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/storage.c
M src/storage/driver/remote/fileRead.c
M src/storage/driver/remote/fileWrite.c
M src/storage/driver/remote/protocol.c
M src/storage/driver/remote/storage.c
M src/storage/driver/s3/fileRead.c
M src/storage/driver/s3/fileWrite.c
M src/storage/driver/s3/storage.c
M src/storage/fileRead.c
M src/storage/fileWrite.c
M src/storage/helper.c
M src/storage/storage.c
M test/.gitignore
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
A test/patch/debian-package.patch
A test/patch/rhel-package.patch
M test/test.pl

Update test containers with PostgreSQL minor releases and liblz4.

commit   : 3505559a808855600016dc73c5aec3843e51bfaf    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 24 Apr 2019 13:23:32 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 24 Apr 2019 13:23:32 -0400    

Click here for diff

Update RHEL repos that have changed upstream.  Remove PostgreSQL 9.3 since the RHEL6/7 packages have disappeared.  
  
Remove PostgreSQL versions from U12 that are still getting minor updates so the container does not need to be rebuilt.  
  
LZ4 is included for future development, but this seems like a good time to add it to the containers.  

M doc/xml/release.xml
M test/Vagrantfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm

Add build-max option to set max build processes.

commit   : 1ae8a6a71665c12bec6e20741df29ba4792bf8f8    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 20:52:03 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 20:52:03 -0400    

Click here for diff

Currently this controls make processes via -j.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/test.pl

Reduce ScalityS3 processes since only two are needed.

commit   : c11c936366fd93a6cdb143d7d93a1d8f249ee9c2    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 20:19:31 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 20:19:31 -0400    

Click here for diff

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Add storageInfoList() to get detailed info about all entries in a path.

commit   : 52b0b81976df5583ebe12189d8697dd66d23e84b    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 19:33:55 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 19:33:55 -0400    

Click here for diff

The function provides all the file/path/link information required to build a backup manifest.  
  
Also update storageInfo() to provide the same information for a single file.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h
M src/storage/info.h
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/define.yaml
M test/src/module/storage/posixTest.c
M test/src/valgrind.suppress

Add *Save() functions to most Info objects.

commit   : f492f0571b145d499f4ff9017219794a5a3c7a1b    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 17:08:34 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 17:08:34 -0400    

Click here for diff

At the same time change the way that load constructors work (and are named) so that Ini objects do not persist after the constructors complete.  
  
infoArchiveSave() is excluded from this commit since it is just a trivial call to infoPgSave() and won't be required soon.  

M doc/xml/release.xml
M src/Makefile
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/infoPg.h
M test/define.yaml
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c

Add harnessInfoChecksum/Z() to ease creation of test info files.

commit   : f41112a4634421071d80686ae19042c652576cb3    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 14:02:30 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 14:02:30 -0400    

Click here for diff

M doc/xml/release.xml
A test/src/common/harnessInfo.c
A test/src/common/harnessInfo.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/infoTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c

Extern infoHash() so it can be used by other modules.

commit   : aebac47cd5fa21f6e62ee3dce3ef13abf59ca61f    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 13:58:19 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 13:58:19 -0400    

Click here for diff

This is currently needed only for testing, but there's no reason not to expose the function to everyone.  

M doc/xml/release.xml
M src/info/info.c
M src/info/info.h
M test/src/module/command/infoTest.c
M test/src/module/info/infoBackupTest.c

Add iniSave() and iniMove() to Ini object.

commit   : cddb0c05b444aaed388e53e0cd0b02148307ca87    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 13:03:22 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Apr 2019 13:03:22 -0400    

Click here for diff

iniSave() sorts alphabetically to maintain compatibility with the expect tests, but we plan to change this behavior when the migration is complete.  

M doc/xml/release.xml
M src/common/ini.c
M src/common/ini.h
M test/define.yaml
M test/src/module/common/iniTest.c

Add ioWriteStr() and ioWriteStrLine().

commit   : c650134a0414c154eb21609b363aeceafbfdda60    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 18:46:29 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 18:46:29 -0400    

Click here for diff

These function names make it clearer what is being written.  
  
The old ioWriteLine() has been repurposed to write buffers.  

M doc/xml/release.xml
M src/common/io/http/client.c
M src/common/io/write.c
M src/common/io/write.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/protocol/client.c
M src/protocol/server.c
M src/storage/driver/remote/fileWrite.c
M src/storage/driver/remote/protocol.c
M test/src/module/command/archivePushTest.c
M test/src/module/common/execTest.c
M test/src/module/common/ioTest.c
M test/src/module/protocol/protocolTest.c

Add separate functions to encode/decode each JSON type.

commit   : 81f652137c8f3179aad24d3283a4141b0075ae66    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 18:41:01 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 18:41:01 -0400    

Click here for diff

In most cases the JSON type is known so this is more efficient than converting to Variant first, both in terms of memory and time.  
  
Also rename some of the existing functions for consistency.  

M doc/xml/release.xml
M src/Makefile
M src/command/info/info.c
M src/common/type/json.c
M src/common/type/json.h
M src/info/info.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/perl/config.c
M src/protocol/client.c
M src/protocol/command.c
M src/protocol/server.c
M test/define.yaml
M test/src/module/command/infoTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/protocol/protocolTest.c

Add constant for maximum buffer sizes required by cvt*() functions.

commit   : f100ea0ff44e173299e8f691115b9e7ce68d422b    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 17:52:23 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 17:52:23 -0400    

Click here for diff

Also update Variant to use cvt*() in all cases.  Variant was written before these functions were available and not all cases were updated.  

M doc/xml/release.xml
M src/common/io/tls/client.c
M src/common/type/convert.h
M src/common/type/variant.c

Add true and false String constants.

commit   : f5739051eba3dae6cbee6d1fc0f72c36705c3be1    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 17:17:56 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 17:17:56 -0400    

Click here for diff

M doc/xml/release.xml
M src/common/type/string.c
M src/common/type/string.h

varNewKv() accepts a KeyValue object rather than creating one.

commit   : 47491e3c47e84be17263ce3b897ab3fcd8879a3f    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 16:04:04 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 16:04:04 -0400    

Click here for diff

This allows for more flexibility about when the Variant is created.  

M doc/xml/release.xml
M src/command/info/info.c
M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/variant.c
M src/common/type/variant.h
M src/config/parse.c
M src/perl/config.c
M test/src/module/command/commandTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/config/configTest.c
M test/src/module/perl/configTest.c

Refactor Ini interface to expose String values instead of Variant.

commit   : fdf19e5ab68c64f6c063c265bd3ddc5b39b56620    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 09:00:50 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 09:00:50 -0400    

Click here for diff

Variants were being used to expose String and StringList types but this can be done more simply with an additional method.  
  
Using only strings also allows for a more efficient implementation down the road.  

M doc/xml/release.xml
M src/Makefile
M src/common/ini.c
M src/common/ini.h
M src/config/parse.c
M src/info/info.c
M src/info/infoBackup.c
M src/info/infoPg.c
M test/src/module/common/iniTest.c

Refactor main() as a switch() statement.

commit   : cc39bddc15b7cb5bba5dd8443325ee199e8aa12b    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 08:47:32 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 22 Apr 2019 08:47:32 -0400    

Click here for diff

This makes it clearer what has been migrated and where the new C implementations should go.  

M doc/xml/release.xml
M src/main.c

Fix missing const.

commit   : 38286272596d7b53b88c3f4f081d6c7dfc9360d8    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 21 Apr 2019 14:59:32 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 21 Apr 2019 14:59:32 -0400    

Click here for diff

M src/common/ini.c
M src/common/ini.h

Only process next filter in IoFilterGroup when input buffer is full or flushing.

commit   : e7255be108b6b85bb736934b19abdef32cbd39e8    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 20 Apr 2019 11:25:04 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 20 Apr 2019 11:25:04 -0400    

Click here for diff

This greatly reduces calls to filter processing, which is a performance benefit, but also makes the trace logs smaller and easier to read.  
  
However, this means that ioWriteFlush() will no longer work with filters since a full flush of IoFilterGroup would require an expensive reset.  Currently ioWriteFlush() is not used in this scenario so for now just add an assert to ensure it stays that way.  

M doc/xml/release.xml
M src/common/io/filter/group.c
M src/common/io/write.c
M test/src/module/common/ioTest.c

Add macros to create constant Buffer objects.

commit   : e513c52c0973772465a81a5db2ba18d0f956d686    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 20 Apr 2019 08:16:17 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 20 Apr 2019 08:16:17 -0400    

Click here for diff

These are more efficient than creating buffers in place when needed.  
  
After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them.  This required using the macros in tests which is not the usual pattern.  

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/get/file.c
M src/command/archive/push/file.c
M src/common/io/write.c
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/info/info.c
M src/storage/driver/s3/storage.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Improve performance of non-blocking reads by using maximum buffer size.

commit   : c9168028c6c43d535a5663cb17ce93945951cb7e    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 14:38:11 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 14:38:11 -0400    

Click here for diff

Since the introduction of blocking read drivers (e.g. IoHandleRead, TlsClient) the non-blocking drivers have used the same rules for determining maximum buffer size, i.e. read only as much as requested.  This is necessary so the blocking drivers don't get stuck waiting for data that might not be coming.  
  
Instead mark blocking drivers so IoRead knows how much buffer to allow for the read.  The non-blocking drivers can now request the maximum number of bytes allowed by buffer-size.  

M doc/xml/release.xml
M src/common/exec.c
M src/common/io/handleRead.c
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/tls/client.c
M test/src/module/common/ioTest.c

Update code to use new unsigned int Variant type and config methods.

commit   : 0c866f52c69ee2bd6af20e67af78bad94c4048af    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 11:40:39 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 11:40:39 -0400    

Click here for diff

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M src/command/info/info.c
M src/command/local/local.c
M src/command/remote/remote.c
M src/config/load.c
M src/info/infoBackup.c
M src/protocol/helper.c
M src/storage/driver/remote/fileWrite.c
M src/storage/driver/remote/protocol.c
M test/src/module/command/infoTest.c

Add cfgOptionUInt() and cfgOptionUInt64().

commit   : 9f0829cbf2fcd9be869cac21fbf974fd7ecce019    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 11:29:08 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 11:29:08 -0400    

Click here for diff

These are better than casting from cfgOptionInt() since bounds checking is performed.  

M doc/xml/release.xml
M src/config/config.c
M src/config/config.h
M test/src/module/config/configTest.c

Add unsigned int Variant type.

commit   : 1adcbc5c913bddac3a2078b2c5b7329617fa504c    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 11:22:43 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 11:22:43 -0400    

Click here for diff

This is better than using (unsigned int)varUInt64() because bounds checking is performed.  

M doc/xml/release.xml
M src/common/type/variant.c
M src/common/type/variant.h
M test/define.yaml
M test/src/module/common/typeVariantTest.c

Begin v2.14 development.

commit   : c45ae5f221ba9cc5e65f585920cc1ea23e3dd293    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 08:41:17 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Apr 2019 08:41:17 -0400    

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h