pgBackRest 2.9 commit log · Life with Open Source

30 Jan 2019, 22:37 by Yogesh Sharma

`v2.09: Minor Improvements and Bug Fixes`

commit   : a89a3761191444ef28b22434f94bc42fd84b83ee    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 22:37:35 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 22:37:35 +0200

Click here for diff

Bug Fixes:  
  
* Fix issue with multiple async status files causing a hard error. (Reported by Vidhya Gurumoorthi, Joe Ayers, Douglas J Hunley.)  
  
Improvements:  
  
* The info command is implemented entirely in C.  
* Simplify info command text message when no stanzas are present by replacing the repository path with "the repository".  
* Add _DARWIN_C_SOURCE flag to Makefile for MacOS builds. (Contributed by Douglas J Hunley.)  
* Update address lookup in C TLS client to use modern methods. (Suggested by Bruno Friedmann.)  
* Include Posix-compliant header for strcasecmp() and fd_set. (Suggested by ucando.)

M README.md
M doc/resource/exe.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h

`Add openssl-devel install to documentation build in 711b3e67.`

commit   : 02dff53d45b3db2daf69ee45ac694696dc051cc3    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 22:02:42 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 22:02:42 +0200

Click here for diff

M doc/README.md
M doc/xml/documentation.xml

`Add latex install to documentation build in 711b3e67.`

commit   : fb342c312591d2980bbbd490a04fb26967d04fb1    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 21:11:52 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 21:11:52 +0200

Click here for diff

M doc/README.md
M doc/xml/documentation.xml

`Allow primary gid for the test user to be different from uid.`

commit   : aa3e5b8c72a75a4f5fea93b543a8fbedcd713f02    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 17:03:17 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 17:03:17 +0200

Click here for diff

Apparently up until now they have always been the same, which is pretty typical.  However, if they were not then ContainerTest.pm was not happy.

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

`Add examples for building a documentation host.`

commit   : 711b3e67cbfc7b404ffe2ce196ffb42159c95040    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 16:53:59 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 16:53:59 +0200

Click here for diff

M doc/README.md
M doc/xml/documentation.xml
M doc/xml/release.xml

`Fix hard-coded repository path.`

commit   : dada50ee26e3891ebfca73e12a8f14080cc8ad7e    
  
author   : David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 13:39:22 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 30 Jan 2019 13:39:22 +0200

Click here for diff

This prevented packages from being passed to the documentation unless they were in the /backrest directory on the host.  
  
Also make the local path /pgbackrest instead of the deprecated /backrest.  
  
Reported by Heath Lord.

M doc/xml/release.xml
M doc/xml/user-guide.xml

`Include Posix-compliant header for fd_set.`

commit   : 8634eb8c43c8666634fe64a49a39c6384caee64d    
  
author   : David Steele <[email protected]>    
date     : Tue, 29 Jan 2019 12:08:14 +0200    
  
committer: David Steele <[email protected]>    
date     : Tue, 29 Jan 2019 12:08:14 +0200

Click here for diff

Similar to f3ae3c4f, add Posix-compliant header to make FreeBSD happy.  
  
Suggested by ucando.

M doc/xml/release.xml
M src/common/io/handleRead.c
M src/common/io/tls/client.c

`Improve P/PP type macro handling.`

commit   : b5a103f2df4999219ef509356ba2c46c292ecccb    
  
author   : David Steele <[email protected]>    
date     : Mon, 28 Jan 2019 22:33:29 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 28 Jan 2019 22:33:29 +0200

Click here for diff

Rather than create _P/_PP variants for every type that needs to pass/return pointers, create FUNCTION_*_P/PP() macros that will properly pass or return any single/double pointer types.  
  
There remain a few unresolved edge cases such as CHARPY but this handles the majority of types well.

M doc/xml/release.xml
M src/command/info/info.c
M src/common/debug.c
M src/common/debug.h
M src/common/encode.c
M src/common/encode/base64.c
M src/common/io/filter/filter.c
M src/common/io/read.c
M src/common/io/tls/client.c
M src/common/io/write.c
M src/common/memContext.c
M src/common/type/buffer.c
M src/common/type/convert.c
M src/common/type/json.c
M src/common/type/list.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/variant.c
M src/common/type/xml.c
M src/config/define.c
M src/config/parse.c
M src/crypto/cipherBlock.c
M src/crypto/crypto.c
M src/crypto/hash.c
M src/info/infoPg.c
M src/info/infoPg.h
M src/postgres/pageChecksum.c
M src/storage/fileRead.c
M src/storage/fileWrite.c
M src/storage/storage.c
M test/src/common/harnessDebug.h
M test/src/module/common/debugOnTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/logTest.c

`Remove unused type parameter from FUNCTION_TEST_RETURN().`

commit   : 8b2a344cfecc2438aafeba7f1c1b7b0572973562    
  
author   : David Steele <[email protected]>    
date     : Mon, 28 Jan 2019 15:06:28 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 28 Jan 2019 15:06:28 +0200

Click here for diff

This parameter was always useless but commit 7333b630 removed all references to it so remove the parameter at all call sites as well.  
  
The original intention was probably to allow logging of TEST return values but that never happened.

M doc/xml/release.xml
M src/command/control/control.c
M src/command/info/info.c
M src/common/debug.h
M src/common/encode.c
M src/common/encode/base64.c
M src/common/exec.c
M src/common/exit.c
M src/common/ini.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/group.c
M src/common/io/filter/size.c
M src/common/io/handleRead.c
M src/common/io/handleWrite.c
M src/common/io/http/client.c
M src/common/io/http/common.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/io.c
M src/common/io/read.c
M src/common/io/tls/client.c
M src/common/io/write.c
M src/common/log.c
M src/common/memContext.c
M src/common/regExp.c
M src/common/time.c
M src/common/type/buffer.c
M src/common/type/convert.c
M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/list.c
M src/common/type/string.c
M src/common/type/stringList.c
M src/common/type/variant.c
M src/common/type/variantList.c
M src/common/type/xml.c
M src/compress/gzipCompress.c
M src/compress/gzipDecompress.c
M src/config/config.c
M src/config/define.c
M src/crypto/cipherBlock.c
M src/crypto/crypto.c
M src/crypto/hash.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/perl/config.c
M src/perl/exec.c
M src/postgres/interface.c
M src/postgres/interface/v083.c
M src/postgres/interface/v084.c
M src/postgres/interface/v090.c
M src/postgres/interface/v091.c
M src/postgres/interface/v092.c
M src/postgres/interface/v093.c
M src/postgres/interface/v094.c
M src/postgres/interface/v095.c
M src/postgres/interface/v096.c
M src/postgres/interface/v100.c
M src/postgres/interface/v110.c
M src/postgres/pageChecksum.c
M src/protocol/client.c
M src/protocol/helper.c
M src/storage/driver/posix/common.c
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/storage.c
M src/storage/driver/remote/fileRead.c
M src/storage/driver/remote/storage.c
M src/storage/driver/s3/fileRead.c
M src/storage/driver/s3/storage.c
M src/storage/fileRead.c
M src/storage/fileWrite.c
M src/storage/helper.c
M src/storage/storage.c

`Improve CONST type macro handling.`

commit   : 7333b6302fb26191a322cc1513fd85c76b066346    
  
author   : David Steele <[email protected]>    
date     : Mon, 28 Jan 2019 09:44:10 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 28 Jan 2019 09:44:10 +0200

Click here for diff

Rather than create a CONST_ variant for every type that needs to be returned const, create a FUNCTION_LOG_RETURN_CONST() macro that will return any type as const.

M doc/xml/release.xml
M src/command/help/help.c
M src/common/debug.h
M src/common/ini.c
M src/common/io/filter/filter.c
M src/common/io/filter/group.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/variant.h
M src/common/type/variantList.h
M src/config/config.c
M src/crypto/crypto.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoPg.c
M src/protocol/client.c
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/storage.h
M src/storage/driver/remote/fileRead.c
M src/storage/driver/s3/fileRead.c
M src/storage/fileRead.c
M src/storage/fileWrite.c

`Allocate extra space for concatenations in the String object.`

commit   : d29aa6128681a3912c1c48ed71cdad9395d0d347    
  
author   : David Steele <[email protected]>    
date     : Sun, 27 Jan 2019 11:50:09 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 27 Jan 2019 11:50:09 +0200

Click here for diff

The string object was reallocating memory with every concatenation which is not very efficient.  This is especially true for JSON rendering which does a lot of concatenations.  
  
Instead allocate a pool of extra memory on the first concatenation (50% of size) to be used for future concatenations and reallocate when needed.  
  
Also add a 1GB size limit to ensure that there are no overflows.

M doc/xml/release.xml
M src/common/type/string.c
M src/common/type/string.h
M test/src/module/common/typeStringTest.c

`Move MACRO_TO_STR() to common/debug.h.`

commit   : 82c2d615b3ecdb851029ad691f5a8061f949535b    
  
author   : David Steele <[email protected]>    
date     : Sun, 27 Jan 2019 11:34:12 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 27 Jan 2019 11:34:12 +0200

Click here for diff

This macro is handy for constructing static message strings so move it where the core code can see it.

M doc/xml/release.xml
M src/common/debug.h
M test/src/common/harnessTest.h

`Update contributor name.`

commit   : 4027123ef19629ff1d5638a75677fa81b3831d03    
  
author   : David Steele <[email protected]>    
date     : Sun, 27 Jan 2019 08:01:17 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 27 Jan 2019 08:01:17 +0200

Click here for diff

M doc/xml/release.xml

`Fix issue with multiple async status files causing a hard error.`

commit   : 8f6d324b2c2576456bd0ba7f691297df1f5aba1e    
  
author   : David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 16:59:54 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 16:59:54 +0200

Click here for diff

Multiple status files were being created by asynchronous archiving if a high-level error occurred after one or more WAL segments had already been transferred successfully.  Error files were being written for every file in the queue regardless of whether it had already succeeded.  To fix this, add an option to skip writing error files when an ok file already exists.  
  
There are other situations where both files might exist (various fsync and filesystem error scenarios) so it seems best to retry in the case that multiple status files are found rather than throwing a hard error (which then means that archiving is completely stuck).  In the case of multiple status files, a warning will be logged to alert the user that something unusual is happening and the command will be retried.  
  
Reported by fpa-postgres, Joe Ayers, Douglas J Hunley.

M doc/xml/release.xml
M lib/pgBackRest/Archive/Common.pm
M lib/pgBackRest/Archive/Push/Async.pm
M src/command/archive/common.c
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Archive/ArchiveCommonPerlTest.pm
M test/src/module/archive/commonTest.c

`Include Posix-compliant header for strcasecmp().`

commit   : f3ae3c4f9d0dd63c583ab27ef1f6aa869db12a3a    
  
author   : David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 13:48:46 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 13:48:46 +0200

Click here for diff

gcc has apparently merged this function in string.h but Posix specifies that it should be in strings.h.  FreeBSD at at least is sticking to the standard.  
  
In the long run it might be better to implement our own strcasecmp() function but for now just add the header.  
  
Suggested by ucando.

M doc/xml/release.xml
M src/common/io/tls/client.c

`Fix release note typo.`

commit   : 1401c023f0cfdd0266144bdd8b181e8194bbd91e    
  
author   : David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 12:02:48 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 12:02:48 +0200

Click here for diff

M doc/xml/release.xml

`Update address lookup in C TLS client to use modern methods.`

commit   : 053972bfe01948a463fed56b11e3522b3b10457f    
  
author   : David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 12:01:18 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 12:01:18 +0200

Click here for diff

The implementation using gethostbyname() was only intended to be used during prototyping but was forgotten when the code was finalized.  
  
Replace it with gettaddrinfo() which is more modern and supports IPv6.  
  
Suggested by Bruno Friedmann.

M doc/xml/release.xml
M src/common/io/tls/client.c
M test/src/module/common/ioTlsTest.c
M test/src/valgrind.suppress

`Replace FileOpenError with HostConnectError in TlsClient.`

commit   : 4d1060ea6bdf60d8e9fc7ca372386f53461ed6d1    
  
author   : David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 10:39:51 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 26 Jan 2019 10:39:51 +0200

Click here for diff

HostConnectError is more appropriate in this context.

M doc/xml/release.xml
M src/common/io/tls/client.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c

`Allow if condition in documentation lists and list items.`

commit   : 33d39d248c5ae6fab6b4bb7564ca85b18786332c    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Jan 2019 16:45:28 +0200    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Jan 2019 16:45:28 +0200

Click here for diff

M doc/xml/dtd/doc.dtd
M doc/xml/release.xml

`Allow if in manifest variables.`

commit   : bec52b6f41a6101ab1357fb5b65ce36faefee9e4    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Jan 2019 11:11:12 +0200    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Jan 2019 11:11:12 +0200

Click here for diff

The code supported this feature but the manifest DTD did now allow it.

M doc/xml/dtd/manifest.dtd
M doc/xml/release.xml

`Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency.`

commit   : db08656537b22c39e5278f9f2770b816278e771a    
  
author   : David Steele <[email protected]>    
date     : Mon, 21 Jan 2019 17:41:59 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 21 Jan 2019 17:41:59 +0200

Click here for diff

Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do.  Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging.  Leave FUNCTION_TEST_* macros as they are.  
  
Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds.  It was difficult to figure out when an assert would be checked with all the different types in play.  When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.

M doc/xml/release.xml
M src/Makefile
M src/command/archive/common.c
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/command.c
M src/command/control/control.c
M src/command/help/help.c
M src/command/info/info.c
M src/common/assert.h
M src/common/debug.h
M src/common/encode.c
M src/common/encode/base64.c
M src/common/error.h
M src/common/exec.c
M src/common/exec.h
M src/common/exit.c
M src/common/fork.c
M src/common/ini.c
M src/common/ini.h
M src/common/io/bufferRead.c
M src/common/io/bufferRead.h
M src/common/io/bufferWrite.c
M src/common/io/bufferWrite.h
M src/common/io/filter/buffer.c
M src/common/io/filter/buffer.h
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/size.c
M src/common/io/filter/size.h
M src/common/io/handleRead.c
M src/common/io/handleRead.h
M src/common/io/handleWrite.c
M src/common/io/handleWrite.h
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/common.c
M src/common/io/http/header.c
M src/common/io/http/header.h
M src/common/io/http/query.c
M src/common/io/http/query.h
M src/common/io/io.c
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M src/common/io/write.c
M src/common/io/write.h
M src/common/io/write.intern.h
M src/common/lock.c
M src/common/log.c
M src/common/memContext.c
M src/common/memContext.h
M src/common/regExp.c
M src/common/regExp.h
M src/common/stackTrace.c
M src/common/time.c
M src/common/time.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/convert.c
M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/keyValue.h
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/common/type/variantList.c
M src/common/type/variantList.h
M src/common/type/xml.c
M src/common/type/xml.h
M src/common/wait.c
M src/common/wait.h
M src/compress/gzipCompress.c
M src/compress/gzipCompress.h
M src/compress/gzipDecompress.c
M src/compress/gzipDecompress.h
M src/config/config.c
M src/config/define.c
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M src/crypto/cipherBlock.c
M src/crypto/cipherBlock.h
M src/crypto/crypto.c
M src/crypto/hash.c
M src/crypto/hash.h
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoPg.c
M src/info/infoPg.h
M src/main.c
M src/perl/config.c
M src/perl/exec.c
M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/interface/v083.c
M src/postgres/interface/v084.c
M src/postgres/interface/v090.c
M src/postgres/interface/v091.c
M src/postgres/interface/v092.c
M src/postgres/interface/v093.c
M src/postgres/interface/v094.c
M src/postgres/interface/v095.c
M src/postgres/interface/v096.c
M src/postgres/interface/v100.c
M src/postgres/interface/v110.c
M src/postgres/pageChecksum.c
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/helper.c
M src/storage/driver/posix/common.c
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileRead.h
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/fileWrite.h
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h
M src/storage/driver/remote/fileRead.c
M src/storage/driver/remote/fileRead.h
M src/storage/driver/remote/storage.c
M src/storage/driver/remote/storage.h
M src/storage/driver/s3/fileRead.c
M src/storage/driver/s3/fileRead.h
M src/storage/driver/s3/storage.c
M src/storage/driver/s3/storage.h
M src/storage/fileRead.c
M src/storage/fileRead.h
M src/storage/fileRead.intern.h
M src/storage/fileWrite.c
M src/storage/fileWrite.h
M src/storage/fileWrite.intern.h
M src/storage/helper.c
M src/storage/info.h
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/define.yaml
M test/src/common/harnessDebug.h
M test/src/module/common/assertOffTest.c
M test/src/module/common/assertOnTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/logTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/stackTraceTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/waitTest.c
M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/postgres/pageChecksumTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

`The info command is implemented entirely in C.`

commit   : d245f8eb425322f0efcd05ab4c12b3b45cc87138    
  
author   : David Steele <[email protected]>    
date     : Mon, 21 Jan 2019 13:51:45 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 21 Jan 2019 13:51:45 +0200

Click here for diff

The C info code has already been committed but this commit wires it into main.  
  
Also remove the info Perl code and tests since they are no longer called.

M doc/xml/release.xml
D lib/pgBackRest/Info.pm
M lib/pgBackRest/Protocol/Helper.pm
M lib/pgBackRest/Protocol/Remote/Minion.pm
M src/command/info/info.c
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-003.log
D test/lib/pgBackRestTest/Module/Info/InfoUnitPerlTest.pm

`Increase timeout in storage/s3 module to improve reliability.`

commit   : f79af47bd4ffff693e1a882955323fe73972eeb6    
  
author   : David Steele <[email protected]>    
date     : Sat, 19 Jan 2019 20:44:40 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 19 Jan 2019 20:44:40 +0200

Click here for diff

M doc/xml/release.xml
M test/src/module/storage/s3Test.c

`Fix escaping in JSON string parser.`

commit   : e57f1b529327ebeca41405b6b57602013de9d625    
  
author   : David Steele <[email protected]>    
date     : Sat, 19 Jan 2019 12:57:40 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 19 Jan 2019 12:57:40 +0200

Click here for diff

M doc/xml/release.xml
M src/common/type/json.c
M test/src/module/common/typeJsonTest.c
M test/src/module/storage/remoteTest.c

`Add remote storage objects.`

commit   : 7355248d6b2b1db0a7ca997db0dc2943331325a0    
  
author   : David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 22:04:37 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 22:04:37 +0200

Click here for diff

This is a partial implementation of remote storage with just enough functionality to get the info command working.  The client is written in C but the server is still in Perl, which limits progress until a C server is written.

M doc/xml/release.xml
M src/Makefile
M src/storage/driver/posix/storage.c
A src/storage/driver/remote/fileRead.c
A src/storage/driver/remote/fileRead.h
A src/storage/driver/remote/storage.c
A src/storage/driver/remote/storage.h
M src/storage/driver/s3/storage.c
M src/storage/helper.c
M src/storage/storage.c
M src/storage/storage.intern.h
M test/define.yaml
A test/src/module/storage/remoteTest.c

`Add ProtocolClient object and helper functions.`

commit   : 88201f37a3b7156dcf6f4aaa659136215485c40d    
  
author   : David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 21:32:51 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 21:32:51 +0200

Click here for diff

This is a complete protocol client implementation in C.  
  
Currently there is no C server implementation so the C client is talking to a Perl server.  This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure.  While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run.  
  
Just as in Perl there are helper functions to make constructing protocol objects easier.  Currently only repository remotes are supported.

M doc/xml/release.xml
M src/Makefile
M src/common/exec.c
M src/common/exec.h
M src/common/exit.c
A src/protocol/client.c
A src/protocol/client.h
A src/protocol/helper.c
A src/protocol/helper.h
M test/define.yaml
M test/src/module/common/execTest.c
A test/src/module/protocol/protocolTest.c

`Fix comment typo.`

commit   : 0986db630cf3c931ccb135a86e60ee098fdc2dec    
  
author   : David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 13:15:43 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 13:15:43 +0200

Click here for diff

M test/src/module/common/ioTest.c

`Add Exec object.`

commit   : 9cac403f610ff3fd4a176b9cf5a7c0543bdd892c    
  
author   : David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 11:45:40 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 11:45:40 +0200

Click here for diff

Executes a child process and allows the calling process to communicate with it using read/write io.  
  
This object is specially tailored to implement the protocol layer and may or may not be generally applicable to general purpose  
execution.

M build/error.yaml
M doc/xml/release.xml
M lib/pgBackRest/Common/ExceptionAuto.pm
M src/Makefile
M src/common/error.auto.c
M src/common/error.auto.h
A src/common/exec.c
A src/common/exec.h
M src/common/io/handleRead.c
M src/perl/embed.auto.c
M test/define.yaml
A test/src/module/common/execTest.c

`Add ioReadBuf() to easily read into a buffer.`

commit   : 797f8098d1383d05960566174859f75d4e835613    
  
author   : David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 11:14:44 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 18 Jan 2019 11:14:44 +0200

Click here for diff

Moves some boilerplate into a function and makes it easier to get coverage in cases where a single buffer read captures all the data.

M doc/xml/release.xml
M src/Makefile
M src/common/io/io.c
M src/common/io/io.h
M test/src/module/common/ioTest.c

`Add cfgExecParam() to generate parameters for executing commands.`

commit   : 06d41b4dc0f84f756b57d44cc2e2ceb6b5aec493    
  
author   : David Steele <[email protected]>    
date     : Thu, 17 Jan 2019 22:29:19 +0200    
  
committer: David Steele <[email protected]>    
date     : Thu, 17 Jan 2019 22:29:19 +0200

Click here for diff

Parameters for the local/remote commands are based on parameters that are passed to the current command.  
  
Generate parameters for the new command based on the intersection of parameters between the current command and the command to be executed.

M doc/xml/release.xml
A src/config/exec.c
A src/config/exec.h
M test/define.yaml
A test/src/module/config/execTest.c

`Add IoHandleRead and IoHandleWrite objects.`

commit   : ecd56105e688e74ad71f8a1fc0fc3e785989abe5    
  
author   : David Steele <[email protected]>    
date     : Thu, 17 Jan 2019 22:08:31 +0200    
  
committer: David Steele <[email protected]>    
date     : Thu, 17 Jan 2019 22:08:31 +0200

Click here for diff

General i/o objects for reading and writing file descriptors, in particular those that can block.  In other words, these are not generally to be used with file descriptors for actual files, but rather pipes, sockets, etc.

M doc/xml/release.xml
M src/Makefile
A src/common/io/handleRead.c
A src/common/io/handleRead.h
M src/common/io/handleWrite.c
M src/common/io/handleWrite.h
M test/define.yaml
M test/src/module/common/ioTest.c

`Add const VariantList * debug type.`

commit   : bf0c41d9d6d5fd53cc4bf190b0c577f506323b5b    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 22:23:25 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 22:23:25 +0200

Click here for diff

M doc/xml/release.xml
M src/common/type/variantList.h

`Fix difference in cipher type reporting missed in 8304d452.`

commit   : 7d4bbf290cf33de06fca9854f21ec9690cfd51bb    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 22:16:50 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 22:16:50 +0200

Click here for diff

The C code can't get the cipher type from the storage object because the C storage object does not have encryption baked in like the Perl code does.  
  
Instead, check backup.info to see if encryption is enabled.  This will need to rethought if another cipher type is added but for now it works fine.

M src/command/info/info.c
M test/src/module/command/infoTest.c

`Simplify info command text message when no stanza are present.`

commit   : e68d1e73042342c4cea0881e216e6979a3bcb140    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 19:23:10 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 19:23:10 +0200

Click here for diff

Replace the repository path with just "the repository".  The path is not important in this context and it is clearer to state where the stanzas are missing from.

M doc/xml/release.xml
M lib/pgBackRest/Info.pm
M src/command/info/info.c
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/src/module/command/infoTest.c

`Update Storage::Local->list() to accept an undefined path.`

commit   : ef9dc89e080d7b364a00fec138d1961c703dd02a    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 18:49:12 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 18:49:12 +0200

Click here for diff

The Perl code has a tendency to generate absolute paths even when they are not needed. This change helps the C and Perl storage work together via the protocol layer.

M doc/xml/release.xml
M lib/pgBackRest/Storage/Local.pm

`Update Perl repo rules to work when stanza is not specified.`

commit   : b4146b6bff99c8d82445bfda96b237ef0cff3050    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 18:45:19 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 18:45:19 +0200

Click here for diff

The C storage object strives to use rules whenever possible instead of generating absolute paths.  This change helps the C and Perl storage work together via the protocol layer.

M doc/xml/release.xml
M lib/pgBackRest/Protocol/Storage/Helper.pm

`Fix escaping in JSON string renderer.`

commit   : 0014e159443354a46197e58f0631d7363d038cde    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 18:34:50 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 18:34:50 +0200

Click here for diff

M doc/xml/release.xml
M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

`Fix missing linefeed and param debug.`

commit   : 49bd8e0e3e6e2c7f5e840c613919e55ee0074414    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 12:04:35 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 12:04:35 +0200

Click here for diff

M src/storage/driver/s3/fileRead.c

`Fix null output in JSON renderer.`

commit   : c0a904853664396d3ee9ffd5599634f489d924a1    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 09:11:01 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 09:11:01 +0200

Click here for diff

In some cases nulls were being rendered as "(null)" because no special logic existed to handle them.

M doc/xml/release.xml
M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

`Allow string Variant objects to contain null.`

commit   : ec26a0594e2f1da3f2f0041ceef606f70eaeef5e    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 09:05:13 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Jan 2019 09:05:13 +0200

Click here for diff

This is more useful than the variant itself being null as it saves conditionals when creating.

M doc/xml/release.xml
M src/common/type/variant.c
M test/src/module/common/typeVariantTest.c

`Add kvMove() and varLstMove().`

commit   : 1b334da87ffcddc5393fa0f900ec8c0395155aa7    
  
author   : David Steele <[email protected]>    
date     : Mon, 14 Jan 2019 22:02:23 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 14 Jan 2019 22:02:23 +0200

Click here for diff

Allow these objects to be moved to another mem context.

M doc/xml/release.xml
M src/common/type/keyValue.c
M src/common/type/keyValue.h
M src/common/type/variantList.c
M src/common/type/variantList.h
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeVariantTest.c

`Return UnknownError from errorTypeFromCode() for invalid error codes.`

commit   : aab9e38b9a070e8707c0a316784b456ecaeba64e    
  
author   : David Steele <[email protected]>    
date     : Mon, 14 Jan 2019 21:34:22 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 14 Jan 2019 21:34:22 +0200

Click here for diff

The prior behavior was to throw an exception but this was not very helpful when something unexpected happened.  Better to at least emit the error message even if the error code is not very helpful.

M doc/xml/release.xml
M src/common/error.c
M test/src/module/common/errorTest.c

`Allow empty arrays in JSON parser.`

commit   : 2b02d37602e3bfc451d598032d0e3793a06e8673    
  
author   : David Steele <[email protected]>    
date     : Mon, 14 Jan 2019 21:21:37 +0200    
  
committer: David Steele <[email protected]>    
date     : Mon, 14 Jan 2019 21:21:37 +0200

Click here for diff

M doc/xml/release.xml
M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

`Make the C version of the info command conform to the Perl version.`

commit   : 8304d452b3683d5d1d59cf7a6cc14801ddc35efd    
  
author   : David Steele <[email protected]>    
date     : Sun, 13 Jan 2019 22:44:58 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 13 Jan 2019 22:44:58 +0200

Click here for diff

There were some small differences in ordering and how the C version handled missing directories.  It may be that the C version is more consistent, but for now it is more important to be compatible with the Perl version.  
  
These differences were missed because the C info command was not wired into main.c so it was not being tested in regression.  This commit does not fix the wiring issue because there will likely be a release soon and it is too big a change to put in at the last moment.

M doc/xml/release.xml
M src/Makefile
M src/command/info/info.c
M src/main.c
M src/storage/helper.c
M src/storage/helper.h
M test/src/module/command/infoTest.c

`Improve accuracy of strSizeFormat().`

commit   : f314a1f8aa101b5f73cbb0902af7ebed9804a143    
  
author   : David Steele <[email protected]>    
date     : Sun, 13 Jan 2019 22:22:15 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 13 Jan 2019 22:22:15 +0200

Click here for diff

Casting to int caused large values to be slightly inaccurate so cast to uint64_t instead.  
  
Also, use multiplication where possible since the compiler should precompute multiplied values.

M doc/xml/release.xml
M src/common/type/string.c
M test/src/module/common/typeStringTest.c

`Null-terminate list returned by strLstPtr().`

commit   : cebcb3f027d2a95f5b350df113efef78d449c72f    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Jan 2019 09:49:28 +0200    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Jan 2019 09:49:28 +0200

Click here for diff

The null-terminator is required by many C functions that accept a string list so just add it in all cases rather than making it optional.

M doc/xml/release.xml
M src/common/type/stringList.c
M test/src/module/common/typeStringTest.c

`Fix typos in monitoring examples.`

commit   : f8a9fc203909696293e57d84bc8f6d83d737097b    
  
author   : Magnus Hagander <[email protected]>    
date     : Wed, 9 Jan 2019 19:20:38 +0200    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Jan 2019 19:20:38 +0200

Click here for diff

M doc/example/pgsql-pgbackrest-info.sql

`Ignore SIGPIPE signals and check EPIPE result instead.`

commit   : ade2c3102d7459d274f0988d7d3cc2de49c7c9b4    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 17:42:44 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 17:42:44 +0200

Click here for diff

SIGPIPE immediately terminates the process but we would rather catch the EPIPE error and gracefully shutdown.  
  
Ignore SIGPIPE and throw the EPIPE error via normal error handling.

M doc/xml/release.xml
M src/common/exit.c
M test/src/test.c

`Add _DARWIN_C_SOURCE flag to Makefile for MacOS builds.`

commit   : 7272d6e2475f7fe8dd327206591b83cf4b9f498b    
  
author   : Douglas J Hunley <[email protected]>    
date     : Sun, 6 Jan 2019 17:28:17 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 17:28:17 +0200

Click here for diff

For some reason adding -D_POSIX_C_SOURCE=200112L caused MacOS builds to stop working.  Combining both flags seems to work fine for all tested systems.  
  
Contributed by Douglas J Hunley.

M doc/xml/release.xml
M src/Makefile

`Move C module include in test.c above headers included for testing.`

commit   : 9560baf659cafec2614e90ce72d988635cb945c1    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 15:52:59 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 15:52:59 +0200

Click here for diff

Including the C module after the headers required for testing meant that if headers were missing from the C module they were not caught while directly testing the C module.  
  
The missing headers were caught in general testing, but it is frustrating to get an error in a module that has already passed while testing another module or running CI.  
  
Move the C module include to the very top so missing headers cause immediate failures.

M doc/xml/release.xml
M test/src/test.c

`Add debug macros and extra space missed in 256b727a.`

commit   : 0e0000f59c571a3889927288a536c8e220f97fac    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 15:27:30 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 15:27:30 +0200

Click here for diff

M src/storage/driver/s3/storage.c
M src/storage/driver/s3/storage.h

`Rename common/io/handle module to common/io/handleWrite.`

commit   : 1de22cac2b0d2f9bddc794693dd2b97e61efdf85    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 14:37:39 +0200    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Jan 2019 14:37:39 +0200

Click here for diff

ioHandleWriteOneStr() will become a helper function for the IoHandleWrite object.

M doc/xml/release.xml
M libc/Makefile.PL
M src/Makefile
M src/command/help/help.c
M src/command/info/info.c
R095 src/common/io/handle.c src/common/io/handleWrite.c
R076 src/common/io/handle.h src/common/io/handleWrite.h
M src/common/lock.c
M test/code-count/file-type.yaml
M test/define.yaml

`Fix object struct comment to be conforming.`

commit   : f52e0187e61b6f74c66573580523018509541f98    
  
author   : David Steele <[email protected]>    
date     : Sat, 5 Jan 2019 18:10:30 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 5 Jan 2019 18:10:30 +0200

Click here for diff

M src/common/io/bufferWrite.c

`Fix typo in error message.`

commit   : 842147321ff39a523608e281ade7ccb5fcad7bba    
  
author   : David Steele <[email protected]>    
date     : Sat, 5 Jan 2019 14:43:40 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 5 Jan 2019 14:43:40 +0200

Click here for diff

M src/common/io/handle.c
M test/src/module/common/ioTest.c

`Command-line help update missed in 200fd3aa.`

commit   : 59d7352f31a646c2f2491aacb922116caf125c50    
  
author   : David Steele <[email protected]>    
date     : Sat, 5 Jan 2019 14:38:31 +0200    
  
committer: David Steele <[email protected]>    
date     : Sat, 5 Jan 2019 14:38:31 +0200

Click here for diff

M src/config/define.auto.c

`Clarify that encryption is always performed client-side.`

commit   : 200fd3aa05174e109488f533d57db503d18ce25a    
  
author   : David Steele <[email protected]>    
date     : Fri, 4 Jan 2019 12:25:58 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 4 Jan 2019 12:25:58 +0200

Click here for diff

Suggested by Bruce Burdick.

M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml

`Begin v2.09 development.`

commit   : 50717aa846a41852179feaa69b35a924bffc66c6    
  
author   : David Steele <[email protected]>    
date     : Fri, 4 Jan 2019 11:00:59 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 4 Jan 2019 11:00:59 +0200

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h