pgBackRest 2.18 commit log

v2.17: C Migrations and Bug Fixes

commit   : ce2bf2999892f689001b084af4d5feb63dc9b402    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 Sep 2019 16:39:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 Sep 2019 16:39:32 -0400    

Click here for diff

Bug Fixes:  
  
* Improve slow manifest build for very large quantities of tables/segments. (Reported by Jens Wilke.)  
* Fix exclusions for special files. (Reported by CluelessTechnologist, Janis Puris, Rachid Broum.)  
  
Improvements:  
  
* The stanza-create/update/delete commands are implemented entirely in C. (Contributed by Cynthia Shang.)  
* The start/stop commands are implemented entirely in C. (Contributed by Cynthia Shang.)  
* Create log directories/files with 0750/0640 mode. (Suggested by Damiano Albani.)  
  
Documentation Bug Fixes:  
  
* Fix yum.p.o package being installed when custom package specified. (Reported by Joe Ayers, John Harvey.)  
  
Documentation Improvements:  
  
* Build pgBackRest as an unprivileged user. (Suggested by Laurenz Albe.)  

M README.md
M doc/RELEASE.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Fix yum.p.o package being installed when custom package specified.

commit   : 0b5720c642c040e16373f72a9058df094d05ca10    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 Sep 2019 14:34:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 Sep 2019 14:34:49 -0400    

Click here for diff

The {[os-type-is-centos]} expression was missing parens which meant "and" expressions built on it would always evaluate true if the os-type was centos6.  
  
Reported by Joe Ayers, John Harvey.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Don't decode manifest data when it is generated on a remote.

commit   : 7d8068f27b6b695f9616fd985517692236dda47d    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 Sep 2019 12:30:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 Sep 2019 12:30:45 -0400    

Click here for diff

Decoding a manifest from the JSON provided by C to the hash required by Perl is an expensive process.  If manifest() was called on a remote it was being decoded into a hash and then immediately re-encoded into JSON for transmission over the protocol layer.  
  
Instead, provide a function for the remote to get the raw JSON which can be transmitted as is and decoded in the calling process instead.  
  
This makes remote manifest calls as fast as they were before 2.16, but local calls must still pay the decoding penalty and are therefore slower.  This will continue to be true until the Perl storage interface is retired at the end of the C migration.  
  
Note that for reasonable numbers of tables there is no detectable difference.  The case in question involved 250K tables with a 10 minute decode time (which was being doubled) on a fast workstation.  

M lib/pgBackRest/Protocol/Remote/Minion.pm
M lib/pgBackRest/Protocol/Storage/Remote.pm
M lib/pgBackRest/Storage/Storage.pm
M src/perl/embed.auto.c

Add more commonly-used buffer constants.

commit   : 1e55b876206c73d0ce32424814d100ef82d8da05    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 Sep 2019 07:49:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 Sep 2019 07:49:04 -0400    

Click here for diff

M src/common/type/buffer.c
M src/common/type/buffer.h

Move constants from the infoManifest module to the infoBackup module.

commit   : 7ade3fc1c31f09425547d1869e82863582b1b137    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Sep 2019 21:09:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Sep 2019 21:09:43 -0400    

Click here for diff

These constants should be kept separate because the implementation of any info file might change in the future and only the interface should be expected to remain consistent.  
  
In any case, infoBackup requires Variant constants while infoManifest uses String constants so they are not shareable.  Modern compilers should combine the underlying const char * constants.  

M src/Makefile.in
M src/info/infoBackup.c
D src/info/infoManifest.c
M src/info/infoManifest.h

Disable S3 and encryption on u18 integration tests for mock/all/1.

commit   : 3a28b68b8bde996252ad8941107744c74a062453    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Sep 2019 19:06:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Sep 2019 19:06:12 -0400    

Click here for diff

This test is commonly used for sanity checking but the combination of S3 and encryption makes it hard to use and encourages temporary changes to make it usable.  
  
Acknowledge this and disable S3 and encryption for this test and move them to mock/all/2.  

M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Add OBJECT_DEFINE_MOVE() macro.

commit   : 05abab5e72e2e4d5d394b79533621dcf06d1f324    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Aug 2019 14:36:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Aug 2019 14:36:02 -0400    

Click here for diff

The object *Move() functions are very repetitive so replace them with a macro that works similarly to OBJECT_DEFINE_FREE().  

M src/common/ini.c
M src/common/io/filter/filter.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/object.h
M src/common/type/buffer.c
M src/common/type/keyValue.c
M src/common/type/list.c
M src/db/db.c
M src/postgres/client.c
M src/protocol/client.c
M src/protocol/command.c
M src/protocol/parallelJob.c
M src/protocol/server.c
M src/storage/read.c
M src/storage/write.c
M test/src/module/common/objectTest.c

Add ioReadLineParam() to allow return on eof.

commit   : d1675b7e914a3e25a3d6cff2c73f36f69ec9f729    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Aug 2019 10:46:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Aug 2019 10:46:54 -0400    

Click here for diff

ioReadLine() errors on eof because it has previously been used only for protocol reads.  
  
Returning on eof is handy for reading lines from files where eof is not considered an error.  

M src/common/io/read.c
M src/common/io/read.h
M test/src/module/common/ioTest.c

Add required header.

commit   : a605117a2329965d65f985b4b8e991a5267416c2    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Aug 2019 10:36:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Aug 2019 10:36:21 -0400    

Click here for diff

Apparently this header is already included everywhere convert.h is included.  

M src/common/type/convert.h

Fix typos in variable names.

commit   : 8074ca6a2613f6b9310654b5e0f5c468ae7821e8    
  
author   : Josh Soref <[email protected]>    
date     : Mon, 26 Aug 2019 12:30:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 12:30:22 -0400    

Click here for diff

Contributed by Josh Soref.  

M src/command/expire/expire.c
M src/command/info/info.c

Fix typos in the release notes.

commit   : 4a88791a0af78ba6f5496ec4e4351e9726662546    
  
author   : Josh Soref <[email protected]>    
date     : Mon, 26 Aug 2019 12:29:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 12:29:43 -0400    

Click here for diff

Contributed by Josh Soref.  

M doc/resource/git-history.cache
M doc/xml/release.xml

Fix typos in the documentation.

commit   : 545ccfa8786b631476c232a1d78791687e638432    
  
author   : Josh Soref <[email protected]>    
date     : Mon, 26 Aug 2019 12:26:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 12:26:00 -0400    

Click here for diff

Contributed by Josh Soref.  

M doc/doc.pl
M doc/xml/faq.xml
M doc/xml/index.xml
M doc/xml/test.xml
M doc/xml/user-guide.xml
M test/README.md

Fix typo in stack trace message.

commit   : 08bac74d38b126515e2620cdf129cd4cbd53f6e8    
  
author   : Josh Soref <[email protected]>    
date     : Mon, 26 Aug 2019 12:20:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 12:20:51 -0400    

Click here for diff

Contributed by Josh Soref.  

M src/common/stackTrace.c
M test/src/module/common/stackTraceTest.c

Fix comment typos.

commit   : c2771e5469ecd94899243e86668a4bcc0d2c65f5    
  
author   : Josh Soref <[email protected]>    
date     : Mon, 26 Aug 2019 12:05:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 12:05:36 -0400    

Click here for diff

This includes some variable names in tests which don't seem important enough for their own commits.  
  
Contributed by Josh Soref.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/release.pl
M doc/xml/dtd/doc.dtd
M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Backup/Common.pm
M lib/pgBackRest/Backup/Info.pm
M lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Common/Io/Process.pm
M lib/pgBackRest/Common/Log.pm
M lib/pgBackRest/Config/Config.pm
M lib/pgBackRest/Db.pm
M lib/pgBackRest/Main.pm
M lib/pgBackRest/Manifest.pm
M lib/pgBackRest/Protocol/Local/Process.pm
M lib/pgBackRest/Restore.pm
M lib/pgBackRest/Storage/Storage.pm
M libc/LibC.h
M src/command/archive/get/get.c
M src/command/backup/common.h
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/stanza/delete.c
M src/common/assert.h
M src/common/error.c
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/tls/client.c
M src/common/macro.h
M src/common/type/buffer.c
M src/common/type/string.c
M src/common/type/xml.h
M src/config/parse.c
M src/info/info.c
M src/info/infoPg.c
M src/storage/helper.c
M src/storage/storage.c
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Backup/BackupFileUnitPerlTest.pm
M test/lib/pgBackRestTest/Module/Info/InfoInfoArchivePerlTest.pm
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessFork.h
M test/src/common/harnessPq.h
M test/src/common/harnessTest.h
M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/stackTraceTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/config/parseTest.c
M test/src/module/storage/posixTest.c
M test/test.pl

Add storage tests for files beginning with dots.

commit   : 22aa532be1be83c5b1df8db5fa85047e1255a789    
  
author   : David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 11:37:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 11:37:21 -0400    

Click here for diff

Prevent a regression of the issue fixed in f88012ce by adding some tests.  

M test/lib/pgBackRestTest/Module/Storage/StoragePerlTest.pm
M test/src/module/storage/posixTest.c

Fix typo in httpClientReponseHeader() function name.

commit   : 8c9df8c290f760b6a1f8a6a0acb6b4d9b8fa4056    
  
author   : Josh Soref <[email protected]>    
date     : Mon, 26 Aug 2019 11:35:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 11:35:35 -0400    

Click here for diff

Contributed by Josh Soref.  

M src/common/io/http/client.c
M src/common/io/http/client.h
M src/storage/s3/storage.c
M test/src/module/common/ioHttpTest.c

Fix typo in CFGDEFDATA_OPTION_OPTIONAL_COMMAND_OVERRIDE macro name.

commit   : d1cc22f886a865d5b7e0c3e699e4fbe89c11aade    
  
author   : Josh Soref <[email protected]>    
date     : Mon, 26 Aug 2019 11:25:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Aug 2019 11:25:14 -0400    

Click here for diff

Contributed by Josh Soref.  

M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M src/config/define.auto.c
M src/config/define.c

Fix exclusions for special files.

commit   : 01c2669b9764491ec3ca96ffd0de6676bef9e5dc    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Aug 2019 07:47:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Aug 2019 07:47:54 -0400    

Click here for diff

Prior to 2.16 the Perl manifest code would skip any file that began with a dot.  This was not intentional but it allowed PostgreSQL socket files to be located in the data directory.  The new C code in 2.16 did not have this unintentional exclusion so socket files in the data directory caused errors.  
  
Worse, the file type error was being thrown before the exclusion check so there was really no way around the issue except to move the socket files out of the data directory.  
  
Special file types (e.g. socket, pipe) will now be automatically skipped and a warning logged to notify the user of the exclusion.  The warning can be suppressed with an explicit --exclude.  
  
Reported by CluelessTechnologist, Janis Puris, Rachid Broum.  

M doc/xml/release.xml
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Add special file type to storageInfo().

commit   : 2862f480cd1712ee9c188c28d79f2f9ed1bb9c6b    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Aug 2019 07:24:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Aug 2019 07:24:25 -0400    

Click here for diff

There's not much we can do with special files, but higher level logic can at least exclude them gracefully rather than throwing a hard error.  

M libc/xs/storage/storage.xsh
M src/storage/info.h
M src/storage/posix/storage.c
M test/lib/pgBackRestTest/Module/Storage/StoragePerlTest.pm
M test/src/module/storage/posixTest.c

Remove unneeded strFree() in storagePosixInfoListEntry().

commit   : 04e4fde5733bbe885ef94098f0114a3a160a88e5    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Aug 2019 10:28:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 22 Aug 2019 10:28:08 -0400    

Click here for diff

This strFree() was the primary culprit in the performance issue fixed in 9eaeb33c.  
  
Since the parent mem context is now freed regularly, this strFree() performs better, but still adds time so removing it seems best.  

M src/storage/posix/storage.c

Fix regexp to ignore ./.. directories in the Posix driver.

commit   : f88012cef3acc29fb5311c73ea1a84c1409d88ee    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Aug 2019 10:18:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 22 Aug 2019 10:18:34 -0400    

Click here for diff

In versions <= 2.15 the old regexp caused any file or directory beginning with . to be ignored during a backup.  This has caused behavioral differences in 2.16 because the new C code correctly excludes ./.. directories.  
  
This Perl code is only used for testing now, but it should still match the output of the C functions.  

M test/lib/pgBackRestTest/Common/StoragePosix.pm

Move info file checksum to the end of the file.

commit   : c002a2ce2fb7a1bc4bbeaa5dee2f3e7d719ccc24    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 19:45:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 19:45:48 -0400    

Click here for diff

Putting the checksum at the beginning of the file made it impossible to stream the file out when saving.  The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning.  
  
Instead place the checksum at the end.  This does not break the existing Perl or C code since the read is not order dependent.  
  
There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient.  
  
Reviewed by Cynthia Shang.  

M doc/xml/release.xml
M lib/pgBackRest/Common/Ini.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/lib/pgBackRestTest/Module/Common/CommonIniPerlTest.pm

The stanza-create/update/delete commands are implemented entirely in C.

commit   : c733319063bbc9d67ba99b96c178f1d9f863b8fe    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 21 Aug 2019 16:26:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 16:26:28 -0400    

Click here for diff

Contributed by Cynthia Shang.  

M doc/xml/release.xml
M lib/pgBackRest/Common/Lock.pm
M lib/pgBackRest/Main.pm
D lib/pgBackRest/Stanza.pm
M src/Makefile.in
A src/command/stanza/common.c
A src/command/stanza/common.h
A src/command/stanza/create.c
A src/command/stanza/create.h
A src/command/stanza/delete.c
A src/command/stanza/delete.h
A src/command/stanza/upgrade.c
A src/command/stanza/upgrade.h
M src/main.c
M src/perl/embed.auto.c
M src/postgres/interface.h
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/expect/real-all-004.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
D test/lib/pgBackRestTest/Module/Stanza/StanzaAllPerlTest.pm
A test/src/module/command/stanzaTest.c

Add checkDbConfig() to compare pgBackRest/PostgreSQL configs.

commit   : 53f27da3a6d770608de6aa6059c40526b83b31dd    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 21 Aug 2019 15:41:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 15:41:52 -0400    

Click here for diff

Checking the PostgreSQL-reported path and version against the pgBackRest configuration helps ensure that pgBackRest is operating against the correct cluster.  
  
In Perl this functionality was in the Db object, but check seems like a better place for it in C.  
  
Contributed by Cynthia Shang.  

M src/Makefile.in
A src/command/check/common.c
A src/command/check/common.h
M test/define.yaml
M test/src/module/command/checkTest.c

Add storagePg*() variants to get storage for a specified host id.

commit   : 8b93fdf3493d96fcbbc127604b502d246928ec8e    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 15:26:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 15:26:52 -0400    

Click here for diff

Previously the host id to use was pulled from the host-id option or defaulted to 1.  
  
The stanza, check, and backup commands will all need the ability to address a specified pg host, so add functions to make that possible.  

M src/storage/helper.c
M src/storage/helper.h
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Allow Info* objects to be created from scratch in C.

commit   : fa640f22add9af88c2ddd584812d8fde93c8f8c5    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 21 Aug 2019 15:12:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 15:12:00 -0400    

Click here for diff

Previously, info files (e.g. archive.info, backup.info) were created in Perl and only loaded in C.  
  
The upcoming stanza commands in C need to create these files so refactor the Info* objects to allow new, empty objects to be created.  Also, add functions needed to initialize each Info* object to a valid state.  
  
Contributed by Cynthia Shang.  

M src/Makefile.in
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoPg.c
M src/info/infoPg.h
M test/define.yaml
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c

Fix comment indents.

commit   : aa6f7eb862f71bf0b8f64d3c5e3d67903b048214    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 14:14:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 14:14:30 -0400    

Click here for diff

M src/protocol/helper.c

Fix comment typo.

commit   : 71d96531f8b581d460ff24a229b676607fc8992b    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 21 Aug 2019 12:10:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 12:10:09 -0400    

Click here for diff

Contributed by Cynthia Shang.  

M test/src/module/command/expireTest.c

Add Db object getters missed in e4901d50.

commit   : c4533765fb5cd4b72f9fcb27f53db2a30165b9b2    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 21 Aug 2019 11:59:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 11:59:51 -0400    

Click here for diff

dbOpen() loaded the version and data directory but there was no way to get the data externally.  
  
Contributed by Cynthia Shang.  

M src/db/db.c
M src/db/db.h
M test/src/module/db/dbTest.c

Add repoIsLocalVerify() to verify repository locality.

commit   : 27e823581201374df831abc4cf299ce4da29b910    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 21 Aug 2019 11:41:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 11:41:36 -0400    

Click here for diff

Some commands can only be run on a host where the repository is local.  This function centralizes the check and error.  
  
Contributed by Cynthia Shang.  

M src/protocol/helper.c
M src/protocol/helper.h
M test/src/module/protocol/protocolTest.c

Require storage when calling pgControlFromFile().

commit   : 6a09d9294d0ef13158fbc96c9d38a28ac7a6f150    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 21 Aug 2019 11:29:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Aug 2019 11:29:30 -0400    

Click here for diff

Previously storageLocal() was being used internally but loading pg_control from remote storage is often required.  
  
Also, storagePg() is more appropriate than storageLocal() for all current usage.  
  
Contributed by Cynthia Shang.  

M src/Makefile.in
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/main.c
M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

Build pgBackRest as an unprivileged user.

commit   : 286a106ae4932ae8213d62528de934afa99c246e    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Aug 2019 09:46:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Aug 2019 09:46:29 -0400    

Click here for diff

pgBackRest was being built by root in the documentation which is definitely not best practice.  
  
Instead build as the unprivileged default container user.  Sudo privileges are still required to install.  
  
Suggested by Laurenz Albe.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Remove Perl info command invocation missed in 205525b6.

commit   : 6b5366a663f8c408e56bb2f8e5b27821898a91cc    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Aug 2019 21:57:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Aug 2019 21:57:14 -0400    

Click here for diff

M lib/pgBackRest/Main.pm
M src/perl/embed.auto.c

Properly reset conflicting pg-* options for the remote protocol.

commit   : f6aef6e466ccba368fb53cc00450b501f35782fc    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Aug 2019 21:45:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Aug 2019 21:45:54 -0400    

Click here for diff

The pg1-socket-path and pg1-port options were not being reset when options from a higher index were being pushed down for processing by a remote.  Since remotes only talk to one cluster they always use the options in index 1.  This requires moving options from the original index to 1 before starting the remote.  All options already set on index 1 must be removed if they are not being overwritten.  

M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Fix slow manifest build for very large quantities of tables/segments.

commit   : 9eaeb33c882b79a1cade3ce65b5d1b51b6978b05    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Aug 2019 21:36:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Aug 2019 21:36:01 -0400    

Click here for diff

storagePosixInfoList() processed each directory in a single memory context.  If the directory contained hundreds of thousands of files processing became very slow due to the number of allocations.  
  
Instead, reset the memory context every thousand files to minimize the number of allocations active at once, improving both speed and memory consumption.  
  
Reported by Jens Wilke.  

M doc/xml/release.xml
M src/storage/posix/storage.c

Add reset to temp memory contexts to save memory and processing time.

commit   : d411321d28d2c94f3376b82984981624fe30e287    
  
author   : David Steele <[email protected]>    
date     : Mon, 19 Aug 2019 21:16:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Aug 2019 21:16:10 -0400    

Click here for diff

Processing large datasets in a memory context can lead to high memory usage and long allocation times.  Add a new MEM_CONTEXT_TEMP_RESET_BEGIN() macro that allows temp allocations to be automatically freed after N iterations.  

M src/common/memContext.h
M test/src/module/common/memContextTest.c

Add MostCommonValue object.

commit   : 7d97d49f41e961dcaa0d54b56538d3c5a0b8f6ce    
  
author   : David Steele <[email protected]>    
date     : Sun, 18 Aug 2019 20:46:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 18 Aug 2019 20:46:34 -0400    

Click here for diff

Calculate the most common value in a list of variants.  If there is a tie then the first value passed to mcvUpdate() wins.  
  
mcvResult() can be called multiple times because it does not end processing, but there is a cost to calculating the result each time  
since it is not stored.  

M src/Makefile.in
A src/common/type/mcv.c
A src/common/type/mcv.h
M test/define.yaml
A test/src/module/common/typeMcvTest.c

Add backup type conversion functions.

commit   : 8aa1e552b00cb3a9cf5665812e6de782f0805675    
  
author   : David Steele <[email protected]>    
date     : Sun, 18 Aug 2019 20:09:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 18 Aug 2019 20:09:44 -0400    

Click here for diff

Convert back and forth between the string and enum representations of backup types.  

M src/command/backup/common.c
M src/command/backup/common.h
M test/define.yaml
M test/src/module/command/backupCommonTest.c

Fix test writing "null" into manifest files.

commit   : 3df075bf4022e84f0238591b18bc5e866d6a2aa8    
  
author   : David Steele <[email protected]>    
date     : Sun, 18 Aug 2019 15:29:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 18 Aug 2019 15:29:18 -0400    

Click here for diff

"null" is not allowed in the manifest format (null values should be missing instead) but Perl was treating the invalid values written by this test as if they were missing.  
  
Update the test code to remove the values rather than setting them to "null".  

M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Remove RHEL package patch merged to upstream.

commit   : 8edc57b5819a41d43524452de378ec409d05f4d9    
  
author   : David Steele <[email protected]>    
date     : Sun, 18 Aug 2019 15:19:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 18 Aug 2019 15:19:10 -0400    

Click here for diff

D test/patch/rhel-package.patch
M test/test.pl

Add cvtZToMode() replacement missed in 969e5817.

commit   : 0da3352d59c3d985d6fad7b08d28446900932857    
  
author   : David Steele <[email protected]>    
date     : Sat, 17 Aug 2019 17:52:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 17 Aug 2019 17:52:00 -0400    

Click here for diff

M libc/xs/storage/storage.xs
M src/perl/libc.auto.c

Add bool rendering to jsonFromVar().

commit   : d0dee2e26027f866c282050457d61d6907f3263f    
  
author   : David Steele <[email protected]>    
date     : Sat, 17 Aug 2019 17:47:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 17 Aug 2019 17:47:13 -0400    

Click here for diff

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Fix expire not immediately writing into separate file after backup.

commit   : 8fc1d3883b2d96d4f518ff2bfa810b082cc48b38    
  
author   : David Steele <[email protected]>    
date     : Sat, 17 Aug 2019 17:43:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 17 Aug 2019 17:43:56 -0400    

Click here for diff

Logging stayed in the backup log until the Perl code started.  Fix this so it logs to the correct file and will still work after the Perl code is removed.  

M src/config/load.c
M src/config/load.h
M src/main.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/config/loadTest.c

Create log directories/files with 0750/0640 mode.

commit   : 41b6795a374391d66dcfb2df0521f1d7acf84b3e    
  
author   : David Steele <[email protected]>    
date     : Sat, 17 Aug 2019 14:15:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 17 Aug 2019 14:15:37 -0400    

Click here for diff

The log directories/files were being created with a mix of modes depending on whether they were created in C or Perl.  In particular, the C code was creating log files with the execute bit set for the user and group which was just odd.  
  
Standardize on 750/640 for both code paths.  
  
Suggested by Damiano Albani.  

M doc/xml/release.xml
M lib/pgBackRest/Common/Log.pm
M src/common/log.c
M src/perl/embed.auto.c

Change comment for searchability.

commit   : bc5385142c75f5fa390531a8ff1bd660dd1380ef    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Aug 2019 15:18:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Aug 2019 15:18:42 -0400    

Click here for diff

M test/Vagrantfile

The start/stop commands are implemented entirely in C.

commit   : 382ed9282504b2adced37275937d2414632bb968    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 9 Aug 2019 15:17:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Aug 2019 15:17:18 -0400    

Click here for diff

The Perl versions remain because they are still being used by the Perl stanza commands.  Once the stanza commands are migrated they can be removed.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M lib/pgBackRest/Main.pm
M src/Makefile.in
M src/command/control/common.c
M src/command/control/common.h
A src/command/control/start.c
A src/command/control/start.h
A src/command/control/stop.c
A src/command/control/stop.h
M src/main.c
M src/perl/embed.auto.c
M test/define.yaml
M test/src/module/command/controlTest.c

Update RHEL patch for upstream changes.

commit   : fe196cb0dffacb5d2d1488e3b592cc8ba3f39a5e    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Aug 2019 14:52:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Aug 2019 14:52:26 -0400    

Click here for diff

This isn't correct, but we can't test anything else if the package doesn't build.  

M test/patch/rhel-package.patch
M test/test.pl

Constify some globals.

commit   : 0e7134d114af1767addce36a3857ea3fd38abc5f    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 18:50:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 18:50:54 -0400    

Click here for diff

M src/common/lock.c
M src/common/log.c
M src/common/type/buffer.h
M src/common/type/string.h
M src/common/type/variant.c
M src/common/type/variant.h

Add hash constants for zero-length data.

commit   : e9517dcec05c75a03fa73029250de4f5b21fdfd3    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 18:47:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 18:47:02 -0400    

Click here for diff

No need to calculate a hash when the data length is known to be zero.  Use one of these constants instead.  

M src/common/crypto/hash.c
M src/common/crypto/hash.h
M src/storage/s3/storage.c
M test/src/module/common/cryptoTest.c
M test/src/module/storage/s3Test.c

Fix typo in parameter logging.

commit   : 56c24b7669f1e636a8221106547469065eb7d161    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 16:20:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 16:20:51 -0400    

Click here for diff

M src/info/infoBackup.c

Remove Debian package patch merged to upstream.

commit   : 1076b142e38a4960af1157bd117fa2aacee7d8e6    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 16:15:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 16:15:42 -0400    

Click here for diff

D test/patch/debian-package.patch

Add cvtZToMode().

commit   : 969e5817c5d29890cbe12fa95422cd8703abf8d9    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 11:59:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 11:59:39 -0400    

Click here for diff

The companion function to cvtModeToZ().  

M src/common/type/convert.c
M src/common/type/convert.h
M test/src/module/common/typeConvertTest.c

Add kvGetDefault() to KeyValue object.

commit   : 1f93e147fa8563028db8b58621db4a1a12387c87    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 11:41:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 11:41:49 -0400    

Click here for diff

Return the default only if the key is not found, not when the value is NULL.  

M src/common/type/keyValue.c
M src/common/type/keyValue.h
M test/src/module/common/typeKeyValueTest.c

Allow NULLs in strEq().

commit   : 289b47902ba1f49a1baea9fdce53eec52d5358bb    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 10:50:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 10:50:25 -0400    

Click here for diff

Bring this function more in line with the way varEq() works.  NULL == NULL but NULL != NOT NULL.  

M src/common/type/string.c
M test/src/module/common/typeStringTest.c

Add constant for lock file extension.

commit   : feec674b6ff88bdfa62b20dd706d724c79ca1bcd    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 8 Aug 2019 06:48:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Aug 2019 06:48:25 -0400    

Click here for diff

Contributed by Cynthia Shang.  

M src/common/lock.c
M src/common/lock.h
M test/src/module/command/remoteTest.c
M test/src/module/common/lockTest.c

Begin v2.17 development.

commit   : efc62c90573bd7a0595d81bcf8a261f732fa1fa9    
  
author   : David Steele <[email protected]>    
date     : Mon, 5 Aug 2019 12:32:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 5 Aug 2019 12:32:06 -0400    

Click here for diff

M doc/RELEASE.md
M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h