pgBackRest 2.19 (upcoming) commit log

Sort and find improvements to List and StringList objects.

commit   : 506c10f7f270806aca8acecaa303bf4458463720    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 12 Sep 2019 12:04:25 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 12 Sep 2019 12:04:25 -0400    

Click here for diff

Push the responsibility for sort and find down to the List object by introducing a general comparator function that can be used for both sorting and finding.  
  
Update insert and add functions to return the item added rather than the list.  This is more useful in the core code, though numerous updates to the tests were required.  

M src/Makefile.in
M src/command/archive/common.c
M src/command/expire/expire.c
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/common/type/variantList.c
M src/config/parse.c
M src/storage/s3/storage.c
M test/src/common/harnessTest.h
M test/src/module/command/expireTest.c
M test/src/module/common/execTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/config/configTest.c
M test/src/module/config/parseTest.c

Add missing header in common/user module.

commit   : e4a071ce033ae53dafb35eb9ccd4daba1030064c    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 12 Sep 2019 10:56:45 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 12 Sep 2019 10:56:45 -0400    

Click here for diff

M src/common/user.h

Update incorrect pipelining references to connection reuse.

commit   : b53305cc361e9978807d506e3e8ec13b3ae93511    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 12 Sep 2019 08:17:07 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 12 Sep 2019 08:17:07 -0400    

Click here for diff

Connection reuse and pipelining are not the same thing and should not have been conflated.  
  
Update comments and release notes to reflect the correct usage.  

M doc/resource/git-history.cache
M src/common/io/http/client.h
M src/common/io/tls/client.h

Move documentation job first for Travis CI.

commit   : dca5b63f970be8a13cd11222f4b5a9d4ca71261e    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Sep 2019 13:06:44 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Sep 2019 13:06:44 -0400    

Click here for diff

Since this job has been running long recently this should improved overall performance when multiple commits are queued up.  

M .travis.yml
M test/lib/pgBackRestTest/Common/CiTest.pm

Add groupIdFromName() and userIdFromName() to user module.

commit   : f4f21d0df7217bcb583189d0fc4ecaac3faed146    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Sep 2019 13:02:05 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Sep 2019 13:02:05 -0400    

Click here for diff

Update StorageWritePosix to use the new functions.  
  
A side effect is that storageWritePosixOpen() will no longer error when the user/group name does not exist.  It will simply retain the original user/group, i.e. the user that executed the restore.  
  
In general this is a feature since completing a restore is more important than setting permissions exactly from the source host.  However, some notification of this omission to the user would be beneficial.  

M src/Makefile.in
M src/common/user.c
M src/common/user.h
M src/storage/posix/write.c
M test/src/module/common/userTest.c
M test/src/module/storage/posixTest.c

Increase process timeout and emit occasional warnings.

commit   : f8d0574759340f0260f06e377da412041987f582    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 10 Sep 2019 12:29:36 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 10 Sep 2019 12:29:36 -0400    

Click here for diff

Travis will timeout after 10 minutes with no output.  Emit a warning every 5 minutes to keep Travis alive and increase the total timeout to 20 minutes.  
  
Documentation builds have been timing out a lot recently so hopefully this will help.  

M test/lib/pgBackRestTest/Common/ExecuteTest.pm

Update comment missed in d957acb3.

commit   : e043c6b1bc388422aee9255077544c3644b8f69d    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 9 Sep 2019 07:37:57 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 9 Sep 2019 07:37:57 -0400    

Click here for diff

M src/postgres/interface.h

Add user module for managing system users/groups.

commit   : 1049632873d5a70cda90e5c4ed87312bfe4810d6    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 8 Sep 2019 20:11:51 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 8 Sep 2019 20:11:51 -0400    

Click here for diff

Centralize the management of users and groups.  
  
Also update Posix storage driver where users/groups were already in use.  

M libc/Makefile.PL
M src/Makefile.in
A src/common/user.c
A src/common/user.h
M src/storage/posix/storage.c
M test/define.yaml
A test/src/module/common/userTest.c

Add function to generate PostgreSQL tablespace identifier.

commit   : d957acb36b6b50c6166d9a68b03f239cdc508fa7    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 8 Sep 2019 06:53:23 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 8 Sep 2019 06:53:23 -0400    

Click here for diff

In PostgreSQL >= 9.0 each tablespace data is stored in a specially named directory so different major versions can share the same tablespace path.  

M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

Add test macro for comparing String to zero-terminated strings.

commit   : 051128ed9ee0ebc63fb35e69818c6213397a053f    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 8 Sep 2019 06:47:52 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 8 Sep 2019 06:47:52 -0400    

Click here for diff

This macro avoids the need to wrap the first parameter in strPtr() and should improve readability.  

M test/src/common/harnessTest.h

Remove most references to PostgreSQL control and catalog versions.

commit   : 0a96764cb8daa615121069e840268e216d1983ff    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 7 Sep 2019 18:04:39 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 7 Sep 2019 18:04:39 -0400    

Click here for diff

The control and catalog versions were stored a variety of places in the optimistic hope that they would be useful.  In fact they never were.  
  
We can't remove them from the backup.info and backup.manifest files due to backwards compatibility concerns, but we can at least avoid loading and storing them in C structures.  
  
Add functions to the PostgreSQL interface which will return the control and catalog versions for any supported version of PostgreSQL to allow backwards compatibility for backup.info and backup.manifest.  These functions will be useful in other ways, e.g. generating the tablespace identifier in PostgreSQL >= 9.0.  

M src/command/stanza/create.c
M src/command/stanza/upgrade.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoPg.c
M src/info/infoPg.h
M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/interface/version.h
M src/postgres/interface/version.intern.h
M test/define.yaml
M test/src/module/command/stanzaTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/postgres/interfaceTest.c

Add user/group id to storageInfo().

commit   : 843a602080e13ea0f205720892f7bfee58e075fb    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 7 Sep 2019 15:41:55 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 7 Sep 2019 15:41:55 -0400    

Click here for diff

This is required for some operations on Posix storage.  

M src/storage/info.h
M src/storage/posix/storage.c
M test/src/module/storage/posixTest.c

Remove extraneous semicolons and add spaces.

commit   : 1c7c1042b4509ff3ad42dad240d895617f01d644    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 7 Sep 2019 12:32:25 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 7 Sep 2019 12:32:25 -0400    

Click here for diff

M src/common/debug.h

Rename lstRemove() to lstRemoveIdx().

commit   : 2eb1d1fee315e1e757856d3d88eb74b8434dad74    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 6 Sep 2019 16:50:57 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 6 Sep 2019 16:50:57 -0400    

Click here for diff

This works better with the names of upcoming functions and seems sensible even in isolation.  

M src/common/type/list.c
M src/common/type/list.h
M src/info/infoBackup.c
M src/protocol/parallel.c
M test/src/module/common/typeListTest.c

Rename INFO_MANIFEST_FILE to MANIFEST_FILE.

commit   : dde0c2b0c9c9743a863b6064e7dbb69de26fa364    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 6 Sep 2019 16:43:53 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 6 Sep 2019 16:43:53 -0400    

Click here for diff

This was missed in 5c314df0 when the module was renamed.  

M src/command/expire/expire.c
M src/command/stanza/delete.c
M src/info/manifest.h
M test/src/module/command/expireTest.c

Improve performance of info file load/save.

commit   : 4d84820021ad2a96922c87263d9200aace42358f    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 6 Sep 2019 13:48:28 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 6 Sep 2019 13:48:28 -0400    

Click here for diff

Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load.  
  
This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files.  
  
Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum.  
  
Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests.  
  
Reviewed by Cynthia Shang.  

M doc/xml/release.xml
M src/Makefile.in
M src/command/archive/get/file.c
M src/command/archive/push/push.c
M src/command/check/check.c
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/stanza/create.c
M src/command/stanza/upgrade.c
M src/common/ini.c
M src/common/ini.h
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoPg.c
M src/info/infoPg.h
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/common/harnessInfo.c
M test/src/common/harnessInfo.h
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/common/iniTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c

Add helper function for adding CipherBlock filters to groups.

commit   : 7334f30c350b5275106b9acf28b60d999287d800    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 6 Sep 2019 13:35:28 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 6 Sep 2019 13:35:28 -0400    

Click here for diff

Reviewed by Cynthia Shang.  

M doc/xml/release.xml
M src/Makefile.in
M src/common/crypto/cipherBlock.c
M src/common/crypto/cipherBlock.h
M test/src/module/common/cryptoTest.c

Rename infoManifest module to manifest.

commit   : 5c314df098ee2295af400fb0b6b4f0cccd96fb69    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 5 Sep 2019 19:53:00 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 5 Sep 2019 19:53:00 -0400    

Click here for diff

The manifest is not an info file so if anything it should be called backupManifest.  But that seems too long for such a commonly used object so manifest seems better.  
  
Note that unlike Perl there is no storage manifest method so this stands as the only manifest in the C code, as befits its importance.  

M src/Makefile.in
M src/command/expire/expire.c
M src/command/stanza/delete.c
M src/info/infoBackup.c
R085 src/info/infoManifest.h src/info/manifest.h
M test/code-count/file-type.yaml

Fix sudo missed in "Build pgBackRest as an unprivileged user".

commit   : 8df7d68c8dca2c61b665bbbd1f26800c0dca3ae8    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 3 Sep 2019 18:28:53 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 3 Sep 2019 18:28:53 -0400    

Click here for diff

286a106a updated the documentation to build pgBackRest as an unprivileged user, but the wget command was missed.  This command is not actually run, just displayed, because the release is not yet available when the documentation is built.  
  
Update the wget command to run as the local user.  

M doc/xml/user-guide.xml

Begin v2.18 development.

commit   : 005684bf1f55206f122e1d0fcb4164cc84875ff5    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 3 Sep 2019 17:53:50 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 3 Sep 2019 17:53:50 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/perl/embed.auto.c
M src/version.h