pgBackRest 2.32 (upcoming) commit log

Improve performance of large file lists in backup/restore commands.

commit   : 770b65de804fc56a3e6a00d3916958c99774880d    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Mon, 26 Oct 2020 12:18:45 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Mon, 26 Oct 2020 12:18:45 -0400    

Click here for diff

lstRemoveIdx(list, 0) resulted in the entire list being moved down to the first position which could take a long time for big lists. This is a common pattern in backup/restore when processing file queues.
  

  
Instead simply move the list pointer up when first item is removed. Then on insert check if there is space at the beginning when there is no longer space at the end and do the move then. This way if a list is built and then drained without any new inserts then no move is required.

M doc/xml/release.xml
M src/common/type/list.c
M test/define.yaml
M test/src/module/common/typeListTest.c
M test/src/module/performance/typeTest.c

Use zero-based indexes when referring to option indexes.

commit   : d452e9cc38001e2258cc05137508ab692d226df2    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Mon, 26 Oct 2020 10:25:16 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Mon, 26 Oct 2020 10:25:16 -0400    

Click here for diff

There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa.
  

  
Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user.
  

  
Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/check/common.c
M src/command/check/common.h
M src/command/restore/restore.c
M src/command/stanza/common.c
M src/command/verify/verify.c
M src/db/helper.c
M src/db/helper.h
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/helper.c
M src/storage/helper.h
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/db/dbTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c

Remove unused FUNCTION_LOG_VERIFY_WAL_RANGE* defines.

commit   : ae35c4f0292e410c06e71eed6c5d8565017351d1    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Mon, 26 Oct 2020 07:41:08 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Mon, 26 Oct 2020 07:41:08 -0400    

Click here for diff

The defines for FUNCTION_LOG_VERIFY_WAL_RANGE* are not used in the current verify.c and are currently not planned in the continuing development of the verify command, so they are dead code and are therefore being removed.

M src/command/verify/verify.c

Replace misuse of bufSize() with bufUsed().

commit   : ea49151746a2d349ccb3c25e8e41df1e88d59a06    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 24 Oct 2020 13:18:02 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 24 Oct 2020 13:18:02 -0400    

Click here for diff

bufSize() should only be used whem checking the total size of the buffer, not how much of it is currently used.  
  
In these cases bufUsed() and bufSize() are returning the same value but benign-looking code changes could break this assumption.  

M src/common/crypto/cipherBlock.c
M src/common/io/read.c
M src/common/type/xml.c
M src/storage/s3/storage.c

Clarify intended usage of harnessCfgLoadRaw().

commit   : 80a770ed2ea8d060d8ea77dabb239cf7e861774d    
  
author   : David Steele <david@pgmasters.net>    
date     : Sat, 24 Oct 2020 11:16:06 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sat, 24 Oct 2020 11:16:06 -0400    

Click here for diff

M test/src/common/harnessConfig.h

Allow [, #, and space as the first character in database names.

commit   : 76cfd8ca703f4d6a71c6fe10b683209a2040368c    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Sat, 24 Oct 2020 11:07:07 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Sat, 24 Oct 2020 11:07:07 -0400    

Click here for diff

iniLoad() was trimming lines which meant that a leading space would not pass checksum validation when a manifest was reloaded. Remove the trims since files we write should never contain extraneous spaces. This further diverges the format for the functions that read conf files (e.g. pgbackrest.conf) and those that read info (e.g. manifest) files.
  

  
While we are at it also allow [ and # as initial characters. # was reserved for comments but we never put comments into info files. [ denotes a section but we can get around this by never allowing arrays as values in info files, so if a line ends in ] it must be a section. This is currently the case but enforce it by adding an assert to info/info.c.

M doc/xml/release.xml
M src/common/ini.c
M src/common/ini.h
M src/info/info.c
M test/src/module/command/backupTest.c
M test/src/module/common/iniTest.c
M test/src/module/info/manifestTest.c

Add indexed option > 1 to config/exec unit test.

commit   : c573ef281489e274c9b59185275f9c595885f359    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 20 Oct 2020 16:14:03 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 20 Oct 2020 16:14:03 -0400    

Click here for diff

M test/src/module/config/execTest.c

Use harnessCfgLoadRaw() in command/command and common/exit unit tests.

commit   : 176cf0bf609ac2fc49c411537d4aa518dccd33d9    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 20 Oct 2020 14:54:28 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 20 Oct 2020 14:54:28 -0400    

Click here for diff

The tests were originally written by loading values directly into the configuration before the parser was available.  
  
Update to use harnessCfgLoadRaw() to simplify the tests and make them compatible with upcoming config changes.  
  
Note that some unreachable conditions were removed since they could not be reached via a parsed config, only by munging values directly into the config. cfgOptionTest(optionId) was removed because a non-default value must always be set. cfgOptionValid(cfgOptLogTimestamp) was removed because it is true for all commands except for cfgCmdNone, which is checked with an assert.  

M src/command/command.c
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/module/command/commandTest.c
M test/src/module/common/exitTest.c

Get target-action default from cfgOptionDefault() in restore command.

commit   : 156b7d48cce789cc6e435219062ceeadbbb7b494    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 20 Oct 2020 12:39:23 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 20 Oct 2020 12:39:23 -0400    

Click here for diff

cfgDefOptionDefault() worked but the default is available without having to peek into config definitions.  

M src/command/restore/restore.c

Remove cfgOptionId() and replace it with cfgParseOption().

commit   : 41789d70d16a3921c68132187f08a1b8fa058cf1    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 20 Oct 2020 11:24:26 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 20 Oct 2020 11:24:26 -0400    

Click here for diff

cfgOptionId() did not recognize deprecated options which made the help command throw errors when they were specified on the command line. cfgParseOption() will correctly identify deprecated options.  
  
cfgParseOption() can also be used in cfgParse() to reduce code duplication when parsing info out of the option value returned by optionFind().  
  
Finally, code the option key index separately in parse.auto.c. For now they are simply added back together but future code will need them separated.  

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/command/help/help.c
M src/config/config.c
M src/config/config.h
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M src/config/protocol.c
M test/src/module/command/helpTest.c
M test/src/module/config/configTest.c
M test/src/module/config/parseTest.c

Remove ConfigDefineCommand enum.

commit   : 6414ae9707ab637f298df1ce55fe4bcf46fbe910    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 18:17:47 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 18:17:47 -0400    

Click here for diff

This has always been equivalent to the ConfigCommand enum so it just adds complexity.  
  
It was created for symmetry with ConfigDefineOption, which will also be removed soon.  

M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M src/command/command.c
M src/command/help/help.c
M src/command/restore/restore.c
M src/config/config.c
M src/config/config.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/define.c
M src/config/define.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M src/protocol/helper.c
M test/src/common/harnessConfig.c
M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/parseTest.c

Fix repo/pg-local option being overwritten.

commit   : ba29e24684414c610e58374c000cb66e28315975    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 17:17:47 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 17:17:47 -0400    

Click here for diff

This option would get overwritten when the option values were moved to index 1.  
  
Set the repo/pg-local option after options are moved to fix this.  

M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Reset additional options in real/all integration test.

commit   : 2d38d2fc824db1bb3f56899addd51ab018977d12    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 17:06:52 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 17:06:52 -0400    

Click here for diff

Currently indexes above 1 do not have dependencies checked, so this doesn't error.  
  
In a future commit we will enable those checks and this will error if it is not fixed.  

M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Use harnessCfgLoadRaw() in command/help unit test.

commit   : 996387ea9312459db664e59d68b94455c9c56acc    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 16:27:52 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 16:27:52 -0400    

Click here for diff

This correctly set defaults from cfgLoadUpdateOption(), e.g. repo-host-cmd.  

M test/src/module/command/helpTest.c

Remove extraneous calls to cfgOptionValid() in unit tests.

commit   : 303b9b17e700dca00aa12a3afe23b1f7c9741752    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 14:19:28 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 14:19:28 -0400    

Click here for diff

M test/src/module/command/repoTest.c
M test/src/module/storage/remoteTest.c

Remove indexed option constants.

commit   : 7d069a2b9178498293372913b0fa287005aa93b4    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 14:03:48 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 19 Oct 2020 14:03:48 -0400    

Click here for diff

These constants don't scale well as the index total is increased for an option.  
  
The core code rarely uses these options and they are easily replaced with cfgOptionName().  
  
The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.  

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/command/archive/common.c
M src/command/archive/push/push.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/protocol/helper.c
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Expire history files.

commit   : 86275c4f85668d4576c16c85c629a7f3515ade59    
  
author   : Stefan Fercot <stefan.fercot@enterprisedb.com>    
date     : Fri, 16 Oct 2020 13:42:03 +0200    
  
committer: GitHub <noreply@github.com>    
date     : Fri, 16 Oct 2020 13:42:03 +0200    

Click here for diff

WAL timeline history files were not being expired because they were small and generally not very plentiful.
  

  
However, in some cases large numbers of history files may be generated so it makes sense to remove useless history files to keep things tidy.
  

  
The history file for the oldest retained timeline is kept for debugging purposes even though it is not used for recovery.

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/expire/expire.c
M test/src/module/command/expireTest.c

Assert when buffer used is greater than size limit.

commit   : 4b8dfd34c0a9ced0977c43c2b4159eb700eadf28    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 8 Oct 2020 17:09:21 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 8 Oct 2020 17:09:21 -0400    

Click here for diff

099281c5 reduced used but this could lead to surprising behavior so assert instead.  
  
Call bufUsedSet() before bufLimitSet() to avoid the assertion.  

M doc/xml/release.xml
M src/common/type/buffer.c
M test/src/module/common/typeBufferTest.c

Optimize small reads in IoRead.

commit   : 401e4f1df41a40860c784f1908994c6ee51f6a51    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Thu, 8 Oct 2020 16:09:38 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Thu, 8 Oct 2020 16:09:38 -0400    

Click here for diff

Instead of using memmove() to manage the internal output buffer for every small read, track the current buffer position and only move data when the small read cannot be satisfied and more data is needed.

M doc/xml/release.xml
M src/common/io/read.c

Add option groups.

commit   : e0f09687e49805a4349933f5e244a2e88b27220b    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Thu, 8 Oct 2020 10:52:19 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Thu, 8 Oct 2020 10:52:19 -0400    

Click here for diff

Group related options together so operations (e.g. valid, test, index total) can be performed on all options in the group.
  

  
Previously, options at the top of the hierarchy of the related options were used to do these tests. This was prone to error as option relationships changed and it was not always clear which option (or options) should be used.

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/command/check/check.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.h
M src/config/load.c
M src/db/helper.c
M test/src/module/config/configTest.c
M test/src/module/config/loadTest.c

Add missing assertion in lstSort().

commit   : dc8a9dab1b4e9b00a0e3b1394f519464c702cd60    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 8 Oct 2020 08:37:53 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 8 Oct 2020 08:37:53 -0400    

Click here for diff

M src/common/type/list.c

Reduce buffer used when size limit set.

commit   : 099281c51070fce5c801f90388b8f4316cea7f64    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 18:02:05 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 18:02:05 -0400    

Click here for diff

If the size is reduced then the amount used should be reduced as well.  
  
This is a bug but does not affect current use cases.  

M src/common/type/buffer.c
M test/src/module/common/typeBufferTest.c

Add bufSizeAlloc().

commit   : 39aca3269c440e635ae6ff04235628a9f9d30a79    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 16:34:13 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 16:34:13 -0400    

Click here for diff

M src/common/type/buffer.h
M test/src/module/common/typeBufferTest.c

Add missing assertion in lstFind().

commit   : c8a94d9eb6bb3972071eb3a4eb8e54ece1a5b56e    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 15:17:16 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 15:17:16 -0400    

Click here for diff

M src/common/type/list.c

Remove unused cfgDefOptionPrefix() function and data.

commit   : 98f6a1cffdd6eb2667a3d042aa5ec3c192129700    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 12:15:50 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 12:15:50 -0400    

Click here for diff

This function was made obsolete when Perl was removed in f0ef73db.  

M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M src/config/define.auto.c
M src/config/define.c
M src/config/define.h
M test/src/module/config/defineTest.c

Declare cfgDefOptionAllowListValue/Total() static.

commit   : 434282d343c131387ffc080ec5db9aa02ba3c413    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 12:03:23 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 7 Oct 2020 12:03:23 -0400    

Click here for diff

These have not been used externally since Perl was removed in f0ef73db.  

M src/config/define.c
M src/config/define.h

Add repo-azure-endpoint option.

commit   : 9377d050724d334a18706b49090575f2160433f1    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Tue, 6 Oct 2020 17:15:48 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Tue, 6 Oct 2020 17:15:48 -0400    

Click here for diff

This option allows alternate endpoints (e.g. Azure Government) to be configured.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/storage/azure/storage.c
M src/storage/azure/storage.h
M src/storage/helper.c
M test/src/module/command/helpTest.c
M test/src/module/storage/azureTest.c

Update config.guess and config.sub to latest versions.

commit   : 3ce6b3e5058a7b1893aeb0c587fe8938956bbef5    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 5 Oct 2020 13:12:48 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 5 Oct 2020 13:12:48 -0400    

Click here for diff

M src/build/config.guess
M src/build/config.sub

Begin v2.31 development.

commit   : 3c8774fbab80a4300fa1d6935bc87036e41057dc    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 5 Oct 2020 12:02:31 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 5 Oct 2020 12:02:31 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h