v2.31: Minor Bug Fixes and Improvements
commit : e116b535e6ad21248c00a6d0d948575ed06c6e38
author : David Steele <[email protected]>
date : Mon, 7 Dec 2020 09:55:00 -0500
committer: David Steele <[email protected]>
date : Mon, 7 Dec 2020 09:55:00 -0500
Bug Fixes:
* Allow [, #, and space as the first character in database names. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Jefferson Alexandre.)
* Create standby.signal only on PostgreSQL 12 when restore type is standby. (Fixed by Stefan Fercot. Reviewed by David Steele. Reported by Keith Fiske.)
Features:
* Expire history files. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Report page checksum errors in info command text output. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
* Add repo-azure-endpoint option. (Reviewed by Cynthia Shang, Brian Peterson. Suggested by Brian Peterson.)
* Add pg-database option. (Reviewed by Cynthia Shang.)
Improvements:
* Improve info command output when a stanza is specified but missing. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele. Suggested by uspen.)
* Improve performance of large file lists in backup/restore commands. (Reviewed by Cynthia Shang, Oscar.)
* Add retries to PostgreSQL sleep when starting a backup. (Reviewed by Cynthia Shang. Suggested by Vitaliy Kukharik.)
Documentation Improvements:
* Replace RHEL/CentOS 6 documentation with RHEL/CentOS 8.
M CONTRIBUTING.md
M README.md
M doc/release.pl
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
Add RHEL/CentOS 8 documentation.
commit : 31becf05b7445f201250d5956e3b725dd4b9338e
author : David Steele <[email protected]>
date : Fri, 4 Dec 2020 10:59:57 -0500
committer: David Steele <[email protected]>
date : Fri, 4 Dec 2020 10:59:57 -0500
Update RHEL/CentOS 7 to cover the versions that were previously covered by RHEL/CentOS 6.
Since RHEL/CentOS 7/8 work the same update the documentation logic and labels to reflect this compatibility.
M doc/README.md
M doc/release.pl
M doc/xml/documentation.xml
M doc/xml/release.xml
M doc/xml/user-guide-index.xml
M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Common/VmTest.pm
Add pg-database option.
commit : b0ea3379651f610113cb63f0743b6245dd9d5ffe
author : David Steele <[email protected]>
date : Wed, 2 Dec 2020 22:42:50 -0500
committer: David Steele <[email protected]>
date : Wed, 2 Dec 2020 22:42:50 -0500
In some rare cases there is no postgres database so this option may be used to specify an alternate database.
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/parse.auto.c
M src/db/helper.c
M src/db/protocol.c
M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/db/dbTest.c
Fix incorrect usage of TEST_RESULT_DOUBLE().
commit : 1913ecee639e0d40e0d7bfedb898f2d881ac49f0
author : David Steele <[email protected]>
date : Wed, 2 Dec 2020 22:42:05 -0500
committer: David Steele <[email protected]>
date : Wed, 2 Dec 2020 22:42:05 -0500
These work fine but it's better to use the correct macro.
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeVariantTest.c
Add retries to PostgreSQL sleep when starting a backup.
commit : d4211d3aaf6137f0a2e635c1ac9b089771356bfd
author : David Steele <[email protected]>
date : Wed, 2 Dec 2020 22:41:14 -0500
committer: David Steele <[email protected]>
date : Wed, 2 Dec 2020 22:41:14 -0500
Inaccuracies in sleep time or clock skew might make a single sleep insufficient to reach the next second.
Add a few retries to make the process more reliable but still avoid an infinite loop if something is seriously wrong.
M doc/xml/release.xml
M src/command/backup/backup.c
M test/src/module/command/backupTest.c
Remove CentOS 6 from tests and documentation.
commit : ec9f23d31f809902156466a0e4c96ef042d31edb
author : David Steele <[email protected]>
date : Wed, 2 Dec 2020 16:23:05 -0500
committer: David Steele <[email protected]>
date : Wed, 2 Dec 2020 16:23:05 -0500
CentOS6 EOL'd and the mirrors were swiftly deleted, leading to failures in tests and documentation.
Remove CentOS 6 for now to get builds going again with the intention to replace it in the near future with CentOS 8.
M .travis.yml
M doc/RELEASE.md
M doc/release.pl
M doc/xml/documentation.xml
M doc/xml/user-guide-index.xml
M doc/xml/user-guide.xml
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/test.pl
Improve error message when PQgetCancel() returns NULL.
commit : ffc50719d90832d1c24ca9b56ac36f76f5bfcaa6
author : David Steele <[email protected]>
date : Tue, 1 Dec 2020 15:15:35 -0500
committer: David Steele <[email protected]>
date : Tue, 1 Dec 2020 15:15:35 -0500
There is not a lot to be done in this case since it looks like PostgreSQL disconnected while the query was running, but at least improve the error message and remove the assert, which indicates a coding error.
M src/postgres/client.c
M test/src/common/harnessPq.c
M test/src/module/postgres/clientTest.c
Remove extraneous storageRepo() calls.
commit : d1d25c710d736a0946d53164b814f1c914603bdd
author : David Steele <[email protected]>
date : Mon, 30 Nov 2020 18:03:24 -0500
committer: David Steele <[email protected]>
date : Mon, 30 Nov 2020 18:03:24 -0500
These calls are not required since cipher info is passed explicitly. They are probably a copy-pasto from some past time when one of these functions required it.
M src/command/archive/get/protocol.c
M src/command/restore/protocol.c
M src/command/verify/protocol.c
Report page checksum errors in info command text output.
commit : 5488de8b6a77e8fcd635af707cbd604f985d1b9a
author : Stefan Fercot <[email protected]>
date : Wed, 25 Nov 2020 18:14:03 +0100
committer: GitHub <[email protected]>
date : Wed, 25 Nov 2020 18:14:03 +0100
This feature currently only works for text output. JSON output is planned for the future.
M doc/xml/release.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c
Conform retry in lockAcquireFile() to the common retry pattern.
commit : 3ed7b93b90eb61d40221701e0aef1318c1ea2b2b
author : Cynthia Shang <[email protected]>
date : Tue, 24 Nov 2020 09:40:44 -0500
committer: David Steele <[email protected]>
date : Tue, 24 Nov 2020 09:40:44 -0500
M doc/xml/release.xml
M src/common/lock.c
Prepare configuration module for multi-repository support.
commit : 117f03eba15378cfa15097828041331b67d16f12
author : David Steele <[email protected]>
date : Mon, 23 Nov 2020 15:55:46 -0500
committer: David Steele <[email protected]>
date : Mon, 23 Nov 2020 15:55:46 -0500
Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum).
Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage.
Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption.
Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy.
Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn.
Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related.
Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/check/common.c
M src/command/command.c
M src/command/help/help.c
M src/common/type/stringz.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.h
A src/config/config.intern.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/define.c
M src/config/define.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M src/config/protocol.c
M src/db/helper.c
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/helper.c
M src/storage/helper.h
M test/define.yaml
M test/src/common/harnessConfig.c
M test/src/module/command/localTest.c
D test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c
Allow multiple remote locks from the same main process.
commit : 7fda83b31e73467c881dca43cc6165e4a3ae3963
author : David Steele <[email protected]>
date : Mon, 23 Nov 2020 12:41:54 -0500
committer: GitHub <[email protected]>
date : Mon, 23 Nov 2020 12:41:54 -0500
Improve locking on remote processes by introducing an exec-id that is unique to the main process and passed to all remote processes. This allows the remote processes to determine if a lock is held by a remote from the same main process. If so, the lock is allowed.
The exec-id is also useful for associating remote logs with main logs for debugging purposes.
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/control/stop.c
M src/command/info/info.c
M src/command/remote/remote.c
M src/command/restore/restore.c
M src/common/lock.c
M src/common/lock.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/src/common/harnessConfig.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/protocol/protocolTest.c
Update Vagrant box version.
commit : 77198dce97dd70fdf59ade2886b41e9029553217
author : David Steele <[email protected]>
date : Sun, 22 Nov 2020 16:26:30 -0500
committer: David Steele <[email protected]>
date : Sun, 22 Nov 2020 16:26:30 -0500
M test/Vagrantfile
Create standby.signal only on PostgreSQL 12 when restore type is standby.
commit : 191b8ec18ba6a2fceb55fb280c64f1a49767b646
author : Stefan Fercot <[email protected]>
date : Thu, 19 Nov 2020 22:57:19 +0100
committer: GitHub <[email protected]>
date : Thu, 19 Nov 2020 22:57:19 +0100
When restore type standby is provided, the recovery.signal isn't needed and may lead to some confusion (see #1236).
Lately, when using pg_basebackup --write-recovery-conf, only the standby.signal file is created. This change would then align with that behaviour.
M doc/xml/release.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c
Move retrieval of archiveResult before while loop in verifyArchive().
commit : 9aacd3c54a82ca27d6950300a34fbad8bd1ec45c
author : Cynthia Shang <[email protected]>
date : Wed, 18 Nov 2020 18:19:49 -0500
committer: GitHub <[email protected]>
date : Wed, 18 Nov 2020 18:19:49 -0500
The result structure for the archive id being processed only needs to be retrieved once so moving it outside of the WAL path list processing loop is more efficient.
M src/command/verify/verify.c
Rename hrnCfgEnvId*() to hrnCfgEnvKey*().
commit : 62d9f2396127c305be2a2463cfe98ad872709403
author : David Steele <[email protected]>
date : Tue, 17 Nov 2020 10:48:40 -0500
committer: David Steele <[email protected]>
date : Tue, 17 Nov 2020 10:48:40 -0500
The hrnCfg*() functions were renamed before commit but these got missed.
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
Fix override of user option reset.
commit : dd708e6d27ac7b5917704fd07b40bb6fe2a2eee7
author : David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:37:05 -0500
committer: David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:37:05 -0500
If a user reset an option such as pg-default on the command-line then an override in the code would not take effect.
Ignore a reset when the code explicitly sets an option to prevent this.
M src/config/exec.c
M test/src/module/config/execTest.c
Fix test title in storage/remote test.
commit : 6acd76b60ce9bacb7180da1661745f3eaa7d6674
author : David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:28:53 -0500
committer: David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:28:53 -0500
M test/src/module/storage/remoteTest.c
Fix spacing and typos in backup/manifest modules.
commit : 63c12d25418d6252da643d50020f17cc496f0a11
author : David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:26:43 -0500
committer: David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:26:43 -0500
M src/info/manifest.c
M test/src/module/command/backupTest.c
M test/src/module/info/manifestTest.c
Remove obsolete call to storageRepo() in archivePushProtocol().
commit : 3f7a66fffc1062027166056b9cad2459d160c613
author : David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:19:37 -0500
committer: David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:19:37 -0500
This call to storageRepo() was used to fetch cipher options from a remote to determine if a repo cipher was enabled.
Now the main process does this work and passes the cipher options directly to the local so there is no need to pre-load the repo storage here.
M src/command/archive/push/protocol.c
Check archive push queue limit before checking repository.
commit : cbb9b8fd2b8579ea1e9f129c2b79e4a7151a2757
author : David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:16:41 -0500
committer: David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:16:41 -0500
If the push queue limit has been exceeded then nothing will be pushed to the repo so there is no point in checking it. Worse, a failure in the check would cause drop not to run and potentially fill up the disk, exactly the case this feature was designed to prevent.
The async version already checks the push queue limit before checking the repository so now both versions have the same behavior.
M src/command/archive/push/push.c
Log warnings in archive-push async log.
commit : 8cf47f82e49cb3151cd48c8430a1a38c7b3ccaa0
author : David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:10:59 -0500
committer: David Steele <[email protected]>
date : Mon, 9 Nov 2020 16:10:59 -0500
These warnings were only being reported to PostgreSQL on the console. Now they are also recorded in the async log increasing the chance that they will be seen.
This also improves coverage by requiring a warning during async processing to have a test case, which has been added.
M src/command/archive/push/push.c
M test/src/module/command/archivePushTest.c
Use a constant to check restore target action.
commit : d5d1ec6f6f4820808f30f4e9015bf22f2534de4e
author : David Steele <[email protected]>
date : Wed, 4 Nov 2020 11:09:05 -0500
committer: David Steele <[email protected]>
date : Wed, 4 Nov 2020 11:09:05 -0500
Checking the default here was fragile. If the default were to change the code would break.
This also removes the only dependency on cfgOptionDefault() outside of the help command.
M src/command/restore/restore.c
Fix typos.
commit : 021a34a55bef5370441c7a416514efebb7086177
author : Cynthia Shang <[email protected]>
date : Thu, 29 Oct 2020 17:16:15 -0400
committer: David Steele <[email protected]>
date : Thu, 29 Oct 2020 17:16:15 -0400
M src/config/parse.c
Spell out three exclamation points in git-history.cache.
commit : f87a117b71ce0d99137b6df5405fa1fd592af4b4
author : David Steele <[email protected]>
date : Wed, 28 Oct 2020 08:59:43 -0400
committer: David Steele <[email protected]>
date : Wed, 28 Oct 2020 08:59:43 -0400
Three exclamation points are used by convention as a marker for code that needs attention before it can be committed to integration.
If the markers are in this file they come up in every search.
M doc/resource/git-history.cache
Use harnessCfgLoad() in config/load unit tests.
commit : d25e965c475e0a36aad8e4a2df7e0d840185cded
author : David Steele <[email protected]>
date : Tue, 27 Oct 2020 13:29:58 -0400
committer: David Steele <[email protected]>
date : Tue, 27 Oct 2020 13:29:58 -0400
The tests were originally written by loading values directly into the configuration before the parser was available.
Update to use harnessCfgLoad() to simplify the tests and make them compatible with upcoming config changes.
M test/src/module/config/loadTest.c
Fix comment.
commit : d91e6014bd1b6ff6f68abee7944250ed16ee254c
author : David Steele <[email protected]>
date : Tue, 27 Oct 2020 13:18:25 -0400
committer: David Steele <[email protected]>
date : Tue, 27 Oct 2020 13:18:25 -0400
M src/config/parse.c
Improve info command output when a stanza is specified but missing.
commit : abe9d90c891fe61b0b0e7d1729dc89ca10b3b2e8
author : Stefan Fercot <[email protected]>
date : Tue, 27 Oct 2020 13:34:18 +0100
committer: GitHub <[email protected]>
date : Tue, 27 Oct 2020 13:34:18 +0100
Return a path missing error when a stanza is specified for the info command but the stanza does not exist in the repository.
Previously [] was returned, which is still the case if no stanza is specified and the repository does not exist.
M doc/xml/release.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c
Improve performance of large file lists in backup/restore commands.
commit : 770b65de804fc56a3e6a00d3916958c99774880d
author : David Steele <[email protected]>
date : Mon, 26 Oct 2020 12:18:45 -0400
committer: GitHub <[email protected]>
date : Mon, 26 Oct 2020 12:18:45 -0400
lstRemoveIdx(list, 0) resulted in the entire list being moved down to the first position which could take a long time for big lists. This is a common pattern in backup/restore when processing file queues.
Instead simply move the list pointer up when first item is removed. Then on insert check if there is space at the beginning when there is no longer space at the end and do the move then. This way if a list is built and then drained without any new inserts then no move is required.
M doc/xml/release.xml
M src/common/type/list.c
M test/define.yaml
M test/src/module/common/typeListTest.c
M test/src/module/performance/typeTest.c
Use zero-based indexes when referring to option indexes.
commit : d452e9cc38001e2258cc05137508ab692d226df2
author : David Steele <[email protected]>
date : Mon, 26 Oct 2020 10:25:16 -0400
committer: GitHub <[email protected]>
date : Mon, 26 Oct 2020 10:25:16 -0400
There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa.
Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user.
Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/check/common.c
M src/command/check/common.h
M src/command/restore/restore.c
M src/command/stanza/common.c
M src/command/verify/verify.c
M src/db/helper.c
M src/db/helper.h
M src/protocol/helper.c
M src/protocol/helper.h
M src/storage/helper.c
M src/storage/helper.h
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/db/dbTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c
Remove unused FUNCTION_LOG_VERIFY_WAL_RANGE* defines.
commit : ae35c4f0292e410c06e71eed6c5d8565017351d1
author : Cynthia Shang <[email protected]>
date : Mon, 26 Oct 2020 07:41:08 -0400
committer: GitHub <[email protected]>
date : Mon, 26 Oct 2020 07:41:08 -0400
The defines for FUNCTION_LOG_VERIFY_WAL_RANGE* are not used in the current verify.c and are currently not planned in the continuing development of the verify command, so they are dead code and are therefore being removed.
M src/command/verify/verify.c
Replace misuse of bufSize() with bufUsed().
commit : ea49151746a2d349ccb3c25e8e41df1e88d59a06
author : David Steele <[email protected]>
date : Sat, 24 Oct 2020 13:18:02 -0400
committer: David Steele <[email protected]>
date : Sat, 24 Oct 2020 13:18:02 -0400
bufSize() should only be used whem checking the total size of the buffer, not how much of it is currently used.
In these cases bufUsed() and bufSize() are returning the same value but benign-looking code changes could break this assumption.
M src/common/crypto/cipherBlock.c
M src/common/io/read.c
M src/common/type/xml.c
M src/storage/s3/storage.c
Clarify intended usage of harnessCfgLoadRaw().
commit : 80a770ed2ea8d060d8ea77dabb239cf7e861774d
author : David Steele <[email protected]>
date : Sat, 24 Oct 2020 11:16:06 -0400
committer: David Steele <[email protected]>
date : Sat, 24 Oct 2020 11:16:06 -0400
M test/src/common/harnessConfig.h
Allow [, #, and space as the first character in database names.
commit : 76cfd8ca703f4d6a71c6fe10b683209a2040368c
author : David Steele <[email protected]>
date : Sat, 24 Oct 2020 11:07:07 -0400
committer: GitHub <[email protected]>
date : Sat, 24 Oct 2020 11:07:07 -0400
iniLoad() was trimming lines which meant that a leading space would not pass checksum validation when a manifest was reloaded. Remove the trims since files we write should never contain extraneous spaces. This further diverges the format for the functions that read conf files (e.g. pgbackrest.conf) and those that read info (e.g. manifest) files.
While we are at it also allow [ and # as initial characters. # was reserved for comments but we never put comments into info files. [ denotes a section but we can get around this by never allowing arrays as values in info files, so if a line ends in ] it must be a section. This is currently the case but enforce it by adding an assert to info/info.c.
M doc/xml/release.xml
M src/common/ini.c
M src/common/ini.h
M src/info/info.c
M test/src/module/command/backupTest.c
M test/src/module/common/iniTest.c
M test/src/module/info/manifestTest.c
Add indexed option > 1 to config/exec unit test.
commit : c573ef281489e274c9b59185275f9c595885f359
author : David Steele <[email protected]>
date : Tue, 20 Oct 2020 16:14:03 -0400
committer: David Steele <[email protected]>
date : Tue, 20 Oct 2020 16:14:03 -0400
M test/src/module/config/execTest.c
Use harnessCfgLoadRaw() in command/command and common/exit unit tests.
commit : 176cf0bf609ac2fc49c411537d4aa518dccd33d9
author : David Steele <[email protected]>
date : Tue, 20 Oct 2020 14:54:28 -0400
committer: David Steele <[email protected]>
date : Tue, 20 Oct 2020 14:54:28 -0400
The tests were originally written by loading values directly into the configuration before the parser was available.
Update to use harnessCfgLoadRaw() to simplify the tests and make them compatible with upcoming config changes.
Note that some unreachable conditions were removed since they could not be reached via a parsed config, only by munging values directly into the config. cfgOptionTest(optionId) was removed because a non-default value must always be set. cfgOptionValid(cfgOptLogTimestamp) was removed because it is true for all commands except for cfgCmdNone, which is checked with an assert.
M src/command/command.c
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/module/command/commandTest.c
M test/src/module/common/exitTest.c
Get target-action default from cfgOptionDefault() in restore command.
commit : 156b7d48cce789cc6e435219062ceeadbbb7b494
author : David Steele <[email protected]>
date : Tue, 20 Oct 2020 12:39:23 -0400
committer: David Steele <[email protected]>
date : Tue, 20 Oct 2020 12:39:23 -0400
cfgDefOptionDefault() worked but the default is available without having to peek into config definitions.
M src/command/restore/restore.c
Remove cfgOptionId() and replace it with cfgParseOption().
commit : 41789d70d16a3921c68132187f08a1b8fa058cf1
author : David Steele <[email protected]>
date : Tue, 20 Oct 2020 11:24:26 -0400
committer: David Steele <[email protected]>
date : Tue, 20 Oct 2020 11:24:26 -0400
cfgOptionId() did not recognize deprecated options which made the help command throw errors when they were specified on the command line. cfgParseOption() will correctly identify deprecated options.
cfgParseOption() can also be used in cfgParse() to reduce code duplication when parsing info out of the option value returned by optionFind().
Finally, code the option key index separately in parse.auto.c. For now they are simply added back together but future code will need them separated.
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/command/help/help.c
M src/config/config.c
M src/config/config.h
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M src/config/protocol.c
M test/src/module/command/helpTest.c
M test/src/module/config/configTest.c
M test/src/module/config/parseTest.c
Remove ConfigDefineCommand enum.
commit : 6414ae9707ab637f298df1ce55fe4bcf46fbe910
author : David Steele <[email protected]>
date : Mon, 19 Oct 2020 18:17:47 -0400
committer: David Steele <[email protected]>
date : Mon, 19 Oct 2020 18:17:47 -0400
This has always been equivalent to the ConfigCommand enum so it just adds complexity.
It was created for symmetry with ConfigDefineOption, which will also be removed soon.
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M src/command/command.c
M src/command/help/help.c
M src/command/restore/restore.c
M src/config/config.c
M src/config/config.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/define.c
M src/config/define.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M src/protocol/helper.c
M test/src/common/harnessConfig.c
M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/parseTest.c
Fix repo/pg-local option being overwritten.
commit : ba29e24684414c610e58374c000cb66e28315975
author : David Steele <[email protected]>
date : Mon, 19 Oct 2020 17:17:47 -0400
committer: David Steele <[email protected]>
date : Mon, 19 Oct 2020 17:17:47 -0400
This option would get overwritten when the option values were moved to index 1.
Set the repo/pg-local option after options are moved to fix this.
M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c
Reset additional options in real/all integration test.
commit : 2d38d2fc824db1bb3f56899addd51ab018977d12
author : David Steele <[email protected]>
date : Mon, 19 Oct 2020 17:06:52 -0400
committer: David Steele <[email protected]>
date : Mon, 19 Oct 2020 17:06:52 -0400
Currently indexes above 1 do not have dependencies checked, so this doesn't error.
In a future commit we will enable those checks and this will error if it is not fixed.
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
Use harnessCfgLoadRaw() in command/help unit test.
commit : 996387ea9312459db664e59d68b94455c9c56acc
author : David Steele <[email protected]>
date : Mon, 19 Oct 2020 16:27:52 -0400
committer: David Steele <[email protected]>
date : Mon, 19 Oct 2020 16:27:52 -0400
This correctly set defaults from cfgLoadUpdateOption(), e.g. repo-host-cmd.
M test/src/module/command/helpTest.c
Remove extraneous calls to cfgOptionValid() in unit tests.
commit : 303b9b17e700dca00aa12a3afe23b1f7c9741752
author : David Steele <[email protected]>
date : Mon, 19 Oct 2020 14:19:28 -0400
committer: David Steele <[email protected]>
date : Mon, 19 Oct 2020 14:19:28 -0400
M test/src/module/command/repoTest.c
M test/src/module/storage/remoteTest.c
Remove indexed option constants.
commit : 7d069a2b9178498293372913b0fa287005aa93b4
author : David Steele <[email protected]>
date : Mon, 19 Oct 2020 14:03:48 -0400
committer: David Steele <[email protected]>
date : Mon, 19 Oct 2020 14:03:48 -0400
These constants don't scale well as the index total is increased for an option.
The core code rarely uses these options and they are easily replaced with cfgOptionName().
The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/command/archive/common.c
M src/command/archive/push/push.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/protocol/helper.c
M test/src/common/harnessConfig.c
M test/src/common/harnessConfig.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c
Expire history files.
commit : 86275c4f85668d4576c16c85c629a7f3515ade59
author : Stefan Fercot <[email protected]>
date : Fri, 16 Oct 2020 13:42:03 +0200
committer: GitHub <[email protected]>
date : Fri, 16 Oct 2020 13:42:03 +0200
WAL timeline history files were not being expired because they were small and generally not very plentiful.
However, in some cases large numbers of history files may be generated so it makes sense to remove useless history files to keep things tidy.
The history file for the oldest retained timeline is kept for debugging purposes even though it is not used for recovery.
M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/expire/expire.c
M test/src/module/command/expireTest.c
Assert when buffer used is greater than size limit.
commit : 4b8dfd34c0a9ced0977c43c2b4159eb700eadf28
author : David Steele <[email protected]>
date : Thu, 8 Oct 2020 17:09:21 -0400
committer: David Steele <[email protected]>
date : Thu, 8 Oct 2020 17:09:21 -0400
099281c5 reduced used but this could lead to surprising behavior so assert instead.
Call bufUsedSet() before bufLimitSet() to avoid the assertion.
M doc/xml/release.xml
M src/common/type/buffer.c
M test/src/module/common/typeBufferTest.c
Optimize small reads in IoRead.
commit : 401e4f1df41a40860c784f1908994c6ee51f6a51
author : David Steele <[email protected]>
date : Thu, 8 Oct 2020 16:09:38 -0400
committer: GitHub <[email protected]>
date : Thu, 8 Oct 2020 16:09:38 -0400
Instead of using memmove() to manage the internal output buffer for every small read, track the current buffer position and only move data when the small read cannot be satisfied and more data is needed.
M doc/xml/release.xml
M src/common/io/read.c
Add option groups.
commit : e0f09687e49805a4349933f5e244a2e88b27220b
author : David Steele <[email protected]>
date : Thu, 8 Oct 2020 10:52:19 -0400
committer: GitHub <[email protected]>
date : Thu, 8 Oct 2020 10:52:19 -0400
Group related options together so operations (e.g. valid, test, index total) can be performed on all options in the group.
Previously, options at the top of the hierarchy of the related options were used to do these tests. This was prone to error as option relationships changed and it was not always clear which option (or options) should be used.
M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/command/check/check.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.h
M src/config/load.c
M src/db/helper.c
M test/src/module/config/configTest.c
M test/src/module/config/loadTest.c
Add missing assertion in lstSort().
commit : dc8a9dab1b4e9b00a0e3b1394f519464c702cd60
author : David Steele <[email protected]>
date : Thu, 8 Oct 2020 08:37:53 -0400
committer: David Steele <[email protected]>
date : Thu, 8 Oct 2020 08:37:53 -0400
M src/common/type/list.c
Reduce buffer used when size limit set.
commit : 099281c51070fce5c801f90388b8f4316cea7f64
author : David Steele <[email protected]>
date : Wed, 7 Oct 2020 18:02:05 -0400
committer: David Steele <[email protected]>
date : Wed, 7 Oct 2020 18:02:05 -0400
If the size is reduced then the amount used should be reduced as well.
This is a bug but does not affect current use cases.
M src/common/type/buffer.c
M test/src/module/common/typeBufferTest.c
Add bufSizeAlloc().
commit : 39aca3269c440e635ae6ff04235628a9f9d30a79
author : David Steele <[email protected]>
date : Wed, 7 Oct 2020 16:34:13 -0400
committer: David Steele <[email protected]>
date : Wed, 7 Oct 2020 16:34:13 -0400
M src/common/type/buffer.h
M test/src/module/common/typeBufferTest.c
Add missing assertion in lstFind().
commit : c8a94d9eb6bb3972071eb3a4eb8e54ece1a5b56e
author : David Steele <[email protected]>
date : Wed, 7 Oct 2020 15:17:16 -0400
committer: David Steele <[email protected]>
date : Wed, 7 Oct 2020 15:17:16 -0400
M src/common/type/list.c
Remove unused cfgDefOptionPrefix() function and data.
commit : 98f6a1cffdd6eb2667a3d042aa5ec3c192129700
author : David Steele <[email protected]>
date : Wed, 7 Oct 2020 12:15:50 -0400
committer: David Steele <[email protected]>
date : Wed, 7 Oct 2020 12:15:50 -0400
This function was made obsolete when Perl was removed in f0ef73db.
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M src/config/define.auto.c
M src/config/define.c
M src/config/define.h
M test/src/module/config/defineTest.c
Declare cfgDefOptionAllowListValue/Total() static.
commit : 434282d343c131387ffc080ec5db9aa02ba3c413
author : David Steele <[email protected]>
date : Wed, 7 Oct 2020 12:03:23 -0400
committer: David Steele <[email protected]>
date : Wed, 7 Oct 2020 12:03:23 -0400
These have not been used externally since Perl was removed in f0ef73db.
M src/config/define.c
M src/config/define.h
Add repo-azure-endpoint option.
commit : 9377d050724d334a18706b49090575f2160433f1
author : David Steele <[email protected]>
date : Tue, 6 Oct 2020 17:15:48 -0400
committer: GitHub <[email protected]>
date : Tue, 6 Oct 2020 17:15:48 -0400
This option allows alternate endpoints (e.g. Azure Government) to be configured.
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/storage/azure/storage.c
M src/storage/azure/storage.h
M src/storage/helper.c
M test/src/module/command/helpTest.c
M test/src/module/storage/azureTest.c
Update config.guess and config.sub to latest versions.
commit : 3ce6b3e5058a7b1893aeb0c587fe8938956bbef5
author : David Steele <[email protected]>
date : Mon, 5 Oct 2020 13:12:48 -0400
committer: David Steele <[email protected]>
date : Mon, 5 Oct 2020 13:12:48 -0400
M src/build/config.guess
M src/build/config.sub
Begin v2.31 development.
commit : 3c8774fbab80a4300fa1d6935bc87036e41057dc
author : David Steele <[email protected]>
date : Mon, 5 Oct 2020 12:02:31 -0400
committer: David Steele <[email protected]>
date : Mon, 5 Oct 2020 12:02:31 -0400
M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h