pgBackRest 2.34 commit log

v2.33: Multi-Repository and GCS Support

commit   : a3d89143d6b34c6019cd29e059d28487037fd109    
  
author   : David Steele <[email protected]>    
date     : Mon, 5 Apr 2021 09:18:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 5 Apr 2021 09:18:20 -0400    

Click here for diff

Bug Fixes:  
  
* Fix option warnings breaking async archive-get/archive-push. (Reviewed by Cynthia Shang. Reported by Lev Kokotov.)  
* Fix memory leak in backup during archive copy. (Reviewed by Cynthia Shang. Reported by Christian ROUX, Efremov Egor.)  
* Fix stack overflow in cipher passphrase generation. (Reviewed by Cynthia Shang. Reported by bsiara.)  
* Fix repo-ls / on S3 repositories. (Reviewed by Cynthia Shang. Reported by Lesovsky Alexey.)  
  
Features:  
  
* Multiple repository support. (Contributed by Cynthia Shang, David Steele. Reviewed by Stefan Fercot, Stephen Frost.)  
* GCS support for repository storage. (Reviewed by Cynthia Shang.)  
* Add archive-header-check option. (Reviewed by Stephen Frost, Cynthia Shang. Suggested by Hans-Jürgen Schönig.)  
  
Improvements:  
  
* Include recreated system databases during selective restore. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)  
* Exclude content-length from S3 signed headers. (Reviewed by Cynthia Shang. Suggested by Brian P Bockelman.)  
* Consolidate less commonly used repository storage options. (Reviewed by Cynthia Shang.)  
* Allow custom config-path default with ./configure --with-configdir. (Contributed by Michael Schout. Reviewed by David Steele.)  
* Log archive copy during backup. (Reviewed by Cynthia Shang, Stefan Fercot.)  
  
Documentation Improvements:  
  
* Update reference to include links to user guide examples. (Contributed by Cynthia Shang. Reviewed by David Steele.)  
* Update selective restore documentation with caveats. (Reviewed by Cynthia Shang, Stefan Fercot.)  
* Add compress-type clarification to archive-copy documentation. (Reviewed by Cynthia Shang, Stefan Fercot.)  
* Add compress-level defaults per compress-type value. (Contributed by Cynthia Shang. Reviewed by David Steele.)  
* Add note about required NFS settings being the same as PostgreSQL. (Contributed by Cynthia Shang. Reviewed by David Steele.)  

M CONTRIBUTING.md
M README.md
M doc/RELEASE.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/aclocal.m4
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Fix extraneous hrnReplaceKey() calls in unit tests.

commit   : a6a544c7f9cda61af854ef980924d36c02b514bd    
  
author   : David Steele <[email protected]>    
date     : Fri, 2 Apr 2021 15:35:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 2 Apr 2021 15:35:41 -0400    

Click here for diff

hrnReplaceKey() was added to the TEST_ERROR*() macros in 58760486 but some calls to TEST_ERROR*() already used it. This led to the function being called twice on the same buffer which had no effect but valgrind definitely did not like.  
  
Remove extraneous calls to make valgrind happy. Since this is test code there are no implications for production.  

M test/src/module/command/archiveCommonTest.c
M test/src/module/info/manifestTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Add --gen-check option to test.pl to check autogenerated code.

commit   : 1615cb02610668b6eae2e3907519212eabd1ad30    
  
author   : David Steele <[email protected]>    
date     : Wed, 31 Mar 2021 10:36:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 31 Mar 2021 10:36:22 -0400    

Click here for diff

Make sure that auto-generated code does not change during CI.  
  
This is useful for catching missed code generation, especially help.auto.c.  

M test/ci.pl
M test/test.pl

Update to help.auto.c missed in d372dd652c9.

commit   : e89452557ee8b0b883716331e8d542310875eadd    
  
author   : David Steele <[email protected]>    
date     : Wed, 31 Mar 2021 09:41:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 31 Mar 2021 09:41:33 -0400    

Click here for diff

M src/command/help/help.auto.c

commit   : d372dd652c9a01b72444998d87f0f44464676803    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 31 Mar 2021 09:36:56 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 31 Mar 2021 09:36:56 -0400    

Click here for diff

The command-example and command-example-list elements were removed from the documentation rendering some time ago so these tags were dead code. The tags, however, contained some examples and information that were pertinent to the command, so where possible, the information was included in the description of the command and/or the user-guide and links to the relevant user guide sections were added.
  

  
Note that some commands could not be updated with user guide references since doing so would cause a cyclical reference in the user guide. These commands have an internal comment to indicate this.
  

  
In addition, some clarifications were added (e.g. expire --set option) where information was lacking.

M doc/xml/dtd/doc.dtd
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/help/help.auto.c
M test/src/module/command/helpTest.c

commit   : fead2360da5634b278dba621aa98790add120762    
  
author   : David Steele <[email protected]>    
date     : Tue, 30 Mar 2021 15:49:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 30 Mar 2021 15:49:03 -0400    

Click here for diff

This makes it easier to determine which release notes relate to issues and PRs, especially for bug fixes or user requests.  

M doc/xml/dtd/doc.dtd
M doc/xml/release.xml

Update to help.auto.c missed in 75987621.

commit   : 21db7f65ef497d323b37c0aa60c89f4c7c754622    
  
author   : David Steele <[email protected]>    
date     : Mon, 29 Mar 2021 09:49:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 29 Mar 2021 09:49:42 -0400    

Click here for diff

M src/command/help/help.auto.c

Add note about required NFS settings being the same as PostgreSQL.

commit   : 75987621fab5f57d0e95b42eabd301731587c438    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 26 Mar 2021 10:11:06 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 26 Mar 2021 10:11:06 -0400    

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c

Add compress-level defaults per compress-type value.

commit   : 3e206088e79c4a73a20b8f2fda1706c6b95fa188    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 26 Mar 2021 09:25:31 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 26 Mar 2021 09:25:31 -0400    

Click here for diff

Document these defaults until they can be added to the config parser and automated.

M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c

Add archive-header-check option.

commit   : b6106f3c1f586b410c50c9da8dcea583cc157423    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Mar 2021 15:33:50 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 25 Mar 2021 15:33:50 -0400    

Click here for diff

Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located.
  

  
Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.

M doc/xml/reference.xml
M doc/xml/release.xml
M src/build/config/config.yaml
M src/command/archive/push/file.c
M src/command/archive/push/file.h
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M src/command/help/help.auto.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/src/module/command/archivePushTest.c

Improve archive-push command fault tolerance.

commit   : 01b8e2258f6efa5386665c33cb5c267c154376fb    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Mar 2021 12:54:49 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 25 Mar 2021 12:54:49 -0400    

Click here for diff

3b8f0ef missed some cases that could cause archive-push to fail:
  

  
* Checking archive info.
  
* Checking to see if a WAL segment already exists.
  

  
These cases are now handled so archive-push can succeed on any valid repos.

M doc/xml/release.xml
M src/command/archive/push/file.c
M src/command/archive/push/file.h
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/src/module/command/archivePushTest.c

Improve info command fault tolerance.

commit   : 2789d3b6206c4de9299a3bc459e04cdab793abc9    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 25 Mar 2021 12:29:36 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 25 Mar 2021 12:29:36 -0400    

Click here for diff

This improvement reduces the number of errors thrown; these errors will now be reported as a status for the stanza or repo as appropriate. Invalid option configurations are still thrown but all other errors are caught, formatted and reported. This was necessary for multiple repositories so that the command can complete gathering information from each repository and report the results rather than immediately aborting when an error occurs.
  

  
Two new error codes were introduced:
  
6 = requested backup not found
  
99 = other, which is used to indicate an error has occurred that requires more details to be provided
  

  
A new stanza name of "[invalid]" was created for instances where a stanza was not specified and no stanza can be found.
  

  
If there is only one repository configured the error will move up to the stanza level with the standard error formatting of 'error (message)' where the message will be "other" and the details of the error will be listed on the next line(s):
  

  
stanza: stanza1
  
    status: error (other)
  
            [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
  
            CryptoError: cipher header invalid
  
            HINT: is or was the repo encrypted?
  
            FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
  
            HINT: backup.info cannot be opened and is required to perform a backup.
  
            HINT: has a stanza-create been performed?
  
            HINT: use option --stanza if encryption settings are different for the stanza than the global
  
    cipher: aes-256-cbc
  

  
If a backup set is requested but is not found on any repo, a stanza-level status error of 'requested backup not found' is reported when there are no other errors:
  

  
pgbackrest info --stanza=demo --set=bogus
  
stanza: demo
  
    status: error (requested backup not found)
  
    cipher: mixed
  
        repo1: aes-256-cbc
  
        repo2: none
  

  
If there are multiple repositories configured and a single repo is in error but the other repos are ok or have a different error:
  

  
pgbackrest info --stanza=demo --set=20210322-171211F
  
stanza: demo
  
    status: mixed
  
        repo1: error
  
               [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
  
               CryptoError: cipher header invalid
  
               HINT: is or was the repo encrypted?
  
               FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
  
               HINT: backup.info cannot be opened and is required to perform a backup.
  
               HINT: has a stanza-create been performed?
  
               HINT: use option --stanza if encryption settings are different for the stanza than the global
  
        repo2: ok
  
    cipher: mixed
  
        repo1: aes-256-cbc
  
        repo2: none
  

  
    db (current)
  
        wal archive min/max (12): 000000010000000000000001/000000010000000000000003
  

  
        full backup: 20210322-171211F
  
            timestamp start/stop: 2021-03-22 17:12:11 / 2021-03-22 17:12:28
  
            wal start/stop: 000000010000000000000002 / 000000010000000000000002
  
            database size: 23.4MB, database backup size: 23.4MB
  
            repo2: backup set size: 2.8MB, backup size: 2.8MB
  
            database list: postgres (13359)
  

  
Json output will include the repository information and any error information. If no stanzas are found, then [invalid] will be set as the name:
  

  
[
  
   {
  
       "archive":[],
  
       "backup":[],
  
       "cipher":"none",
  
       "db":[],
  
       "name":"[invalid]",
  
       "repo":[
  
           {
  
               "cipher":"none",
  
               "key":1,
  
               "status":{
  
                   "code":99,
  
                   "message":"[PathOpenError] unable to list file info for path '/var/lib/pgbackrest/repo2/backup': [13] Permission denied"
  
               }
  
           }
  
       ],
  
       "status":{
  
           "code":99,
  
           "lock":{"backup":{"held":false}},
  
           "message":"other"
  
           }
  
   }
  
]
  

M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c
M src/command/info/info.c
M test/src/module/command/infoTest.c

Exclude content-length from S3 signed headers.

commit   : 7d7ac0e0eb63a02bbc1dd8bd8fc89c6008c740f8    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Mar 2021 07:07:16 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 25 Mar 2021 07:07:16 -0400    

Click here for diff

The content-length header was being signed since it was the only header that didn't need to be and it seemed simpler just to sign it as well. Also, the S3 documentation encourages signing as many headers as possible to avoid tampering.
  

  
However, some proxies munge this header causing authentication failure, so skip signing content-length.

M doc/xml/release.xml
M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Allow key replacements in TEST_ERROR*() macros.

commit   : 5876048675dceb76b8ddac8757ad47be8b55a817    
  
author   : David Steele <[email protected]>    
date     : Tue, 23 Mar 2021 18:20:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Mar 2021 18:20:26 -0400    

Click here for diff

Allow standard replacements (e.g. {[path]}) in TEST_ERROR*() macros.  
  
Update command/archive-push unit test as an example of usage.  

M test/src/common/harnessTest.h
M test/src/module/command/archivePushTest.c

Fix comment type and phrasing.

commit   : b2440947fb0e585aa515fc6b1edfa0d164606a54    
  
author   : David Steele <[email protected]>    
date     : Thu, 18 Mar 2021 16:29:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 18 Mar 2021 16:29:14 -0400    

Click here for diff

M src/command/archive/push/file.c

Changed unnecessary strNewFmt() to strNew().

commit   : 6800f9e9cd578c61799c8c0dff803fff15e53021    
  
author   : David Steele <[email protected]>    
date     : Thu, 18 Mar 2021 16:26:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 18 Mar 2021 16:26:30 -0400    

Click here for diff

M src/command/archive/get/get.c

Fix comment typo.

commit   : 73e94a4e9c32a3dd32e1a1e763a55eb3ea121e05    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Mar 2021 17:02:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 16 Mar 2021 17:02:03 -0400    

Click here for diff

M src/common/encode.c

Improve protocol handlers.

commit   : 2016fac0d9d5ef9196d7b3058d0b854adf4472ca    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Mar 2021 13:09:34 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 16 Mar 2021 13:09:34 -0400    

Click here for diff

Make protocol handlers have one function per command. This allows the logic of finding the handler to be in ProtocolServer, isolates each command to a function, and removes the need to test the "not found" condition for each handler.

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/archive/get/protocol.h
M src/command/archive/push/protocol.c
M src/command/archive/push/protocol.h
M src/command/archive/push/push.c
M src/command/backup/protocol.c
M src/command/backup/protocol.h
M src/command/local/local.c
M src/command/remote/remote.c
M src/command/restore/protocol.c
M src/command/restore/protocol.h
M src/command/verify/protocol.c
M src/command/verify/protocol.h
M src/config/protocol.c
M src/config/protocol.h
M src/db/protocol.c
M src/db/protocol.h
M src/protocol/helper.c
M src/protocol/server.c
M src/protocol/server.h
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/verifyTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

Fix repo-ls / on S3 repositories.

commit   : b1d945ebb774c25676155af439b0558e8582bbe5    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Mar 2021 12:42:55 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 16 Mar 2021 12:42:55 -0400    

Click here for diff

S3 returns 200 for HEAD / which indicates it is a file but does not return the expected headers which causes an error.
  

  
Rather than fix this for S3, just automatically return / as not existing for any storage that does not support paths.
  

  
Also add some defensive checks to prevent this from generating a segfault if it happens again.

M doc/xml/release.xml
M src/storage/s3/storage.c
M src/storage/storage.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Add back performance tests lost during Github Actions switch.

commit   : 0eb5b22ffa02bb01e8dcf013c7b716c867d4acee    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 Mar 2021 17:24:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 Mar 2021 17:24:58 -0400    

Click here for diff

These tests do not provide code coverage but testing them regularly is important so they work when needed.  

M .github/workflows/test.yml

Include recreated system databases during selective restore.

commit   : 6942ff569dd950a5146faa0bebfee69e241c4179    
  
author   : Stefan Fercot <[email protected]>    
date     : Mon, 15 Mar 2021 17:54:14 +0100    
  
committer: GitHub <[email protected]>    
date     : Mon, 15 Mar 2021 17:54:14 +0100    

Click here for diff

Some standard system databases (e.g. postgres) may be recreated by the user and have an OID that makes them look like user databases.
  

  
Identify the standard three system databases (template0, template1, postgres) and restore them non-zeroed no matter what OID they have.

M doc/xml/release.xml
M src/command/restore/restore.c
M test/expect/mock-all-001.log
M test/src/module/command/restoreTest.c

Explicitly free local processes after restore error unit test.

commit   : c0283eec99325cafe0b4e94750a8efc99976cccf    
  
author   : David Steele <[email protected]>    
date     : Fri, 12 Mar 2021 17:13:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 12 Mar 2021 17:13:34 -0500    

Click here for diff

Local processes are still running after this error and it is best to free them before ending the test.  

M test/src/module/command/restoreTest.c

Pass cipher type directly to backupFileProtocol().

commit   : ec347847e5f1f38a646220bdbeb59b77d1c8e89f    
  
author   : David Steele <[email protected]>    
date     : Fri, 12 Mar 2021 15:19:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 12 Mar 2021 15:19:32 -0500    

Click here for diff

Cipher type was inferred from the presence of cipherSubPass rather than being passed explicitly in order to maintain compatibility with Perl backupFile().  
  
Now that Perl is gone it makes sense to pass it explicitly, as we do elsewhere.  

M src/command/backup/backup.c
M src/command/backup/protocol.c
M test/src/module/command/backupTest.c

Add HRN_STORAGE_TIME() harness macro.

commit   : e07040c2e4a7fd290fa8c26f4297dc7a5b2b9ab1    
  
author   : David Steele <[email protected]>    
date     : Fri, 12 Mar 2021 12:54:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 12 Mar 2021 12:54:34 -0500    

Click here for diff

Makes updating the time of a path/file more streamlined in tests.  
  
Also update all tests where utime() was being used directly.  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/backupTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Add backup delta unit test.

commit   : 3c85a497a618aed8fe7a26463b4e281c513deb50    
  
author   : David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 14:40:14 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 14:40:14 -0500    

Click here for diff

This test was added to take the place of another test, which turned out not to be workable.  
  
Even so, it adds coverages at little cost so it seems worth keeping.  

M test/src/module/command/backupTest.c

Remove extra spaces before macro continuation.

commit   : dc1052f1da0065bf5d8d1b8d50c93d16662dcb51    
  
author   : David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 14:11:21 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 14:11:21 -0500    

Click here for diff

M test/src/module/command/backupTest.c

Update to help.auto.c missed in 9506ffae and 92d12ccb.

commit   : c18abb0af70307b6ec0bfeb08f713e32d8728aaa    
  
author   : David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 12:13:43 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 12:13:43 -0500    

Click here for diff

M src/command/help/help.auto.c

Update selective restore documentation with caveats.

commit   : 92d12ccb9bd0787e30441794e2a2a4d4a4e0fe9c    
  
author   : David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 10:19:50 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 11 Mar 2021 10:19:50 -0500    

Click here for diff

Recovery may error unless --type=immediate is specified. This is because after consistency is reached PostgreSQL will flag zeroed pages as errors even for a full-page write.
  

  
For PostgreSQL ≥ 13 the ignore_invalid_pages setting may be used to ignore invalid pages. In this case it is important to check the logs after recovery to ensure that no invalid pages were reported in the selected databases.

M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml

Log archive copy during backup.

commit   : c862e9654aff198167efeb897957bf93c1afe453    
  
author   : David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 08:22:44 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 11 Mar 2021 08:22:44 -0500    

Click here for diff

Copying can be a fairly expensive operation so it makes sense to log it so the user gets some status during long copy operations.

M doc/xml/release.xml
M src/command/backup/backup.c
M test/src/module/command/backupTest.c

Add compress-type clarification to archive-copy documentation.

commit   : 9506ffae39dd86d2c5454511026f1249097d96a2    
  
author   : David Steele <[email protected]>    
date     : Thu, 11 Mar 2021 07:53:10 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 11 Mar 2021 07:53:10 -0500    

Click here for diff

It is best if the archive-push and backup commands have the same compress-type (e.g. lz4) when using archive-copy. Otherwise, the WAL segments will need to be recompressed with the compress-type used by the backup, which can be fairly expensive depending on how much WAL was generated during the backup.
  

M doc/xml/reference.xml
M doc/xml/release.xml

Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*().

commit   : 28301199ebf647909ff31a0fe6df417bd44b6208    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Mar 2021 18:42:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Mar 2021 18:42:22 -0500    

Click here for diff

When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed.  
  
Update them to make the naming consistent.  

M test/src/common/harnessConfig.c
M test/src/common/harnessDebug.h
M test/src/common/harnessInfo.c
M test/src/common/harnessLog.c
M test/src/common/harnessPack.c
M test/src/common/harnessServer.c
M test/src/common/harnessTest.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/assertOffTest.c
M test/src/module/common/assertOnTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/debugOffTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/encodeTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/execTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/forkTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/logTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/stackTraceTest.c
M test/src/module/common/statTest.c
M test/src/module/common/timeTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeMcvTest.c
M test/src/module/common/typeObjectTest.c
M test/src/module/common/typePackTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/common/typecTest.c
M test/src/module/common/userTest.c
M test/src/module/common/waitTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c
M test/src/test.c

Fix memory leak in backup during archive copy.

commit   : 778adbf19f71ceb90071f01197bf189d59b14df0    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Mar 2021 09:15:35 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Mar 2021 09:15:35 -0500    

Click here for diff

There was already leakage here but when the compression transcoding was added it became a deluge.  
  
There is some argument to be made that the filters should clean themselves up better but a temp mem context makes sense here anyway so do that.  

M doc/xml/release.xml
M src/command/backup/backup.c

Allow stanza-* commands to be run remotely.

commit   : 31c7824a4d84cbf5a0ba0938f4caa546fa67af0a    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 10 Mar 2021 08:10:46 -0500    
  
committer: GitHub <[email protected]>    
date     : Wed, 10 Mar 2021 08:10:46 -0500    

Click here for diff

The stanza-create, stanza-upgrade and stanza-delete were required to be run on the repository host. When there was only one repository allowed this was not a problem.
  

  
However, with the introduction of multiple repository support, this becomes more of a burden to the user, therefore the stanza-create, stanza-upgrade and stanza-delete commands have been improved to allow for them to be run remotely.

M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/config/config.yaml
M src/command/help/help.auto.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/command/stanza/upgrade.c
M src/config/parse.auto.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/stanzaTest.c

Combine multi-repo release notes.

commit   : c4a3dc4e46406634fa0426d80a4fd2f1cd495285    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Mar 2021 07:44:18 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Mar 2021 07:44:18 -0500    

Click here for diff

M doc/xml/release.xml

Print module type (e.g. c or h) in stack trace.

commit   : dde2e2326b8b8d732ebdc6e48764d0fc381062ca    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Mar 2021 17:32:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Mar 2021 17:32:36 -0500    

Click here for diff

Now that there are inline functions in .h files it is important to include the extension so the functions can be found when debugging.  

M src/common/stackTrace.c
M test/src/module/common/errorTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/stackTraceTest.c

Move configuration definition to src/build/config/config.yaml.

commit   : fe4ba455ed7f5c0685ce928d5dec8ca1247736ac    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Mar 2021 16:01:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Mar 2021 16:01:05 -0500    

Click here for diff

Moving to YAML allows the configuration data to be read by C programs.  
  
Also go back to using YAML::XS since it is the only implementation that has proper boolean support.  

M .cirrus.yml
M CONTRIBUTING.md
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M build/lib/pgBackRestBuild/Error/Data.pm
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/xml/contributing.xml
A src/build/config/config.yaml
M test/Vagrantfile
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/test.pl

Multiple repository support.

commit   : 1dbb3bf50b94dcc9d53dfe1e5ad57a17bd6d38e8    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Mar 2021 13:31:13 -0500    
  
committer: GitHub <[email protected]>    
date     : Mon, 8 Mar 2021 13:31:13 -0500    

Click here for diff

Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.
  

  
Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.
  

  
Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.
  

  
The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
  
For single repository configurations there should be no change in behavior.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/index.xml
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/help/help.auto.c
M src/config/load.c
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/helpTest.c
M test/src/module/config/loadTest.c

Update to help.auto.c missed in e7bbdf38.

commit   : 8b87e66448a30b07c12ac00cb2c6ee2f5f29f043    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 17:48:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 17:48:53 -0500    

Click here for diff

M src/command/help/help.auto.c

Remove autoconf cache when configure is built.

commit   : a5f07dff0afdc776a04ca1c4554ad016168f5d63    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 16:27:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 16:27:57 -0500    

Click here for diff

M src/Makefile.in
M test/test.pl

Fix option validity by role and make sections conform to C help.

commit   : e7bbdf38ecae3705908a267c91c7d98adec3f927    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 12:57:07 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 12:57:07 -0500    

Click here for diff

The HTML command reference was showing some options that were not valid because it did not properly understand the new role validity system. Also, the custom section for the new repo option was not being honored.  
  
This is a bit messy because it leads to some duplicated code in help.c but there doesn't seem to be any way to fix that with the Perl data structures as they are.  
  
This code is being migrated to C so it doesn't seem worth messing with it too much with the risk of breaking other things.  

M doc/lib/pgBackRestDoc/Common/DocConfig.pm

GCS support for repository storage.

commit   : 088662d986288843d1bfd118b724091c93df4569    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 12:13:51 -0500    
  
committer: GitHub <[email protected]>    
date     : Fri, 5 Mar 2021 12:13:51 -0500    

Click here for diff

GCS and GCS-compatible object stores can now be used for repository storage.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/index.xml
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/Makefile.in
M src/command/help/help.auto.c
M src/command/help/help.c
M src/command/repo/create.c
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/common/io/http/response.h
M src/config/config.auto.h
M src/config/parse.auto.c
A src/storage/gcs/read.c
A src/storage/gcs/read.h
A src/storage/gcs/storage.c
A src/storage/gcs/storage.h
A src/storage/gcs/storage.intern.h
A src/storage/gcs/write.c
A src/storage/gcs/write.h
M src/storage/helper.c
M test/define.yaml
M test/expect/mock-archive-stop-001.log
M test/expect/mock-stanza-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
A test/lib/pgBackRestTest/Env/Host/HostGcsTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/helpTest.c
A test/src/module/storage/gcsTest.c

Fix .gitignore in doc directory.

commit   : 20bb544e7a321562bf0c9061a91369bb6a12d7c1    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 08:41:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 Mar 2021 08:41:15 -0500    

Click here for diff

doc/output/* was invalid and use output/ to exclude the dir as well as the files in it.  

M doc/.gitignore

Add missing assert.

commit   : 753b60d3d51b89798ce5126f599126c77bbbec03    
  
author   : David Steele <[email protected]>    
date     : Thu, 4 Mar 2021 14:21:30 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 4 Mar 2021 14:21:30 -0500    

Click here for diff

M src/storage/azure/write.c

Make --repo optional for remaining commands except stanza-delete.

commit   : 95063f6812279cd59380fd72e16e67b631582bd0    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 Mar 2021 09:21:06 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 Mar 2021 09:21:06 -0500    

Click here for diff

Some commands (repo-*, verify) still required the --repo option but it makes sense to give them the same treatment as backup and simply use the first repo when one is not specified.  
  
This leaves stanza-delete as the only remaining command that requires --repo. This is by design to enhance safe usage.  

M doc/xml/release.xml
M src/config/load.c
M test/src/module/command/repoTest.c
M test/src/module/command/verifyTest.c

Update to help.auto.c missed in d1aa765a.

commit   : cb26e8c72d74cc36373a8e882a446ef0e37043ed    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 Mar 2021 17:00:08 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 Mar 2021 17:00:08 -0500    

Click here for diff

M src/command/help/help.auto.c

Consolidate less commonly used repository storage options.

commit   : d1aa765a9d5f290738c1de00c5ea99d5f6056ba5    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 Mar 2021 13:51:40 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 2 Mar 2021 13:51:40 -0500    

Click here for diff

The following options are renamed as specified:
  

  
repo1-azure-ca-file -> repo1-storage-ca-file
  
repo1-azure-ca-path -> repo1-storage-ca-path
  
repo1-azure-host -> repo1-storage-host
  
repo1-azure-port -> repo1-storage-port
  
repo1-azure-verify-tls -> repo1-storage-verify-tls
  
repo1-s3-ca-file -> repo1-storage-ca-file
  
repo1-s3-ca-path -> repo1-storage-ca-path
  
repo1-s3-host -> repo1-storage-host
  
repo1-s3-port -> repo1-storage-port
  
repo1-s3-verify-tls -> repo1-storage-verify-tls
  

  
The old option names (e.g. repo1-s3-port) will continue to work for repo1, but repo2, etc. will require the new names.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/help/help.auto.c
M src/command/help/help.c
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/storage/helper.c
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/src/module/command/helpTest.c
M test/src/module/config/parseTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Add HttpUrl object.

commit   : e64999db7744cad55117f582c2cd42fb4b836f6f    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 Mar 2021 13:44:47 -0500    
  
committer: GitHub <[email protected]>    
date     : Mon, 1 Mar 2021 13:44:47 -0500    

Click here for diff

Parse a URL into component parts.

M doc/xml/release.xml
M src/Makefile.in
A src/common/io/http/url.c
A src/common/io/http/url.h
M test/define.yaml
M test/src/module/common/ioHttpTest.c

Add storageInfoLevelType.

commit   : 1d77db31433032041bd1cf5ff318fb3630bd4500    
  
author   : David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 18:02:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 18:02:09 -0500    

Click here for diff

This allows the removal of the callback in the S3/Azure storage drivers that existed only to parse the size/time information.  
  
The extra callback was required because not all callers of storage*ListInternal() want size/time info, so it was wasteful to add it to storage*ListInternal(). Now those callers can request type info only.  

M src/storage/azure/storage.c
M src/storage/info.h
M src/storage/posix/storage.c
M src/storage/s3/storage.c

Make remote storage objects writeable.

commit   : 54c4eb0c10758879b4bf2b96efd2a1523ecccc21    
  
author   : David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 17:42:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 17:42:32 -0500    

Click here for diff

This wasn't exposed before because the remote protocol directly uses the storage driver, which bypasses the writeable checks.  
  
However, the upcoming GCS driver explicitly requests write permissions so remote operations fail when a write is required.  
  
It would be far better if the remote itself was marked as writeable but that will require much more work.  

M src/storage/remote/protocol.c

Fix comments.

commit   : 46922ff2e98a7c1a3c7950c18d2a86653938dcb4    
  
author   : David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 17:31:23 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 17:31:23 -0500    

Click here for diff

M src/storage/helper.c

Suppress implicit-fallthrough warning.

commit   : 0766b7e524f62c00d61d5588ef1491bbcccaf9a0    
  
author   : David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 17:27:48 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 17:27:48 -0500    

Click here for diff

Warning on missing breaks in switch statements works great until it is intended.  
  
Suppressing on a case by case basis varies by compiler and version so is not very practical. Our tests should be sufficient to the task of finding missing breaks.  

M src/build/configure.ac
M src/configure

Make S3/Azure file missing error messages match Posix.

commit   : a1341b4af013a757c60de0f0595e2839f2ac27e6    
  
author   : David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 17:00:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 17:00:41 -0500    

Click here for diff

The S3 driver was missed when the constants were added and then Azure was copied from S3.  

M src/storage/azure/read.c
M src/storage/s3/read.c
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-002.log
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Add missing linefeed.

commit   : 3fb6da6412fd403820ba92e5076cf3b36bda138c    
  
author   : David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 16:01:38 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 28 Feb 2021 16:01:38 -0500    

Click here for diff

M src/storage/posix/read.c

Add write fault-tolerance to archive-push command.

commit   : 3b8f0ef7ae8a81ae969934fff4ad3da299454b24    
  
author   : David Steele <[email protected]>    
date     : Fri, 26 Feb 2021 16:52:59 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 26 Feb 2021 16:52:59 -0500    

Click here for diff

The archive-push command will continue to push even after it gets a write error on one or more repos. The idea is to archive to as many repos as possible even we still need to throw an error to PostgreSQL to prevent it from removing the WAL file.  

M .cirrus.yml
M doc/xml/release.xml
M src/command/archive/push/file.c
M test/src/module/command/archivePushTest.c

Refactor archive-push command warnings to work like archive-get.

commit   : a1280c41e530500e5bbde73a900683f6db207686    
  
author   : David Steele <[email protected]>    
date     : Fri, 26 Feb 2021 15:58:11 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 26 Feb 2021 15:58:11 -0500    

Click here for diff

Warnings are logged individually in the async log rather than all together.  

M doc/xml/release.xml
M src/command/archive/push/file.c
M src/command/archive/push/file.h
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M test/src/module/command/archivePushTest.c

Make --repo optional for backup command.

commit   : 13dc8e68d74fa9e06180e67b7e59994767b4a34d    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 26 Feb 2021 14:49:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 26 Feb 2021 14:49:50 -0500    

Click here for diff

If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.  

M doc/xml/release.xml
M src/command/backup/backup.c
M src/config/load.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/module/command/backupTest.c
M test/src/module/config/loadTest.c

Allow custom config-path default with ./configure --with-configdir.

commit   : 9243962b9504787c6f4af42318af6dc27094136d    
  
author   : Michael Schout <[email protected]>    
date     : Thu, 25 Feb 2021 11:03:44 -0600    
  
committer: GitHub <[email protected]>    
date     : Thu, 25 Feb 2021 11:03:44 -0600    

Click here for diff

Add --with-confdir=DIR option to configure, which can be used to override the default configuration directory of /etc/pgbackrest.
  

  
Probably in the future it would be better to just leverage ${sysconfdir} which is based on prefix, but since previously the config directory was hard coded to /etc/pgbackrest, we retain that default value by not relying on sysconfdir for now.

M doc/xml/release.xml
M src/build/configure.ac
M src/configure

Remove restore default repo from integration tests.

commit   : 0ddc0380ff408f9ed6e8bcd4d328df500681ce77    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 24 Feb 2021 11:32:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 24 Feb 2021 11:32:13 -0500    

Click here for diff

The default is now to scan all repos so update the integration tests to reflect that.  

M doc/xml/release.xml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm

Reduce default file log level for integration tests.

commit   : 8f03c3574b40b589b19a601a48c469da8305fc78    
  
author   : David Steele <[email protected]>    
date     : Wed, 24 Feb 2021 08:27:58 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 24 Feb 2021 08:27:58 -0500    

Click here for diff

The real/all test could fill the ramdisk depending on which vm and pg version were selected.  
  
Debug level should be fine for most purposes and the level can be increased when needed.  

M test/test.pl

Refactor info command repoMin/Max.

commit   : 065b2ff2300a9f06a8c32621fac521451ca942bd    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 23 Feb 2021 16:27:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 16:27:05 -0500    

Click here for diff

M doc/xml/release.xml
M src/command/info/info.c

Enhance restore command multi-repo support.

commit   : 118d9e64fea71c2e049b52802e55f72f5793871b    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 23 Feb 2021 16:17:27 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 16:17:27 -0500    

Click here for diff

The restore command automatically defaults to selecting the latest backup from a single repository. With multiple repositories configured, the restore command will now default to selecting the latest backup from the first repository where backups exist. The order in which the repositories are checked is dictated by the pgbackrest.conf order.  
  
To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.  

M doc/xml/release.xml
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/command/restore/restore.c
M src/config/load.c
M test/expect/mock-all-001.log
M test/src/module/command/restoreTest.c
M test/src/module/config/loadTest.c

Add archive-get command multi-repo support.

commit   : bec3e20b2cb061ef0413df24234ebe246ea53063    
  
author   : David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 15:34:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 15:34:28 -0500    

Click here for diff

Repositories will be searched in order for the requested archive file.  
  
Errors will be reported as warnings as long as a valid copy of the archive file is found.  

M .cirrus.yml
M build/error.yaml
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/archive/get/file.c
M src/command/archive/get/file.h
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/help/help.auto.c
M src/common/error.auto.c
M src/common/error.auto.h
M src/config/load.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/archiveGetTest.c

Expire continues if an error occurs processing a repository.

commit   : e28f6f11e97a80e9c1adf1a6eaeeabbfc991914f    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 23 Feb 2021 12:20:02 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 12:20:02 -0500    

Click here for diff

Errors are logged to the log file rather than thrown. If, after processing all repos, one or more errors occurred, then a single error error will be thrown to indicate there were errors and the log file should be inspected.  
  
Also update log messages to be more consistent with new patterns.  

M build/error.yaml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/expire/expire.c
M src/common/error.auto.c
M src/common/error.auto.h
M test/expect/mock-all-001.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/expireTest.c

Remove unused header files in S3 and Azure storage modules.

commit   : e14036bf57866ea7c2ed4c6fdf7cc204a8f00cb7    
  
author   : David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 07:33:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 07:33:32 -0500    

Click here for diff

M src/storage/azure/read.c
M src/storage/azure/write.c
M src/storage/s3/read.c
M src/storage/s3/write.c

Use list to search for WAL segments to preserve in queueNeed().

commit   : 6fb9de9a48630722df3b398cacb09ae964dc825d    
  
author   : David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 06:35:45 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Feb 2021 06:35:45 -0500    

Click here for diff

The regular expression predates strLstFind() on sorted lists. Using the list is both simpler and faster.  

M src/command/archive/get/get.c

Add base64url encoding.

commit   : 00b60e564eef13f8adba0ab867fcde2c352669d6    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 19:21:06 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 19:21:06 -0500    

Click here for diff

For now only encoding is supported. Decoding is not needed and may never be.  

M src/common/encode.c
M src/common/encode.h
M test/define.yaml
M test/src/module/common/encodeTest.c

Add ASSERT_MSG().

commit   : a1f4fd32a1790e569ccae3598cd9648bb25d13b6    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 19:03:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 19:03:42 -0500    

Click here for diff

Used when execution reaches an invalid location rather than an invalid condition.  

M src/common/assert.h

Do not expose valid/validate functions from encode module.

commit   : f6c3262861252b24e7e0cffd6d67e0c5e3291f3e    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 18:25:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 18:25:50 -0500    

Click here for diff

These functions have never been used externally. Validation is always part of decoding so performing validation separately would be wasteful.  

M src/common/encode.c
M src/common/encode.h
M test/src/module/common/encodeTest.c

Use switch rather than if-else for encoding types.

commit   : edab2a0b89eabc5fd5ba6e795d938670bd5d3105    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 17:57:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 17:57:13 -0500    

Click here for diff

This is more efficient and the error case can be an assert rather than a runtime error.  
  
For extra safety initialize destinationSize to SIZE_MAX to increase the chances of an error if the switch fails.  

M src/common/encode.c
M src/common/encode.h
M test/src/module/common/encodeTest.c

Combine encode module files into a single file.

commit   : abcbe0f9c1804ecb1b93b811cfb0a362c17e1b85    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 17:25:00 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 17:25:00 -0500    

Click here for diff

There is not enough code here to justify multiple files and declaring the functions for each encoding as static allows the compiler to inline where appropriate.  

M src/Makefile.in
M src/common/encode.c
D src/common/encode/base64.c
D src/common/encode/base64.h
M test/define.yaml

Add strNewEncode(), strCatEncode(), and bufNewDecode().

commit   : d4856096587c72daee76f8ad8408db677a16d6dd    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 17:05:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 17:05:15 -0500    

Click here for diff

These constructors wrap encodeToStr() and decodeToBin(), making them convenient and safe by eliminating the need to create intermediate buffers. Encoding/decoding is performed directly into the target String/Buffer. Sizing of the destination buffer is handled by the new functions so it doesn't have to be done at each call site.  

M src/command/stanza/common.c
M src/common/encode.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/string.c
M src/common/type/string.h
M src/info/info.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/s3/storage.c
M test/define.yaml
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Do not lower-case help summaries when first word is an acronym.

commit   : 5b98968605d5ce73918ee84d8bb3ec83b6f21c65    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 10:29:29 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 10:29:29 -0500    

Click here for diff

If the second letter is capital or a digit then the word is likely an acronym so don't lower-case the first letter.  
  
For now only the digit case is checked since there are no summaries with a capital as the second letter.  

M src/command/help/help.c
M test/src/module/command/helpTest.c

Encode path before passing to HttpRequest.

commit   : 66a4ff496afe63305cb1a437bcb52d1efc11edf7    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 09:05:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 09:05:32 -0500    

Click here for diff

GCS requires mixed encoding in the path so encoding inside HttpRequest does not work.  
  
Instead, require the path to be correctly encoded before being passed to HttpRequest.  

M src/common/io/http/request.c
M src/storage/azure/storage.c
M src/storage/s3/storage.c
M test/src/module/common/ioHttpTest.c

Rename uri to path where appropriate in HTTP and storage modules.

commit   : 1b4b3538cc218f66ed1e903c3caeade4b0ab1120    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 08:22:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 08:22:50 -0500    

Click here for diff

The path was originally named uri due to the canonicalized path being called "canonicalized uri" in the S3 authentication documentation. The name got propagated everywhere from there.  
  
This is not correct for general usage, however, so rename to path when describing the path component of an HTTP request.  

M src/common/io/http/request.c
M src/common/io/http/request.h
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/storage.intern.h
M src/storage/azure/write.c
M src/storage/s3/storage.c
M src/storage/s3/storage.intern.h
M test/src/module/common/ioHttpTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Decode JSON \u escaped characters.

commit   : dcb79ab8fb18c63c2d56ecacd9d680b999ea044b    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 07:32:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 07:32:40 -0500    

Click here for diff

ASCII may occasionally be encoded (e.g. &) to prevent ambiguity depending on where the JSON is located.  
  
Only ASCII can be decoded. In general Unicode should not be encoded in JSON.  

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Silence chmod warning of empty path when removing unit test data.

commit   : c4243331de250926328045aa25c60a0c94262fa2    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 07:24:59 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Feb 2021 07:24:59 -0500    

Click here for diff

M test/src/common/harnessTest.c

Fix option warnings breaking async archive-get/archive-push.

commit   : 3837e61a7504b1e211e4b3a70cbe97343c1cf2e6    
  
author   : David Steele <[email protected]>    
date     : Thu, 18 Feb 2021 13:29:09 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 18 Feb 2021 13:29:09 -0500    

Click here for diff

Option warnings will cause the async process to fail because a warning is logged but stdout is closed so the process aborts.
  

  
This bug has existed for quite some time, but it was made worse by abb8ebe because now the async role can have different valid options than the default role. Previously at least a warning would be emitted before the async process died.
  

  
Fix this by only allowing warnings for the default role. Warnings were already suppressed for local and remote roles so the logic already exists.

M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c

Fix tests that ensure log levels are not set for local/remote roles.

commit   : d7befd4189cbfafb8984468fc13318fa087dc812    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Feb 2021 16:28:19 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 16 Feb 2021 16:28:19 -0500    

Click here for diff

These tests were broken because they were being gated by resetLogLevel. So they were not setting the log levels, but not because of the role setting. Because resetLogLevel was being checked last coverage testing indicated that the tests were working.  
  
Fix the resetLogLevel parameter in the tests and move resetLogLevel to be tested first so coverage reporting works as expected. This isn't perfect but it is an improvement.  

M src/config/parse.c
M test/src/module/config/parseTest.c

Fix stack overflow in cipher passphrase generation.

commit   : d29855bd0bf8b957fa8c06445d22e425c443a704    
  
author   : David Steele <[email protected]>    
date     : Fri, 12 Feb 2021 10:08:47 -0500    
  
committer: GitHub <[email protected]>    
date     : Fri, 12 Feb 2021 10:08:47 -0500    

Click here for diff

The destination buffer on the stack was not large enough to contain the zero-terminating character.
  

  
Increase the buffer size and add an assertion to prevent regressions.
  

  
Found on arm64 running musl libc. Other architectures and glibc do not seem to be affected though it is clearly a bug.

M doc/xml/release.xml
M src/command/stanza/common.c

Better exclusions for configure help.

commit   : 920c746adb1f4db5850bc2e5fdfbb5e1b89d313f    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Feb 2021 15:54:37 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Feb 2021 15:54:37 -0500    

Click here for diff

Exclude known unused options rather than trying to include used options. This works better when new options are added.  

M test/test.pl

Set config path in configure script.

commit   : 6a717e032f1309ddc9dc955042b5e2ac2bc7324d    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Feb 2021 14:46:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Feb 2021 14:46:26 -0500    

Click here for diff

This allows the config path to be modified with a parameter to the configure script, though this commit does not do that.  
  
Update the Perl code generator to allow literals so that defaults can be C defines rather than static strings.  

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M src/build.auto.h.in
M src/build/configure.ac
M src/config/parse.auto.c
M src/config/parse.c
M src/configure
M src/version.h

Enhance expire command multi-repo support.

commit   : 3408f1ee2ee3f774699a15de045e0afe3eaf9a26    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 10 Feb 2021 12:03:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Feb 2021 12:03:52 -0500    

Click here for diff

The expire command has been enhanced to expire backups and archives from all configured repositories by default.  
  
In addition, it will accept the --repo option to expire backups and archives only from the specified repository. Using the --repo options the --set option can also be refined further to the specified repo. If --set is provided but the --repo option has not, then all repositories will be searched and retention settings will be applied on each whether the backup set has been found or not.  

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/backup/common.h
M src/command/expire/expire.c
M src/config/load.c
M test/src/module/command/expireTest.c

Use latex sloppypar to fix monospace wrapping in PDF rendering.

commit   : 26cbebbda7b04d0b142e28b41205337717959616    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 Feb 2021 09:10:51 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 Feb 2021 09:10:51 -0500    

Click here for diff

Monospaced identifiers could end up running over if latex was not able to find a place to break the line. Using sloppypar forces breaks so monospaced identifiers don't run over or get broken up.  
  
Also add vspace to admonitions so they have some separation from the prior text.  

M doc/lib/pgBackRestDoc/Common/DocRender.pm
M doc/lib/pgBackRestDoc/Latex/DocLatexSection.pm

Ensure test user has permissions before removing test files.

commit   : c7d7280fa8f2b91cca2517b1bdab4b40603b9bd6    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 16:36:38 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 16:36:38 -0500    

Click here for diff

This allows files to be deleted even when tests have limited the permissions.  

M test/test.pl

Add missing linefeeds.

commit   : b2bba678a0c7de2a8a0e2c949c68c4897e986151    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 16:28:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 16:28:16 -0500    

Click here for diff

M test/src/common/harnessTest.c

Update config.guess and config.sub to latest versions.

commit   : b9f0070d91ef79dab3ee34015418f0d3bd941089    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 13:44:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 13:44:50 -0500    

Click here for diff

M src/build/config.guess
M src/build/config.sub

Begin v2.33 development.

commit   : 00f06065e70916f626e4a5278fed6a1f23898c4c    
  
author   : David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 13:18:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 8 Feb 2021 13:18:22 -0500    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h