pgBackRest 2.34 (upcoming) commit log

Update to help.auto.c missed in e7bbdf38.

commit   : 8b87e66448a30b07c12ac00cb2c6ee2f5f29f043    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 5 Mar 2021 17:48:53 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 5 Mar 2021 17:48:53 -0500    

Click here for diff

M src/command/help/help.auto.c

Remove autoconf cache when configure is built.

commit   : a5f07dff0afdc776a04ca1c4554ad016168f5d63    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 5 Mar 2021 16:27:57 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 5 Mar 2021 16:27:57 -0500    

Click here for diff

M src/Makefile.in
M test/test.pl

Fix option validity by role and make sections conform to C help.

commit   : e7bbdf38ecae3705908a267c91c7d98adec3f927    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 5 Mar 2021 12:57:07 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 5 Mar 2021 12:57:07 -0500    

Click here for diff

The HTML command reference was showing some options that were not valid because it did not properly understand the new role validity system. Also, the custom section for the new repo option was not being honored.  
  
This is a bit messy because it leads to some duplicated code in help.c but there doesn't seem to be any way to fix that with the Perl data structures as they are.  
  
This code is being migrated to C so it doesn't seem worth messing with it too much with the risk of breaking other things.  

M doc/lib/pgBackRestDoc/Common/DocConfig.pm

GCS support for repository storage.

commit   : 088662d986288843d1bfd118b724091c93df4569    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Fri, 5 Mar 2021 12:13:51 -0500    
  
committer: GitHub <noreply@github.com>    
date     : Fri, 5 Mar 2021 12:13:51 -0500    

Click here for diff

GCS and GCS-compatible object stores can now be used for repository storage.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/index.xml
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/Makefile.in
M src/command/help/help.auto.c
M src/command/help/help.c
M src/command/repo/create.c
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/common/io/http/response.h
M src/config/config.auto.h
M src/config/parse.auto.c
A src/storage/gcs/read.c
A src/storage/gcs/read.h
A src/storage/gcs/storage.c
A src/storage/gcs/storage.h
A src/storage/gcs/storage.intern.h
A src/storage/gcs/write.c
A src/storage/gcs/write.h
M src/storage/helper.c
M test/define.yaml
M test/expect/mock-archive-stop-001.log
M test/expect/mock-stanza-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
A test/lib/pgBackRestTest/Env/Host/HostGcsTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/helpTest.c
A test/src/module/storage/gcsTest.c

Fix .gitignore in doc directory.

commit   : 20bb544e7a321562bf0c9061a91369bb6a12d7c1    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 5 Mar 2021 08:41:15 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 5 Mar 2021 08:41:15 -0500    

Click here for diff

doc/output/* was invalid and use output/ to exclude the dir as well as the files in it.  

M doc/.gitignore

Add missing assert.

commit   : 753b60d3d51b89798ce5126f599126c77bbbec03    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 4 Mar 2021 14:21:30 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 4 Mar 2021 14:21:30 -0500    

Click here for diff

M src/storage/azure/write.c

Make --repo optional for remaining commands except stanza-delete.

commit   : 95063f6812279cd59380fd72e16e67b631582bd0    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 3 Mar 2021 09:21:06 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 3 Mar 2021 09:21:06 -0500    

Click here for diff

Some commands (repo-*, verify) still required the --repo option but it makes sense to give them the same treatment as backup and simply use the first repo when one is not specified.  
  
This leaves stanza-delete as the only remaining command that requires --repo. This is by design to enhance safe usage.  

M doc/xml/release.xml
M src/config/load.c
M test/src/module/command/repoTest.c
M test/src/module/command/verifyTest.c

Update to help.auto.c missed in d1aa765a.

commit   : cb26e8c72d74cc36373a8e882a446ef0e37043ed    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 2 Mar 2021 17:00:08 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 2 Mar 2021 17:00:08 -0500    

Click here for diff

M src/command/help/help.auto.c

Consolidate less commonly used repository storage options.

commit   : d1aa765a9d5f290738c1de00c5ea99d5f6056ba5    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Tue, 2 Mar 2021 13:51:40 -0500    
  
committer: GitHub <noreply@github.com>    
date     : Tue, 2 Mar 2021 13:51:40 -0500    

Click here for diff

The following options are renamed as specified:
  

  
repo1-azure-ca-file -> repo1-storage-ca-file
  
repo1-azure-ca-path -> repo1-storage-ca-path
  
repo1-azure-host -> repo1-storage-host
  
repo1-azure-port -> repo1-storage-port
  
repo1-azure-verify-tls -> repo1-storage-verify-tls
  
repo1-s3-ca-file -> repo1-storage-ca-file
  
repo1-s3-ca-path -> repo1-storage-ca-path
  
repo1-s3-host -> repo1-storage-host
  
repo1-s3-port -> repo1-storage-port
  
repo1-s3-verify-tls -> repo1-storage-verify-tls
  

  
The old option names (e.g. repo1-s3-port) will continue to work for repo1, but repo2, etc. will require the new names.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/help/help.auto.c
M src/command/help/help.c
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/storage/helper.c
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/src/module/command/helpTest.c
M test/src/module/config/parseTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Add HttpUrl object.

commit   : e64999db7744cad55117f582c2cd42fb4b836f6f    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Mon, 1 Mar 2021 13:44:47 -0500    
  
committer: GitHub <noreply@github.com>    
date     : Mon, 1 Mar 2021 13:44:47 -0500    

Click here for diff

Parse a URL into component parts.

M doc/xml/release.xml
M src/Makefile.in
A src/common/io/http/url.c
A src/common/io/http/url.h
M test/define.yaml
M test/src/module/common/ioHttpTest.c

Add storageInfoLevelType.

commit   : 1d77db31433032041bd1cf5ff318fb3630bd4500    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 18:02:09 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 18:02:09 -0500    

Click here for diff

This allows the removal of the callback in the S3/Azure storage drivers that existed only to parse the size/time information.  
  
The extra callback was required because not all callers of storage*ListInternal() want size/time info, so it was wasteful to add it to storage*ListInternal(). Now those callers can request type info only.  

M src/storage/azure/storage.c
M src/storage/info.h
M src/storage/posix/storage.c
M src/storage/s3/storage.c

Make remote storage objects writeable.

commit   : 54c4eb0c10758879b4bf2b96efd2a1523ecccc21    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 17:42:32 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 17:42:32 -0500    

Click here for diff

This wasn't exposed before because the remote protocol directly uses the storage driver, which bypasses the writeable checks.  
  
However, the upcoming GCS driver explicitly requests write permissions so remote operations fail when a write is required.  
  
It would be far better if the remote itself was marked as writeable but that will require much more work.  

M src/storage/remote/protocol.c

Fix comments.

commit   : 46922ff2e98a7c1a3c7950c18d2a86653938dcb4    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 17:31:23 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 17:31:23 -0500    

Click here for diff

M src/storage/helper.c

Suppress implicit-fallthrough warning.

commit   : 0766b7e524f62c00d61d5588ef1491bbcccaf9a0    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 17:27:48 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 17:27:48 -0500    

Click here for diff

Warning on missing breaks in switch statements works great until it is intended.  
  
Suppressing on a case by case basis varies by compiler and version so is not very practical. Our tests should be sufficient to the task of finding missing breaks.  

M src/build/configure.ac
M src/configure

Make S3/Azure file missing error messages match Posix.

commit   : a1341b4af013a757c60de0f0595e2839f2ac27e6    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 17:00:41 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 17:00:41 -0500    

Click here for diff

The S3 driver was missed when the constants were added and then Azure was copied from S3.  

M src/storage/azure/read.c
M src/storage/s3/read.c
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-002.log
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Add missing linefeed.

commit   : 3fb6da6412fd403820ba92e5076cf3b36bda138c    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 16:01:38 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 28 Feb 2021 16:01:38 -0500    

Click here for diff

M src/storage/posix/read.c

Add write fault-tolerance to archive-push command.

commit   : 3b8f0ef7ae8a81ae969934fff4ad3da299454b24    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 26 Feb 2021 16:52:59 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Feb 2021 16:52:59 -0500    

Click here for diff

The archive-push command will continue to push even after it gets a write error on one or more repos. The idea is to archive to as many repos as possible even we still need to throw an error to PostgreSQL to prevent it from removing the WAL file.  

M .cirrus.yml
M doc/xml/release.xml
M src/command/archive/push/file.c
M test/src/module/command/archivePushTest.c

Refactor archive-push command warnings to work like archive-get.

commit   : a1280c41e530500e5bbde73a900683f6db207686    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 26 Feb 2021 15:58:11 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Feb 2021 15:58:11 -0500    

Click here for diff

Warnings are logged individually in the async log rather than all together.  

M doc/xml/release.xml
M src/command/archive/push/file.c
M src/command/archive/push/file.h
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M test/src/module/command/archivePushTest.c

Make --repo optional for backup command.

commit   : 13dc8e68d74fa9e06180e67b7e59994767b4a34d    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Fri, 26 Feb 2021 14:49:50 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Feb 2021 14:49:50 -0500    

Click here for diff

If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.  

M doc/xml/release.xml
M src/command/backup/backup.c
M src/config/load.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/module/command/backupTest.c
M test/src/module/config/loadTest.c

Allow custom config-path default with ./configure --with-configdir.

commit   : 9243962b9504787c6f4af42318af6dc27094136d    
  
author   : Michael Schout <mschout@gkg.net>    
date     : Thu, 25 Feb 2021 11:03:44 -0600    
  
committer: GitHub <noreply@github.com>    
date     : Thu, 25 Feb 2021 11:03:44 -0600    

Click here for diff

Add --with-confdir=DIR option to configure, which can be used to override the default configuration directory of /etc/pgbackrest.
  

  
Probably in the future it would be better to just leverage ${sysconfdir} which is based on prefix, but since previously the config directory was hard coded to /etc/pgbackrest, we retain that default value by not relying on sysconfdir for now.

M doc/xml/release.xml
M src/build/configure.ac
M src/configure

Remove restore default repo from integration tests.

commit   : 0ddc0380ff408f9ed6e8bcd4d328df500681ce77    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 24 Feb 2021 11:32:13 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 24 Feb 2021 11:32:13 -0500    

Click here for diff

The default is now to scan all repos so update the integration tests to reflect that.  

M doc/xml/release.xml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm

Reduce default file log level for integration tests.

commit   : 8f03c3574b40b589b19a601a48c469da8305fc78    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 24 Feb 2021 08:27:58 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 24 Feb 2021 08:27:58 -0500    

Click here for diff

The real/all test could fill the ramdisk depending on which vm and pg version were selected.  
  
Debug level should be fine for most purposes and the level can be increased when needed.  

M test/test.pl

Refactor info command repoMin/Max.

commit   : 065b2ff2300a9f06a8c32621fac521451ca942bd    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Tue, 23 Feb 2021 16:27:05 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 16:27:05 -0500    

Click here for diff

M doc/xml/release.xml
M src/command/info/info.c

Enhance restore command multi-repo support.

commit   : 118d9e64fea71c2e049b52802e55f72f5793871b    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Tue, 23 Feb 2021 16:17:27 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 16:17:27 -0500    

Click here for diff

The restore command automatically defaults to selecting the latest backup from a single repository. With multiple repositories configured, the restore command will now default to selecting the latest backup from the first repository where backups exist. The order in which the repositories are checked is dictated by the pgbackrest.conf order.  
  
To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.  

M doc/xml/release.xml
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/command/restore/restore.c
M src/config/load.c
M test/expect/mock-all-001.log
M test/src/module/command/restoreTest.c
M test/src/module/config/loadTest.c

Add archive-get command multi-repo support.

commit   : bec3e20b2cb061ef0413df24234ebe246ea53063    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 15:34:28 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 15:34:28 -0500    

Click here for diff

Repositories will be searched in order for the requested archive file.  
  
Errors will be reported as warnings as long as a valid copy of the archive file is found.  

M .cirrus.yml
M build/error.yaml
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/archive/get/file.c
M src/command/archive/get/file.h
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/help/help.auto.c
M src/common/error.auto.c
M src/common/error.auto.h
M src/config/load.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/archiveGetTest.c

Expire continues if an error occurs processing a repository.

commit   : e28f6f11e97a80e9c1adf1a6eaeeabbfc991914f    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Tue, 23 Feb 2021 12:20:02 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 12:20:02 -0500    

Click here for diff

Errors are logged to the log file rather than thrown. If, after processing all repos, one or more errors occurred, then a single error error will be thrown to indicate there were errors and the log file should be inspected.  
  
Also update log messages to be more consistent with new patterns.  

M build/error.yaml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/expire/expire.c
M src/common/error.auto.c
M src/common/error.auto.h
M test/expect/mock-all-001.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/expireTest.c

Remove unused header files in S3 and Azure storage modules.

commit   : e14036bf57866ea7c2ed4c6fdf7cc204a8f00cb7    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 07:33:32 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 07:33:32 -0500    

Click here for diff

M src/storage/azure/read.c
M src/storage/azure/write.c
M src/storage/s3/read.c
M src/storage/s3/write.c

Use list to search for WAL segments to preserve in queueNeed().

commit   : 6fb9de9a48630722df3b398cacb09ae964dc825d    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 06:35:45 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Feb 2021 06:35:45 -0500    

Click here for diff

The regular expression predates strLstFind() on sorted lists. Using the list is both simpler and faster.  

M src/command/archive/get/get.c

Add base64url encoding.

commit   : 00b60e564eef13f8adba0ab867fcde2c352669d6    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 19:21:06 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 19:21:06 -0500    

Click here for diff

For now only encoding is supported. Decoding is not needed and may never be.  

M src/common/encode.c
M src/common/encode.h
M test/define.yaml
M test/src/module/common/encodeTest.c

Add ASSERT_MSG().

commit   : a1f4fd32a1790e569ccae3598cd9648bb25d13b6    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 19:03:42 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 19:03:42 -0500    

Click here for diff

Used when execution reaches an invalid location rather than an invalid condition.  

M src/common/assert.h

Do not expose valid/validate functions from encode module.

commit   : f6c3262861252b24e7e0cffd6d67e0c5e3291f3e    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 18:25:50 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 18:25:50 -0500    

Click here for diff

These functions have never been used externally. Validation is always part of decoding so performing validation separately would be wasteful.  

M src/common/encode.c
M src/common/encode.h
M test/src/module/common/encodeTest.c

Use switch rather than if-else for encoding types.

commit   : edab2a0b89eabc5fd5ba6e795d938670bd5d3105    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 17:57:13 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 17:57:13 -0500    

Click here for diff

This is more efficient and the error case can be an assert rather than a runtime error.  
  
For extra safety initialize destinationSize to SIZE_MAX to increase the chances of an error if the switch fails.  

M src/common/encode.c
M src/common/encode.h
M test/src/module/common/encodeTest.c

Combine encode module files into a single file.

commit   : abcbe0f9c1804ecb1b93b811cfb0a362c17e1b85    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 17:25:00 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 17:25:00 -0500    

Click here for diff

There is not enough code here to justify multiple files and declaring the functions for each encoding as static allows the compiler to inline where appropriate.  

M src/Makefile.in
M src/common/encode.c
D src/common/encode/base64.c
D src/common/encode/base64.h
M test/define.yaml

Add strNewEncode(), strCatEncode(), and bufNewDecode().

commit   : d4856096587c72daee76f8ad8408db677a16d6dd    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 17:05:15 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 17:05:15 -0500    

Click here for diff

These constructors wrap encodeToStr() and decodeToBin(), making them convenient and safe by eliminating the need to create intermediate buffers. Encoding/decoding is performed directly into the target String/Buffer. Sizing of the destination buffer is handled by the new functions so it doesn't have to be done at each call site.  

M src/command/stanza/common.c
M src/common/encode.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/string.c
M src/common/type/string.h
M src/info/info.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/s3/storage.c
M test/define.yaml
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Do not lower-case help summaries when first word is an acronym.

commit   : 5b98968605d5ce73918ee84d8bb3ec83b6f21c65    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 10:29:29 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 10:29:29 -0500    

Click here for diff

If the second letter is capital or a digit then the word is likely an acronym so don't lower-case the first letter.  
  
For now only the digit case is checked since there are no summaries with a capital as the second letter.  

M src/command/help/help.c
M test/src/module/command/helpTest.c

Encode path before passing to HttpRequest.

commit   : 66a4ff496afe63305cb1a437bcb52d1efc11edf7    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 09:05:32 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 09:05:32 -0500    

Click here for diff

GCS requires mixed encoding in the path so encoding inside HttpRequest does not work.  
  
Instead, require the path to be correctly encoded before being passed to HttpRequest.  

M src/common/io/http/request.c
M src/storage/azure/storage.c
M src/storage/s3/storage.c
M test/src/module/common/ioHttpTest.c

Rename uri to path where appropriate in HTTP and storage modules.

commit   : 1b4b3538cc218f66ed1e903c3caeade4b0ab1120    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 08:22:50 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 08:22:50 -0500    

Click here for diff

The path was originally named uri due to the canonicalized path being called "canonicalized uri" in the S3 authentication documentation. The name got propagated everywhere from there.  
  
This is not correct for general usage, however, so rename to path when describing the path component of an HTTP request.  

M src/common/io/http/request.c
M src/common/io/http/request.h
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/storage.intern.h
M src/storage/azure/write.c
M src/storage/s3/storage.c
M src/storage/s3/storage.intern.h
M test/src/module/common/ioHttpTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Decode JSON \u escaped characters.

commit   : dcb79ab8fb18c63c2d56ecacd9d680b999ea044b    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 07:32:40 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 07:32:40 -0500    

Click here for diff

ASCII may occasionally be encoded (e.g. &) to prevent ambiguity depending on where the JSON is located.  
  
Only ASCII can be decoded. In general Unicode should not be encoded in JSON.  

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Silence chmod warning of empty path when removing unit test data.

commit   : c4243331de250926328045aa25c60a0c94262fa2    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 07:24:59 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 19 Feb 2021 07:24:59 -0500    

Click here for diff

M test/src/common/harnessTest.c

Fix option warnings breaking async archive-get/archive-push.

commit   : 3837e61a7504b1e211e4b3a70cbe97343c1cf2e6    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Thu, 18 Feb 2021 13:29:09 -0500    
  
committer: GitHub <noreply@github.com>    
date     : Thu, 18 Feb 2021 13:29:09 -0500    

Click here for diff

Option warnings will cause the async process to fail because a warning is logged but stdout is closed so the process aborts.
  

  
This bug has existed for quite some time, but it was made worse by abb8ebe because now the async role can have different valid options than the default role. Previously at least a warning would be emitted before the async process died.
  

  
Fix this by only allowing warnings for the default role. Warnings were already suppressed for local and remote roles so the logic already exists.

M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c

Fix tests that ensure log levels are not set for local/remote roles.

commit   : d7befd4189cbfafb8984468fc13318fa087dc812    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 16 Feb 2021 16:28:19 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 16 Feb 2021 16:28:19 -0500    

Click here for diff

These tests were broken because they were being gated by resetLogLevel. So they were not setting the log levels, but not because of the role setting. Because resetLogLevel was being checked last coverage testing indicated that the tests were working.  
  
Fix the resetLogLevel parameter in the tests and move resetLogLevel to be tested first so coverage reporting works as expected. This isn't perfect but it is an improvement.  

M src/config/parse.c
M test/src/module/config/parseTest.c

Fix stack overflow in cipher passphrase generation.

commit   : d29855bd0bf8b957fa8c06445d22e425c443a704    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Fri, 12 Feb 2021 10:08:47 -0500    
  
committer: GitHub <noreply@github.com>    
date     : Fri, 12 Feb 2021 10:08:47 -0500    

Click here for diff

The destination buffer on the stack was not large enough to contain the zero-terminating character.
  

  
Increase the buffer size and add an assertion to prevent regressions.
  

  
Found on arm64 running musl libc. Other architectures and glibc do not seem to be affected though it is clearly a bug.

M doc/xml/release.xml
M src/command/stanza/common.c

Better exclusions for configure help.

commit   : 920c746adb1f4db5850bc2e5fdfbb5e1b89d313f    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 10 Feb 2021 15:54:37 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 10 Feb 2021 15:54:37 -0500    

Click here for diff

Exclude known unused options rather than trying to include used options. This works better when new options are added.  

M test/test.pl

Set config path in configure script.

commit   : 6a717e032f1309ddc9dc955042b5e2ac2bc7324d    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 10 Feb 2021 14:46:26 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 10 Feb 2021 14:46:26 -0500    

Click here for diff

This allows the config path to be modified with a parameter to the configure script, though this commit does not do that.  
  
Update the Perl code generator to allow literals so that defaults can be C defines rather than static strings.  

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M src/build.auto.h.in
M src/build/configure.ac
M src/config/parse.auto.c
M src/config/parse.c
M src/configure
M src/version.h

Enhance expire command multi-repo support.

commit   : 3408f1ee2ee3f774699a15de045e0afe3eaf9a26    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 10 Feb 2021 12:03:52 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 10 Feb 2021 12:03:52 -0500    

Click here for diff

The expire command has been enhanced to expire backups and archives from all configured repositories by default.  
  
In addition, it will accept the --repo option to expire backups and archives only from the specified repository. Using the --repo options the --set option can also be refined further to the specified repo. If --set is provided but the --repo option has not, then all repositories will be searched and retention settings will be applied on each whether the backup set has been found or not.  

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/backup/common.h
M src/command/expire/expire.c
M src/config/load.c
M test/src/module/command/expireTest.c

Use latex sloppypar to fix monospace wrapping in PDF rendering.

commit   : 26cbebbda7b04d0b142e28b41205337717959616    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 10 Feb 2021 09:10:51 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 10 Feb 2021 09:10:51 -0500    

Click here for diff

Monospaced identifiers could end up running over if latex was not able to find a place to break the line. Using sloppypar forces breaks so monospaced identifiers don't run over or get broken up.  
  
Also add vspace to admonitions so they have some separation from the prior text.  

M doc/lib/pgBackRestDoc/Common/DocRender.pm
M doc/lib/pgBackRestDoc/Latex/DocLatexSection.pm

Ensure test user has permissions before removing test files.

commit   : c7d7280fa8f2b91cca2517b1bdab4b40603b9bd6    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 8 Feb 2021 16:36:38 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 8 Feb 2021 16:36:38 -0500    

Click here for diff

This allows files to be deleted even when tests have limited the permissions.  

M test/test.pl

Add missing linefeeds.

commit   : b2bba678a0c7de2a8a0e2c949c68c4897e986151    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 8 Feb 2021 16:28:16 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 8 Feb 2021 16:28:16 -0500    

Click here for diff

M test/src/common/harnessTest.c

Update config.guess and config.sub to latest versions.

commit   : b9f0070d91ef79dab3ee34015418f0d3bd941089    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 8 Feb 2021 13:44:50 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 8 Feb 2021 13:44:50 -0500    

Click here for diff

M src/build/config.guess
M src/build/config.sub

Begin v2.33 development.

commit   : 00f06065e70916f626e4a5278fed6a1f23898c4c    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 8 Feb 2021 13:18:22 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 8 Feb 2021 13:18:22 -0500    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h