pgBackRest 2.39 commit log

v2.38: Minor Bug Fixes and Improvements

commit   : 5249b89a2e94fdbb45461faaf2e1b0304b986ac7    
  
author   : David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 10:30:59 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 6 Mar 2022 10:30:59 -0600    

Click here for diff

IMPORTANT NOTE: Repository size reported by the info command is now entirely based on what pgBackRest has written to storage. Previously, in certain cases, pgBackRest could detect if additional compression was being applied by the storage but this is no longer supported.  
  
Bug Fixes:  
  
* Retry errors in S3 batch file delete. (Reviewed by Reid Thompson. Reported by Alex Richman.)  
* Allow case-insensitive matching of HTTP connection header values. (Reviewed by Reid Thompson. Reported by Rémi Vidier.)  
  
Features:  
  
* Add support for AWS S3 server-side encryption using KMS. (Contributed by Christoph Berg. Reviewed by David Steele, Tharindu Amila.)  
* Add archive-missing-retry option. (Reviewed by Stefan Fercot.)  
* Add backup type filter to info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)  
  
Improvements:  
  
* Retry on page validation failure during backup. (Reviewed by Stephen Frost, David Christensen.)  
* Handle TLS servers that do not close connections gracefully. (Reviewed by Rémi Vidier, David Christensen, Stephen Frost.)  
* Add backup LSNs to info command output. (Contributed by Stefan Fercot. Reviewed by David Steele.)  
* Automatically strip trailing slashes for repo-ls paths. (Contributed by David Christensen. Reviewed by David Steele.)  
* Do not retry fatal errors. (Reviewed by Reid Thompson.)  
* Remove support for PostgreSQL 8.3/8.4. (Reviewed by Reid Thompson, Stefan Fercot.)  
* Remove logic that tried to determine additional file system compression. (Reviewed by Reid Thompson, Stefan Fercot.)  
  
Documentation Bug Fixes:  
  
* Move repo options in TLS documentation to the global section. (Reported by Anton Kurochkin.)  
* Remove unused backup-standby option from stanza commands. (Reported by Stefan Fercot.)  
* Fix typos in help and release notes. (Fixed by Daniel Gustafsson. Reviewed by David Steele.)  
  
Documentation Improvements:  
  
* Add aliveness check to systemd service configuration. (Suggested by Yogesh Sharma.)  
* Add FAQ explaining WAL archive suffix. (Contributed by Stefan Fercot. Reviewed by David Steele.)  
* Note that replications slots are not restored. (Contributed by Reid Thompson. Reviewed by David Steele, Stefan Fercot. Suggested by Christophe Courtois.)  

M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Handle TLS servers that do not close connections gracefully.

commit   : 59a5373cf85e4ad0057b592bfe855495384dfcbd    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Mar 2022 11:38:52 -0600    
  
committer: GitHub <[email protected]>    
date     : Wed, 2 Mar 2022 11:38:52 -0600    

Click here for diff

Some TLS server implementations will simply close the socket rather than correctly closing the TLS connection. This causes problems when connection: close is specified with no content-length or chunked encoding and we are forced to read to EOF. It is hard to know if this is a real EOF or a network error.
  

  
In cases where we can parse the content and (hopefully) ensure it is correct, allow the closed socket to serve as EOF. This is not ideal, but the change in 8e1807c means that currently working servers with this issue will stop working after 2.35 is installed, which seems too risky.

M doc/xml/release.xml
M src/command/server/ping.c
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/common/io/http/response.c
M src/common/io/http/session.c
M src/common/io/http/session.h
M src/common/io/session.h
M src/common/io/session.intern.h
M src/common/io/socket/session.c
M src/common/io/tls/session.c
M src/protocol/helper.c
M test/src/common/harnessServer.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/protocol/protocolTest.c

Add aliveness check to systemd service configuration.

commit   : f1bdf3e04b07dc9a2eaa1fdbe5278cc18371e868    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Mar 2022 10:59:06 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Mar 2022 10:59:06 -0600    

Click here for diff

If the pgbackrest service service fails to start then the user will get an error. However, retries will continue in the background.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Revert PGDG yum repo workaround for aarch64.

commit   : a66ec8d549b8f1f7b38682ad90281f1fcbb2748a    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Mar 2022 10:22:15 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Mar 2022 10:22:15 -0600    

Click here for diff

da0f3a855 used a workaround to get the documentation building on aarch64 but recent changes to the PGDG yum repo have broken this workaround. Installing the regular way still doesn't work, either.  
  
Reverting for now to get the CI pipeline working again.  

M doc/xml/user-guide.xml

Use vagrant user in the Docker container.

commit   : fb5051fde74bcfde85fa3b9a452a1df693e79db9    
  
author   : David Steele <[email protected]>    
date     : Sat, 26 Feb 2022 13:50:30 -0600    
  
committer: David Steele <[email protected]>    
date     : Sat, 26 Feb 2022 13:50:30 -0600    

Click here for diff

This is a bit of legacy from the current Vagrant environment used to do the release, but since it is not as easy to change the user in Vagrant, just make the Docker environment conform.  
  
This allows documentation to be built in a Vagrant environment (or any environment with the same user name) and to be deployed in a Docker environment.  

M test/Dockerfile

Suppress errors when there is stderr output for Docker builds.

commit   : f716e98ad0e9eda94da973211e5f54e88c2c1aed    
  
author   : David Steele <[email protected]>    
date     : Sat, 26 Feb 2022 11:31:52 -0600    
  
committer: David Steele <[email protected]>    
date     : Sat, 26 Feb 2022 11:31:52 -0600    

Click here for diff

Docker outputs build info to stderr even when the build is successful. This seems to be especially true on Mac M1.  
  
ContainerTest.pm already does this suppression so add it the other places where containers are built.  

M doc/doc.pl
M doc/lib/pgBackRestDoc/Common/DocExecute.pm

Move repo options in TLS documentation to the global section.

commit   : 53de3e3aeb71f8209f56167941df35bbfcb7cdc9    
  
author   : David Steele <[email protected]>    
date     : Sat, 26 Feb 2022 11:17:05 -0600    
  
committer: David Steele <[email protected]>    
date     : Sat, 26 Feb 2022 11:17:05 -0600    

Click here for diff

These options were mistakenly added to the stanza section, which works in certain cases, but is not best practice.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Add FAQ explaining WAL archive suffix.

commit   : 98d525dba43ba259ff7989746b8e5be3382c189d    
  
author   : Stefan Fercot <[email protected]>    
date     : Fri, 25 Feb 2022 21:03:09 +0100    
  
committer: GitHub <[email protected]>    
date     : Fri, 25 Feb 2022 21:03:09 +0100    

Click here for diff

M doc/xml/faq.xml
M doc/xml/release.xml

Allow case-insensitive matching of HTTP connection header values.

commit   : b33cabe08c33e29eb8aef65dd46b2c1f3cf2d186    
  
author   : David Steele <[email protected]>    
date     : Fri, 25 Feb 2022 10:51:40 -0600    
  
committer: GitHub <[email protected]>    
date     : Fri, 25 Feb 2022 10:51:40 -0600    

Click here for diff

The specification allows values for the connection header to be case-insensitive. See https://www.rfc-editor.org/rfc/rfc7230#section-6.1.

M doc/xml/release.xml
M src/common/io/http/response.c
M test/src/module/common/ioHttpTest.c

Automatically strip trailing slashes for repo-ls paths.

commit   : 632071232344a1160ef932d6e0d65eb3149ed226    
  
author   : David Christensen <[email protected]>    
date     : Wed, 23 Feb 2022 13:53:02 -0600    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Feb 2022 13:53:02 -0600    

Click here for diff

Trailing slashes in at least some of the repository storage types were preventing repo-ls from displaying any content (presumably due to storage-specific behavior).
  

  
Since the path with the slash should be equivalent to the path without the slash, just remove it if provided by the user.

M doc/xml/release.xml
M src/command/repo/ls.c
M test/src/module/command/repoTest.c

Improve validation of zero pages.

commit   : 53f1b2520434221c01caf92e052403a28c455c1b    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 Feb 2022 13:17:14 -0600    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Feb 2022 13:17:14 -0600    

Click here for diff

Checking that pd_upper == 0 is not enough since this field may be corrupted. Still use pd_upper as a quick check, but when it is zero proceed to check the rest of the page to ensure it is also all zeroes.

M doc/xml/release.xml
M src/command/backup/pageChecksum.c
M test/src/module/command/backupTest.c

Retry on page checksum validation failure during backup.

commit   : 9eec98c61302121134d2067326dbd2cd0f2f0b9c    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 Feb 2022 12:05:53 -0600    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Feb 2022 12:05:53 -0600    

Click here for diff

Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL.
  

  
Also remove tests made redundant by the test merge in b4897077.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/pageChecksum.c
M src/command/backup/pageChecksum.h
M src/command/backup/protocol.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/command/backupTest.c

Fix release note attribution in cacfdd94 and f798458e.

commit   : 5c332a0b100d5a5180b3d95bd90d4d7b521b42db    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 Feb 2022 11:23:32 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 23 Feb 2022 11:23:32 -0600    

Click here for diff

M doc/xml/release.xml

Disconnect help.auto.c from build-help in Makefile.

commit   : f798458e1d18fc8a866b49a4cfa0cb73536cbd41    
  
author   : Daniel Gustafsson <[email protected]>    
date     : Wed, 23 Feb 2022 17:29:17 +0100    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Feb 2022 17:29:17 +0100    

Click here for diff

When there was an issue with the system library path during building, the build-help rule would fail during executing ./build-help with the effect that main.c wouldn't build.
  

  
Break out help.auto.c generation from the build-help stage to allow it to be re-executed when the library path has been corrected.

M doc/xml/release.xml
M src/Makefile.in

Add XML to invalid XML error message.

commit   : 67bdf07e6969f8160347e895454bf729c9165884    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 Feb 2022 10:26:39 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 23 Feb 2022 10:26:39 -0600    

Click here for diff

There have been cases where pgBackRest has failed on invalid XML but it is not possible to determine what was wrong with the XML.  
  
This will only work for XML up to about 8KiB (which is the error message limit) but it should work in most cases.  

M src/common/type/xml.c
M test/src/module/common/typeXmlTest.c

Add archive-missing-retry option.

commit   : 10038db9c9f2b4753c3821e0b246fa8626527184    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 Feb 2022 09:14:27 -0600    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Feb 2022 09:14:27 -0600    

Click here for diff

Retry a WAL segment that was previously reported as missing by the archive-get command. This prevents notifications in the spool path from a prior restore from being used and possibly causing a recovery failure if consistency has not been reached.
  

  
Disabling this option allows PostgreSQL to more reliably recognize when the end of the WAL in the archive has been reached, which permits it to switch over to streaming from the primary. With retries enabled, a steady stream of WAL being archived will cause PostgreSQL to continue getting WAL from the archive rather than switch to streaming.
  

  
When disabling this option it is important to ensure that the spool path for the stanza is empty. The restore command does this automatically if the spool path is configured at restore time. Otherwise, it is up to the user to ensure the spool path is empty.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/archive/get/get.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/src/module/command/archiveGetTest.c

Fix typos in help and release notes.

commit   : cacfdd94d74307285b1b590ec0977d7db3142d0d    
  
author   : Daniel Gustafsson <[email protected]>    
date     : Wed, 23 Feb 2022 15:01:38 +0100    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Feb 2022 15:01:38 +0100    

Click here for diff

M doc/xml/release.xml
M src/build/help/help.xml

Most recent news for postgresql.org.

commit   : e2eb3ebacce18e9c0c2f74457a646142a0eace3d    
  
author   : David Steele <[email protected]>    
date     : Sun, 20 Feb 2022 17:45:41 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 20 Feb 2022 17:45:41 -0600    

Click here for diff

M doc/NEWS.md

Pass file by reference in manifestFileAdd().

commit   : e6e1122dbcf5e667d683295f7e7e45de4bbf56bd    
  
author   : David Steele <[email protected]>    
date     : Sun, 20 Feb 2022 16:45:07 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 20 Feb 2022 16:45:07 -0600    

Click here for diff

Coverity complained that this pass by value was inefficient:  
  
CID 376402:  Performance inefficiencies  (PASS_BY_VALUE)  
Passing parameter file of type "ManifestFile" (size 136 bytes) by value.  
  
This was completely intentional since it gives us a copy of the struct that we can change without bothering the caller. However, updating fields is fine and may benefit the caller at some future data, and in any case does no harm now.  
  
And as usual it is easier not to fight with Coverity.  

M doc/xml/release.xml
M src/command/backup/backup.c
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/info/manifestTest.c

Move command/backup-common tests in the command/backup module.

commit   : b4897077937ee4571ba719276a44d5db0a75510e    
  
author   : David Steele <[email protected]>    
date     : Fri, 18 Feb 2022 17:50:05 -0600    
  
committer: David Steele <[email protected]>    
date     : Fri, 18 Feb 2022 17:50:05 -0600    

Click here for diff

As much as possible it is better to get coverage with more realistic tests. Merging these modules will allow the page checksum code to be covered with real backups.  

M test/define.yaml
D test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c

Limit files that can be bundled.

commit   : efc09db7b9ece6e7b7f92538d56d6ab7b9798f8f    
  
author   : David Steele <[email protected]>    
date     : Thu, 17 Feb 2022 07:25:12 -0600    
  
committer: GitHub <[email protected]>    
date     : Thu, 17 Feb 2022 07:25:12 -0600    

Click here for diff

Limit which files can be added to bundles, which allows resume to work reasonably well. On resume, the bundles are removed and any remaining file is eligible to be to be resumed.
  

  
Also reduce the bundle-size default to 20MiB. This is pretty arbitrary, but a smaller default seems better.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/backup/backup.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/src/module/command/backupTest.c

Bundle files in the repository during backup.

commit   : 34d649579eb3bd1530aa99f0ed1879e7d3125424    
  
author   : David Steele <[email protected]>    
date     : Mon, 14 Feb 2022 13:24:14 -0600    
  
committer: GitHub <[email protected]>    
date     : Mon, 14 Feb 2022 13:24:14 -0600    

Click here for diff

Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest.
  

  
Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented.
  

  
Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups.
  

  
Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example.
  

  
Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling.
  

  
One caveat: resume is currently not supported when bundle is enabled.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/protocol.c
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/command/restore/restore.c
M src/command/verify/file.c
M src/command/verify/file.h
M src/command/verify/protocol.c
M src/command/verify/verify.c
M src/config/config.auto.h
M src/config/parse.auto.c
M src/info/manifest.c
M src/info/manifest.h
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/verifyTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/typeTest.c

Do not retry fatal errors.

commit   : 8046f0630715015d5adcced570c54e9a183c0e73    
  
author   : David Steele <[email protected]>    
date     : Mon, 14 Feb 2022 11:07:02 -0600    
  
committer: GitHub <[email protected]>    
date     : Mon, 14 Feb 2022 11:07:02 -0600    

Click here for diff

There is some evidence that retrying fatal errors, especially out of memory errors, may cause lockups. It makes sense to report fatal errors as quickly as possible and bypass retries. This may or not fix the lockup issue but it is worth doing either way.
  

  
For now, the only fatal errors will be AssertError and MemoryError.

M doc/xml/release.xml
M src/build/error/error.yaml
M src/build/error/parse.c
M src/build/error/parse.h
M src/build/error/render.c
M src/command/archive/push/push.c
M src/common/error.auto.c
M src/common/error.c
M src/common/error.h
M src/main.c
M src/protocol/server.c
M test/src/common/harnessTest.h
M test/src/module/build/errorTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/exitTest.c
M test/src/module/protocol/protocolTest.c
M test/src/test.c

Use normal error for protocol module error retry test.

commit   : 8d0cce66f83abc1ff02a985ec3f0d7e285c09d72    
  
author   : David Steele <[email protected]>    
date     : Sun, 13 Feb 2022 15:19:31 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 13 Feb 2022 15:19:31 -0600    

Click here for diff

Asserts will not be retried in a future commit, so adjust this test now to use non-assert errors.  

M doc/xml/release.xml
M test/src/module/protocol/protocolTest.c

Improve protocol module error test for protocolClientFree().

commit   : 8573a2df14d0bfd5578333b68147cc12c93c6f96    
  
author   : David Steele <[email protected]>    
date     : Sun, 13 Feb 2022 15:11:59 -0600    
  
committer: David Steele <[email protected]>    
date     : Sun, 13 Feb 2022 15:11:59 -0600    

Click here for diff

Using an assert here was never ideal and won't work once we start handling fatal errors differently.  

M doc/xml/release.xml
M test/src/module/protocol/protocolTest.c

Retry errors in S3 batch file delete.

commit   : 551e5bc6f67c91cf1e11da0cefe24f9650353adc    
  
author   : David Steele <[email protected]>    
date     : Fri, 11 Feb 2022 08:11:39 -0600    
  
committer: GitHub <[email protected]>    
date     : Fri, 11 Feb 2022 08:11:39 -0600    

Click here for diff

If the entire batch failed it would be retried, but individual file errors were not retried. This could cause pgBackRest to terminate during expiration or when removing an unresumable backup.
  

  
Rather than retry the entire batch, delete the errored files individually to take advantage of the HTTP retry rather than adding a new retry loop. These errors seem rare enough that it should not be a performance issue.

M doc/xml/release.xml
M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Add backup type filter to info command.

commit   : b26097f8d8a7e2af6e9a21576994cbb47110c6f4    
  
author   : Stefan Fercot <[email protected]>    
date     : Wed, 9 Feb 2022 17:18:39 +0100    
  
committer: GitHub <[email protected]>    
date     : Wed, 9 Feb 2022 17:18:39 +0100    

Click here for diff

Support --type option in the info command to display only a specific backup type.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/info/info.c
M src/config/parse.auto.c
M test/src/module/command/infoTest.c

Remove logic that tried to determine additional file system compression.

commit   : cb630ffe3b5c819b5e97ac11d5974000bd44d632    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Feb 2022 09:32:23 -0600    
  
committer: GitHub <[email protected]>    
date     : Wed, 9 Feb 2022 09:32:23 -0600    

Click here for diff

In theory, the additional stat() call after a file has been copied to the repo can determine if additional compression has been applied by the file system. However, it has been a very long time since we tested this in practice. There are currently no unit tests that accurately test this feature since it requires a compressed file system like ZFS to work, which never seemed worth the extra cost.
  

  
It can also add a lot of time to backups if there are a large quantity of small files.
  

  
In addition, it stands as a blocker for combining files for small file support since it is no longer possible to get per-file sizes from the viewpoint of the file system. There are several ways this could be reworked but none of them are easy while at the same time maintaining current info functionality.
  

  
It doesn't seem worth keeping an untested feature that will only work in some special cases (if it still works) when it is blocking development.

M doc/xml/release.xml
M src/build/help/help.xml
M src/command/backup/file.c
M src/storage/posix/storage.c
M src/storage/storage.h
M test/src/module/command/backupTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Fix invalid file descriptor being passed to close().

commit   : 755bfc4d4015a68fa06890d1d62515b8bbbc3a64    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Feb 2022 08:14:28 -0600    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Feb 2022 08:14:28 -0600    

Click here for diff

Coverity pointed out that a negative number could be passed to close(), which means the lock file would not get closed until the process ended. Proper execution does not require the file to be closed, but it is better to correctly free resources that are no longer needed.  

M src/common/lock.c

Remove unused backup-standby option from stanza commands.

commit   : 5cbb4a3e6b13598b90cae46fad28c31604f9f094    
  
author   : David Steele <[email protected]>    
date     : Fri, 4 Feb 2022 15:55:24 -0600    
  
committer: David Steele <[email protected]>    
date     : Fri, 4 Feb 2022 15:55:24 -0600    

Click here for diff

M doc/xml/release.xml
M src/build/config/config.yaml
M src/config/parse.auto.c

Simplify filename construction in command/verify module.

commit   : 7dd657b7dd163ded9fbf7caef37eea7e08bc5a37    
  
author   : David Steele <[email protected]>    
date     : Fri, 4 Feb 2022 12:48:50 -0600    
  
committer: David Steele <[email protected]>    
date     : Fri, 4 Feb 2022 12:48:50 -0600    

Click here for diff

Remove some duplicated code by tracking the backup label and constructing the filename only when needed.  

M doc/xml/release.xml
M src/command/verify/verify.c

Revert Minio to prior release.

commit   : b1da4e84e8b1fda0299931dbcb9b057ce0d3ce64    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Feb 2022 14:39:39 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Feb 2022 14:39:39 -0600    

Click here for diff

The most recent release of Minio has broken CI builds but there is no logging to indicate what is wrong.  
  
For now, just use the prior release to get CI builds working again. This kind if breakage is not uncommon for Minio but they usually resolve it in the next release.  

M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm

Refactor lock code.

commit   : 9b2f10dbb49a8b869cef2c419d9957ed4c550016    
  
author   : David Steele <[email protected]>    
date     : Mon, 31 Jan 2022 16:48:28 -0600    
  
committer: GitHub <[email protected]>    
date     : Mon, 31 Jan 2022 16:48:28 -0600    

Click here for diff

Update lock code to use standard common/io functions and module patterns. This module was developed before the common/io module existed and our patterns had stabilized.

M doc/xml/release.xml
M src/command/control/stop.c
M src/common/lock.c
M src/common/lock.h
M test/src/module/common/lockTest.c

Add ioBufferReadNewOpen() and ioBufferWriteNewOpen().

commit   : 22734eb376b3183924cb7794f9111c1eeb5d92d6    
  
author   : David Steele <[email protected]>    
date     : Mon, 31 Jan 2022 10:03:56 -0600    
  
committer: David Steele <[email protected]>    
date     : Mon, 31 Jan 2022 10:03:56 -0600    

Click here for diff

These are convenience functions to make the code a bit more compact where possible.  

M doc/xml/release.xml
M src/build/help/render.c
M src/common/io/bufferRead.h
M src/common/io/bufferWrite.h
M test/src/module/common/ioTest.c
M test/src/module/common/typePackTest.c

Remove release note about the default port being provisional.

commit   : e62e6664d3654dd63ae2eda64f981a6936b822d3    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Jan 2022 13:26:22 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Jan 2022 13:26:22 -0600    

Click here for diff

The default port has now been approved by IANA.  

M doc/xml/release.xml

Fix language in rh7 test container for aarch64.

commit   : cf5b3a302f0835be993a7f795038d4d676cd6f01    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Jan 2022 13:22:31 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Jan 2022 13:22:31 -0600    

Click here for diff

The /etc/profile.d/lang.sh script was causing issues but it does not exist on amd64, so it seems the easiest thing was to remove it.  
  
Fix how 32-bit VMs are determined now that another 64-bit architecture has been added.  
  
And remove some obsolete VM hashes.  

M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm

Download correct key on aarch64 in the user guide.

commit   : da0f3a85535df3389c5023afc637124acc6405d3    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Jan 2022 10:15:14 -0600    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Jan 2022 10:15:14 -0600    

Click here for diff

The default key that gets installed is for amd64. This is only required for RHEL8 it seems, so something is definitely amiss.  

M doc/xml/user-guide.xml

Simplify manifest file defaults.

commit   : e4df5b7d3846134832b9315e22786f679d63a46b    
  
author   : David Steele <[email protected]>    
date     : Mon, 24 Jan 2022 16:21:07 -0500    
  
committer: GitHub <[email protected]>    
date     : Mon, 24 Jan 2022 16:21:07 -0500    

Click here for diff

Previously manifest load required two passes through the file list, one to load the data and one to set the defaults. This required each file to be packed twice.
  

  
Instead simply note that the file value is default and then set the file defaults when they are loaded from the manifest. This is made possible by the different internal/external representations for files so the same method cannot be applied to paths and links.
  

  
This change seems to resolve the performance issues noted in 61ce586 but there is no obvious reason why.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/restore/restore.c
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/command/backupTest.c

Centralize common PostgreSQL options in the user guide.

commit   : 044138fbb19f02ec433e52c42684b1ecc616c624    
  
author   : David Steele <[email protected]>    
date     : Sun, 23 Jan 2022 15:07:13 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 23 Jan 2022 15:07:13 -0500    

Click here for diff

Centralize these options so they are consistent across clusters.  
  
Also, there were some options that the user doesn't really need to see, .e.g. log_line_prefix. These can be set in advance so they don't need to be part of the documentation.  

M doc/xml/user-guide.xml

Create create_test_table() function earlier in user guide.

commit   : 13623d6ee50794e87ceac37c61e72ba3a92ad8b6    
  
author   : David Steele <[email protected]>    
date     : Sun, 23 Jan 2022 13:53:52 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 23 Jan 2022 13:53:52 -0500    

Click here for diff

This function (which creates lots of tables) is generally useful for testing (not just stress testing) so create it as soon as the cluster is created.  
  
Also add the data parameter which will insert a single row into the table so the file on disk is not zero bytes.  

M doc/xml/user-guide.xml

Optimization for jsonFromStrInternal().

commit   : ca13f11b4a3858587c7055b0b2b4a8b396dd8451    
  
author   : David Steele <[email protected]>    
date     : Sat, 22 Jan 2022 17:20:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 22 Jan 2022 17:20:25 -0500    

Click here for diff

This is an extremely hot code path when saving the manifest so every little bit helps.  

M doc/xml/release.xml
M src/common/type/json.c

Pack manifest file structs to save memory.

commit   : 61ce58692f3092b72833b6e9ee87037f45b3b702    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Jan 2022 17:05:07 -0500    
  
committer: GitHub <[email protected]>    
date     : Fri, 21 Jan 2022 17:05:07 -0500    

Click here for diff

Manifests with a very large number of files can use a considerable amount of memory. There are a lot of zeroes in the data so it can be stored more efficiently by using base-128 varint encoding for the integers and storing the strings in the same allocation.
  

  
The downside is that the data needs to be unpacked in order to be used, but in most cases this seems fast enough (about 10% slower than before) except for saving the manifest, which is 10% slower up to 10 million files and then gets about 5x slower by 100 million (two minutes on my M1 Mac). Profiling does not show this slowdown so I wonder if this is related to the change in memory layout. Curiously, the function that increased most was jsonFromStrInternal(), which was not modified. That gives more weight to the idea that there is some kind of memory issue going on here and one hopes that servers would be less affected. Either way, they largest use cases we have seen are for about 6 million files so if we can improve that case I believe we will be better off.
  

  
Further analysis showed that most of the time was taken up writing the size and timestamp fields, which makes almost no sense. The same amount of time was used if they were hard-coded to 0, which points to some odd memory issue on the M1 architecture.
  

  
This change has been planned for a while, but the particular impetus at this time is that small file support requires additional fields that would increase manifest memory usage by about 20%, even if the feature is not used.
  

  
Note that the Pack code has been updated to use the new varint encoder, but the decoder remains separate because it needs to fetch one byte at a time.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/info/info.c
M src/command/restore/restore.c
M src/command/verify/verify.c
M src/common/type/convert.c
M src/common/type/convert.h
M src/common/type/pack.c
M src/info/infoBackup.c
M src/info/manifest.c
M src/info/manifest.h
M test/define.yaml
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/typeConvertTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/typeTest.c

Convert varNewUInt64() to VARUINT64() where possible in manifest.

commit   : 575ae77c0dee12b932469d479b2e45dbaf389737    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Jan 2022 15:57:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 Jan 2022 15:57:22 -0500    

Click here for diff

VARUINT64() does not require memory to be allocated from the mem context so should be used when possible.  

M doc/xml/release.xml
M src/info/manifest.c

Simplify manifest defaults.

commit   : 4a73a0286387f3f2185f3f193f9c20ed0d7d926d    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Jan 2022 15:22:48 -0500    
  
committer: GitHub <[email protected]>    
date     : Fri, 21 Jan 2022 15:22:48 -0500    

Click here for diff

Manifest defaults for user, group, and mode were previously generated by scanning the data to find the most common values. This was very accurate but slow and complicated. It could also lead to surprising changes in the manifest when a default value suddenly changed.
  

  
Instead, use the $PGDATA path to generate defaults. In the vast majority of cases the same user/group should own all the path/files and the default file mode is easily derived from the path mode. There may be some edge cases where this generates larger manifests, but in general it reduces time and complexity when saving the manifest.
  

  
Remove the MCV code since it is longer longer used.

M doc/xml/release.xml
M src/Makefile.in
D src/common/type/mcv.c
D src/common/type/mcv.h
M src/info/manifest.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/expect/mock-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
D test/src/module/common/typeMcvTest.c
M test/src/module/info/manifestTest.c

Simplify base path mode in mock/all integration tests.

commit   : b0db4b8ff00db913cb69308017a56155d6070830    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Jan 2022 08:52:51 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 Jan 2022 08:52:51 -0500    

Click here for diff

Change the mode back to 0700 earlier to reduce churn in the expect logs.  
  
This will be especially important in a future commit that gets the defaults exclusively from the base path.  

M doc/xml/release.xml
M test/expect/mock-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Remove primary flag from manifest.

commit   : 8c062e1af824c3dc80ff94b3df8d433f95e88d0f    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Jan 2022 14:01:10 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 20 Jan 2022 14:01:10 -0500    

Click here for diff

This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role.
  

  
Since backup is the only code concerned with the primary flag, move the code into the backup module.
  

  
We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/info/manifest.c
M src/info/manifest.h
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/src/module/command/backupTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/verifyTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/manifestTest.c

Use the PG_FILE_POSTMTRPID constant where appropriate.

commit   : 16559d9e421c66596d21b047c8afe1b2c96ccd82    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Jan 2022 08:41:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 Jan 2022 08:41:05 -0500    

Click here for diff

Do the same in Perl with the MANIFEST_FILE_POSTMTRPID constant.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c

Refactor backupProcessQueue() to use BackupJobData struct.

commit   : 4ca0590a51a665535fbba903b8f55e10557b606a    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Jan 2022 08:06:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 Jan 2022 08:06:42 -0500    

Click here for diff

BackupJobData has several members that backupProcessQueue() needs so it is more efficient to use them rather than passing them separately or getting them from the configuration.  

M doc/xml/release.xml
M src/command/backup/backup.c

Remove extra spaces.

commit   : e21ba7c92bc15e9314eefa80cef8f17cf2f0e224    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 Jan 2022 17:40:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Jan 2022 17:40:53 -0500    

Click here for diff

M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/restore/restore.c
M src/common/io/http/response.c
M src/common/io/tls/server.c
M src/common/log.h
M src/config/load.c
M src/info/infoBackup.c
M src/info/manifest.c
M src/storage/helper.c
M test/src/common/harnessPq.h
M test/src/common/harnessStorage.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/storage/azureTest.c

Fix issue with Posix read offset handling after an error.

commit   : f84909f85f7dfb202ac812c67665cc590a5acad4    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 Jan 2022 17:26:30 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Jan 2022 17:26:30 -0500    

Click here for diff

Coverity pointed out that -1 could be passed to lseek() (added in a79034ae) after a file failed to open because it is missing. Overall it seems simpler to enclose the success code in an else block to prevent any repeats of this mistake in the future.  
  
This was not an active bug because there are currently no cases where we do read offsets in a file that is allowed to be missing.  
  
Also remove the result flag since it is easier to just check that the file descriptor is valid.  

M doc/xml/release.xml
M src/storage/posix/read.c

Fix typo.

commit   : 91721e391a4d83a405a0d1be31b854312f9f157d    
  
author   : David Steele <[email protected]>    
date     : Mon, 17 Jan 2022 15:39:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 17 Jan 2022 15:39:31 -0500    

Click here for diff

M src/common/type/convert.h

Implement restore ownership without updating manifest internals.

commit   : b791f1c82f53fa16866837d4544b58243c682f7f    
  
author   : David Steele <[email protected]>    
date     : Sat, 15 Jan 2022 14:33:38 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 15 Jan 2022 14:33:38 -0500    

Click here for diff

Updating the manifest this way was not a great idea because it broke abstraction for the object. This meant certain changes to the interface and internals were not possible because the code was modifying internal manifest data.  
  
Instead track the user replacements entirely in the restore module.  
  
This also has the benefit of eliminating a pass over the manifest path/file/link lists.  

M doc/xml/release.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Add support for AWS S3 server-side encryption using KMS.

commit   : 3097acd73a21b630aa5352be57a82b031930f96b    
  
author   : Christoph Berg <[email protected]>    
date     : Thu, 13 Jan 2022 14:46:14 +0100    
  
committer: GitHub <[email protected]>    
date     : Thu, 13 Jan 2022 14:46:14 +0100    

Click here for diff

AWS S3 integrates with AWS Key Management Service (AWS KMS) to provide server side encryption of S3 objects. This integration protects objects under encryption keys that never leave AWS KMS unencrypted.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/config/config.auto.h
M src/config/parse.auto.c
M src/storage/s3/helper.c
M src/storage/s3/storage.c
M src/storage/s3/storage.h
M src/storage/s3/storage.intern.h
M src/storage/s3/write.c
M test/src/module/command/helpTest.c
M test/src/module/storage/s3Test.c

cfgOptionGroupName() replacement missed in aced5d47.

commit   : 92ea3e05fb43e5272a19fb1e3208412b75008c24    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Jan 2022 19:15:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Jan 2022 19:15:09 -0500    

Click here for diff

M doc/xml/release.xml
M src/command/archive/get/get.c

Add read range to all storage drivers.

commit   : a79034ae2f6a8ec66b84c62ab6cd4042d2c1a249    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Jan 2022 14:42:53 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 11 Jan 2022 14:42:53 -0500    

Click here for diff

The range feature allows reading out an arbitrary chunk of a file and will be important for efficient small file support.
  

  
Now that all drivers are required to support ranges remove the storageFeatureLimitRead feature flag that was implemented only by the Posix driver.

M doc/xml/release.xml
M src/common/io/http/header.c
M src/common/io/http/header.h
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/storage/azure/read.c
M src/storage/azure/read.h
M src/storage/azure/storage.c
M src/storage/gcs/read.c
M src/storage/gcs/read.h
M src/storage/gcs/storage.c
M src/storage/posix/read.c
M src/storage/posix/read.h
M src/storage/posix/storage.c
M src/storage/read.h
M src/storage/read.intern.h
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/read.h
M src/storage/remote/storage.c
M src/storage/s3/read.c
M src/storage/s3/read.h
M src/storage/s3/storage.c
M src/storage/s3/storage.intern.h
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/src/module/common/ioHttpTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Rename ConfigOptionGroupData.indexDisplay to indexName.

commit   : 2fd100bf12760a4bb1996fb464ae54b5c379988e    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 17:40:18 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 17:40:18 -0500    

Click here for diff

This is more consistent with the function name and purpose.  

M doc/xml/release.xml
M src/config/config.c
M src/config/config.intern.h

Remove obsolete cfgOptionHostPort()/cfgOptionIdxHostPort().

commit   : 2cddbbdee01391ef0885f9262f3acc488382eee2    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 17:20:48 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 17:20:48 -0500    

Click here for diff

These functions were made obsolete by the refactor in 6a124584.  

M doc/xml/release.xml
M src/config/config.c
M src/config/config.h
M test/src/module/config/parseTest.c

Refactor option and option group config structs.

commit   : 7c627c12b735345cc3d733ae66a52b1fe0030e47    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 17:00:58 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 17:00:58 -0500    

Click here for diff

This allows individual structs to be assigned to variables to make the code easier to read and perhaps a bit more efficient.  

M doc/xml/release.xml
M src/config/config.c
M src/config/config.intern.h

Improve implementation of cfgOptionIdxName().

commit   : aeecb500f54561f7d52fa1b637f7f6e1f7a668de    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 14:47:29 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 14:47:29 -0500    

Click here for diff

Cache option names after they are generated rather than regenerating them each time.  

M doc/xml/release.xml
M src/config/config.c
M src/config/config.intern.h
M test/src/module/config/parseTest.c

Replace cfgOptionGroupIdxToKey() with cfgOptionGroupName().

commit   : aced5d47ed8e57ca7f36f92fe068f80b66f81649    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 09:10:06 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Jan 2022 09:10:06 -0500    

Click here for diff

Do the replacement anywhere cfgOptionGroupIdxToKey() is being used to construct a group name in a message. cfgOptionGroupName() is better for this case since it also includes the name of the group so that it does not need to be repeated in each message.  

M doc/xml/release.xml
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/expire/expire.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/command/stanza/upgrade.c
M src/config/config.c
M src/config/config.h
M src/config/config.intern.h
M src/config/load.c
M src/db/helper.c
M test/src/module/config/parseTest.c

Dynamically allocate index to key index map.

commit   : 9657f1b32519272edaaefd379355137afd554e82    
  
author   : David Steele <[email protected]>    
date     : Sun, 9 Jan 2022 19:53:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 9 Jan 2022 19:53:05 -0500    

Click here for diff

Now the config/config module has no notion of max option indexes. The config/parse still needs the max but this will be removed at a later date.  

M doc/xml/release.xml
M src/config/config.intern.h
M src/config/parse.c

Fix inconsistent group display names in messages.

commit   : e4b48eb430a564184ab739908df55813dd970aa4    
  
author   : David Steele <[email protected]>    
date     : Sun, 9 Jan 2022 19:43:44 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 9 Jan 2022 19:43:44 -0500    

Click here for diff

In other instances there are no dashes, e.g. repo1 or pg1. Make these messages match.  

M doc/xml/release.xml
M src/db/helper.c
M test/src/module/command/checkTest.c
M test/src/module/db/dbTest.c

Add ioCopy().

commit   : 5f78a5fc18120c67cb8a68f83a4fb9bcf96982e2    
  
author   : David Steele <[email protected]>    
date     : Sun, 9 Jan 2022 13:19:43 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 9 Jan 2022 13:19:43 -0500    

Click here for diff

Functionality to copy from IoRead to IoWrite is frequently used so centralize it. This also simplifies coverage testing in places where a loop was required before.  

M src/command/repo/get.c
M src/command/repo/put.c
M src/common/io/io.c
M src/common/io/io.h
M src/storage/storage.c
M test/src/module/common/ioTest.c
M test/src/module/performance/storageTest.c

Combine encrypted backupFile() tests with unencrypted tests.

commit   : 47954774c6cc70a6fc9a6c8836d82a87e341ea11    
  
author   : David Steele <[email protected]>    
date     : Sun, 9 Jan 2022 10:11:00 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 9 Jan 2022 10:11:00 -0500    

Click here for diff

This makes it easier to comment out all the tests while developing without getting unused variable errors.  

M test/define.yaml
M test/src/module/command/backupTest.c

Add backup LSNs to info command output.

commit   : d866dd5c297dc046581134a108032cf9694cae0e    
  
author   : Stefan Fercot <[email protected]>    
date     : Fri, 7 Jan 2022 20:09:58 +0100    
  
committer: GitHub <[email protected]>    
date     : Fri, 7 Jan 2022 20:09:58 +0100    

Click here for diff

The backup LSNs are useful for performing LSN-based PITR. LSNs will not be displayed in the general text output (without --set) because they are probably not useful enough to deserve their own line.

M doc/xml/release.xml
M src/command/info/info.c
M src/info/infoBackup.c
M src/info/infoBackup.h
M test/src/module/command/infoTest.c

Remove support for PostgreSQL 8.3/8.4.

commit   : bb4b30ddd3f80c246c565eb4769efc82b9509648    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Jan 2022 15:34:04 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 6 Jan 2022 15:34:04 -0500    

Click here for diff

There is no evidence that users need 8.3/8.4 anymore but it does cost us in terms of development and testing, especially now that we have a number of new backup/restore features planned.
  

  
It seems to make sense to remove this support now. If there are users who need to use/migrate from these versions they can use an older version of pgBackRest.

M README.md
M doc/xml/index.xml
M doc/xml/release.xml
M src/build/help/help.xml
M src/command/backup/backup.c
M src/command/restore/restore.c
M src/db/db.c
M src/info/manifest.c
M src/postgres/interface.c
D src/postgres/interface/v083.c
D src/postgres/interface/v084.c
M src/postgres/interface/version.h
M src/postgres/interface/version.intern.h
M src/postgres/interface/version.vendor.h
M src/postgres/version.h
M test/code-count/file-type.yaml
M test/define.yaml
M test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessPostgres.c
M test/src/common/harnessPostgres.h
D test/src/common/harnessPostgres/harness083.c
D test/src/common/harnessPostgres/harness084.c
M test/src/common/harnessPostgres/harnessVersion.intern.h
M test/src/common/harnessPq.h
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/interfaceTest.c

Fix comment missed in 6bd280f7.

commit   : ef62ef23799f081582295f1f3e8a92691b628f53    
  
author   : Reid Thompson <[email protected]>    
date     : Thu, 6 Jan 2022 14:24:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Jan 2022 14:24:32 -0500    

Click here for diff

M src/command/backup/backup.c

Fix typo in error message.

commit   : fdbeb8e7d64754d35dd30e4f871834756c5d77ca    
  
author   : Reid Thompson <[email protected]>    
date     : Thu, 6 Jan 2022 14:22:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Jan 2022 14:22:56 -0500    

Click here for diff

M src/db/db.c
M test/src/module/db/dbTest.c

Note that replications slots are not restored.

commit   : a82f0179cdd5742e5c5b1e0bdfd23166753fdb38    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 4 Jan 2022 16:11:27 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 4 Jan 2022 16:11:27 -0500    

Click here for diff

Update documentation and help to note that replication slots are not restored and reference the PostgreSQL documentation to explain why.

M doc/xml/release.xml
M src/build/help/help.xml
M test/src/module/command/helpTest.c

Fix typo.

commit   : 226cfbdcde55ca384e74d7d1348c447eecd079fa    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Jan 2022 15:48:00 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 Jan 2022 15:48:00 -0500    

Click here for diff

M src/command/help/help.c

Update config.guess and config.sub to latest versions.

commit   : 591a72c59ec403341d29eaafe3030f4651732192    
  
author   : David Steele <[email protected]>    
date     : Mon, 3 Jan 2022 08:57:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 3 Jan 2022 08:57:05 -0500    

Click here for diff

M src/build/config.guess
M src/build/config.sub

Begin v2.38 development.

commit   : 75e4580919ed139f00db7c4c3141dbefc3da131b    
  
author   : David Steele <[email protected]>    
date     : Mon, 3 Jan 2022 08:52:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 3 Jan 2022 08:52:25 -0500    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h