pgBackRest 2.47 commit log

v2.46: Block Incremental Backup and SFTP Storage

commit   : b461f7c6f884dba8056858464d1b571e708dad77    
  
author   : David Steele <[email protected]>    
date     : Mon, 22 May 2023 11:13:13 +0300    
  
committer: David Steele <[email protected]>    
date     : Mon, 22 May 2023 11:13:13 +0300    

Click here for diff

Features:  
  
* Block incremental backup. (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)  
* SFTP support for repository storage. (Contributed by Reid Thompson. Reviewed by Stephen Frost, David Steele.)  
* PostgreSQL 16 support. (Reviewed by Stefan Fercot.)  
  
Improvements:  
  
* Allow page header checks to be skipped. (Reviewed by David Christensen. Suggested by David Christensen.)  
* Avoid chown() on recovery files during restore. (Reviewed by Stefan Fercot, Marcelo Henrique Neppel. Suggested by Marcelo Henrique Neppel.)  
* Add error retry detail for HTTP retries.  
  
Documentation Improvements:  
  
* Add warning about using recovery type=none. (Reviewed by Stefan Fercot.)  
* Add note about running stanza-create on already-created repositories.  

M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Add warning about using recovery type=none.

commit   : a28b72c62431bcf9df02791219d9d6f5d0e60b54    
  
author   : David Steele <[email protected]>    
date     : Thu, 18 May 2023 09:32:44 +0300    
  
committer: GitHub <[email protected]>    
date     : Thu, 18 May 2023 09:32:44 +0300    

Click here for diff

M doc/xml/release.xml
M src/build/help/help.xml

Add bundling and block incremental to stress testing.

commit   : c61115e8adccd17d8afb6bd527fb6a396ec6b72a    
  
author   : David Steele <[email protected]>    
date     : Mon, 15 May 2023 17:46:41 +0300    
  
committer: David Steele <[email protected]>    
date     : Mon, 15 May 2023 17:46:41 +0300    

Click here for diff

This makes the stress testing more robust and provides additional testing for important features.  

M doc/xml/user-guide.xml

SFTP support for repository storage.

commit   : 87087fac231d2110160b3219daee3f9fb91aec02    
  
author   : Reid Thompson <[email protected]>    
date     : Sat, 13 May 2023 12:16:16 -0400    
  
committer: GitHub <[email protected]>    
date     : Sat, 13 May 2023 12:16:16 -0400    

Click here for diff

M doc/xml/release.xml
M doc/xml/user-guide.xml
M meson.build
M src/Makefile.in
M src/build.auto.h.in
M src/build/config/config.yaml
M src/build/configure.ac
M src/build/help/help.xml
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c.inc
M src/configure
M src/main.c
M src/meson.build
A src/storage/sftp/helper.c
A src/storage/sftp/helper.h
A src/storage/sftp/read.c
A src/storage/sftp/read.h
A src/storage/sftp/storage.c
A src/storage/sftp/storage.h
A src/storage/sftp/storage.intern.h
A src/storage/sftp/write.c
A src/storage/sftp/write.h
M test/Dockerfile
M test/Vagrantfile
A test/certificate/ssh/id_rsa
A test/certificate/ssh/id_rsa.pub
M test/ci.pl
M test/code-count/file-type.yaml
M test/container.yaml
M test/define.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
A test/lib/pgBackRestTest/Env/Host/HostSftpTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/command/test/build.c
A test/src/common/harnessLibSsh2.c
A test/src/common/harnessLibSsh2.h
A test/src/common/harnessSocket.c
A test/src/common/harnessSocket.h
M test/src/module/command/helpTest.c
M test/src/module/config/loadTest.c
A test/src/module/storage/sftpTest.c
M test/src/module/test/testTest.c
M test/src/test.c

Add note about running stanza-create on already-created repositories.

commit   : 0c1f823e7a3abf1cadb44de3832ec1ddb2da7ec8    
  
author   : David Steele <[email protected]>    
date     : Fri, 12 May 2023 15:46:04 +0300    
  
committer: David Steele <[email protected]>    
date     : Fri, 12 May 2023 15:46:04 +0300    

Click here for diff

M doc/xml/release.xml
M src/build/help/help.xml

Increase check timeouts in real/all integration tests.

commit   : c7ac17330405510e56243e967d108b8d3b387ce0    
  
author   : David Steele <[email protected]>    
date     : Wed, 10 May 2023 09:22:26 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 10 May 2023 09:22:26 +0300    

Click here for diff

The prior timeouts were a bit aggressive and were causing timeouts in the Azure tests. There have also been occasional timeouts in other storage drivers.  
  
The performance of CI environments is pretty variable so increased timeouts should make the tests more stable.  

M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Fix typos in test titles.

commit   : 2fc641383709ce1f91f35bb1edf4e7c1ab0f4eb3    
  
author   : David Steele <[email protected]>    
date     : Sat, 6 May 2023 12:16:58 +0300    
  
committer: David Steele <[email protected]>    
date     : Sat, 6 May 2023 12:16:58 +0300    

Click here for diff

M test/src/module/storage/posixTest.c

Fix groupId type in StorageInfo.

commit   : 6f250bf6860054397181f5678220faaa6574993e    
  
author   : David Steele <[email protected]>    
date     : Sat, 6 May 2023 12:15:21 +0300    
  
committer: David Steele <[email protected]>    
date     : Sat, 6 May 2023 12:15:21 +0300    

Click here for diff

gid_t is the correct type but there was no bug because uid_t and gid_t have the same underlying type.  

M src/storage/info.h

Update Debian user guide to build on Ubuntu 20.04.

commit   : ad9196cb49ce86cef678e453046803fb2be2cf70    
  
author   : David Steele <[email protected]>    
date     : Fri, 5 May 2023 10:08:49 +0300    
  
committer: David Steele <[email protected]>    
date     : Fri, 5 May 2023 10:08:49 +0300    

Click here for diff

Ubuntu 18.04 will be EOL at the end of the month so update to a newer version.  

M doc/xml/user-guide.xml

Fix incorrect commit hash in 2f322e2a.

commit   : 023d3c99d8d00497501b21fb20b07f71ea49ee40    
  
author   : David Christensen <[email protected]>    
date     : Thu, 4 May 2023 08:52:31 +0300    
  
committer: David Steele <[email protected]>    
date     : Thu, 4 May 2023 08:52:31 +0300    

Click here for diff

M .git-blame-ignore-revs

Add efac3474 (remove double spacing) to .git-blame-ignore-revs.

commit   : 2f322e2a49f20ff3036db6382363efb40246922e    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 May 2023 09:49:24 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 May 2023 09:49:24 +0300    

Click here for diff

M .git-blame-ignore-revs

Remove double spaces missed in 1bd5530a.

commit   : d49907239eb37c3e4e905f97543824181a1bd406    
  
author   : David Steele <[email protected]>    
date     : Wed, 3 May 2023 09:37:59 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 3 May 2023 09:37:59 +0300    

Click here for diff

M doc/xml/contributing.xml
M doc/xml/documentation.xml
M doc/xml/dtd/doc.dtd
M doc/xml/faq.xml
M test/src/module/command/verifyTest.c

Add 1bd5530a (remove double spacing) to .git-blame-ignore-revs.

commit   : efac3474cffc4f01154c13cc358601c4c98bcd42    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 May 2023 12:59:20 +0300    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 May 2023 12:59:20 +0300    

Click here for diff

M .git-blame-ignore-revs

Remove double spaces from comments and documentation.

commit   : 1bd5530a59cd8ddbabc279802d1ede4f8fbd5314    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 May 2023 12:57:12 +0300    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 May 2023 12:57:12 +0300    

Click here for diff

Double spaces have fallen out of favor in recent years because they no longer contribute to readability.  
  
We have been using single spaces and editing related paragraphs for some time, but now it seems best to update the remaining instances to avoid churn in unrelated commits and to make it clearer what spacing contributors should use.  

M CONTRIBUTING.md
M doc/README.md
M doc/RELEASE.md
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocRender.pm
M doc/lib/pgBackRestDoc/Common/Ini.pm
M doc/lib/pgBackRestDoc/Custom/DocConfigData.pm
M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
M doc/lib/pgBackRestDoc/ProjectInfo.pm
M doc/manifest.xml
M doc/resource/fake-cert/README.md
M doc/resource/git-history.cache
M doc/xml/coding.xml
M doc/xml/contributing.xml
M doc/xml/documentation.xml
M doc/xml/faq.xml
M doc/xml/index.xml
M doc/xml/metric.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/Makefile.in
M src/build/common/regExp.h
M src/build/configure.ac
M src/build/error/error.yaml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/backup/file.c
M src/command/command.c
M src/command/expire/expire.c
M src/command/remote/remote.c
M src/command/repo/get.c
M src/command/repo/ls.c
M src/command/restore/restore.c
M src/common/assert.h
M src/common/compress/helper.h
M src/common/compress/lz4/compress.c
M src/common/crypto/cipherBlock.c
M src/common/error/error.c
M src/common/error/error.h
M src/common/exec.c
M src/common/fork.c
M src/common/io/filter/buffer.h
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/sink.h
M src/common/io/filter/size.h
M src/common/io/http/client.h
M src/common/io/http/header.c
M src/common/io/http/header.h
M src/common/io/http/request.c
M src/common/io/http/response.c
M src/common/io/io.c
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/write.h
M src/common/lock.c
M src/common/lock.h
M src/common/log.h
M src/common/macro.h
M src/common/memContext.c
M src/common/memContext.h
M src/common/stackTrace.c
M src/common/type/buffer.h
M src/common/type/pack.c
M src/common/type/pack.h
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/common/type/xml.c
M src/common/user.h
M src/config/config.h
M src/config/exec.h
M src/config/parse.c
M src/configure
M src/db/db.c
M src/db/db.h
M src/info/info.h
M src/info/manifest.c
M src/info/manifest.h
M src/postgres/client.c
M src/postgres/client.h
M src/postgres/interface/pageChecksum.vendor.c.inc
M src/postgres/interface/static.vendor.h
M src/postgres/interface/version.vendor.h
M src/protocol/helper.c
M src/protocol/parallel.c
M src/protocol/parallel.h
M src/protocol/server.c
M src/storage/helper.c
M src/storage/posix/read.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M src/storage/write.c
M src/version.h
M test/certificate/README.md
M test/container.yaml
M test/define.yaml
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/Io/Buffered.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/StoragePosix.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessFork.h
M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/common/harnessPq.h
M test/src/common/harnessTest.h
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/ioTest.c
M test/src/module/performance/typeTest.c
M test/src/test.c
M test/test.pl

Remove extraneous space from error messages.

commit   : 5ea7b91bf9ab57a29778d2332c0971e84f14c4c6    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 May 2023 12:07:28 +0300    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 May 2023 12:07:28 +0300    

Click here for diff

M src/build/common/yaml.c
M src/command/restore/restore.c
M test/src/module/build/commonTest.c
M test/src/module/command/restoreTest.c

Use boolean variant constants where possible instead of redeclaring.

commit   : 9637d9449091632f158f8e2be8afbad4bda200fc    
  
author   : David Steele <[email protected]>    
date     : Tue, 2 May 2023 09:37:42 +0300    
  
committer: David Steele <[email protected]>    
date     : Tue, 2 May 2023 09:37:42 +0300    

Click here for diff

M src/command/restore/restore.c
M test/src/module/command/repoTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/config/parseTest.c

Finalize block incremental feature.

commit   : 822f2a584260a012b833455a7a2bd1c8487e3010    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 May 2023 19:26:48 +0300    
  
committer: GitHub <[email protected]>    
date     : Mon, 1 May 2023 19:26:48 +0300    

Click here for diff

Remove beta status and update documentation to remove beta references and warnings.
  

  
The repo-block-* sub-options have been marked internal. Most users will be best off with the default behavior and we may still decide to change these options for remove them in the future.

M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/config/config.yaml
M src/config/parse.auto.c.inc

Remove extraneous spaces in user guide.

commit   : c510046dd77c40487d2d6c659834927a3bab1106    
  
author   : David Steele <[email protected]>    
date     : Mon, 1 May 2023 14:27:56 +0300    
  
committer: David Steele <[email protected]>    
date     : Mon, 1 May 2023 14:27:56 +0300    

Click here for diff

M doc/xml/user-guide.xml

Removing chunking and block numbers from incremental block list.

commit   : dd4e52679ecdc43539bd038b5aa9bd4ff9055cbf    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Apr 2023 23:29:12 +0300    
  
committer: GitHub <[email protected]>    
date     : Thu, 27 Apr 2023 23:29:12 +0300    

Click here for diff

These were intended to allow the block list to be scanned without reading the map but were never utilized. They were left in "just in case" and because they did not seem to be doing any harm.
  

  
In fact, it is better not to have the block numbers because this allows us set the block size at a future time as long as it is a factor of the super block size. One way this could be useful is to store older files without super blocks or a map in the full backup and then build a map for them if the file gets modified in a diff/incr backup. This would require reading the file from the full backup to build the map but it would be more space efficient and we could make more intelligent decisions about block size. It would also be possible to change the block size even if one had already been selected in a prior backup.
  

  
Omitting the block numbers makes the chunking unnecessary since there is now no way to make sense of the block list without the map. Also, we might want to build maps for unchunked block lists, i.e. files that were copied normally.

M doc/xml/release.xml
M src/Makefile.in
M src/command/backup/blockIncr.c
M src/command/restore/blockDelta.c
M src/command/restore/file.c
D src/common/io/chunkedRead.c
D src/common/io/filter/chunk.c
D src/common/io/filter/chunk.h
A src/common/io/limitRead.c
R075 src/common/io/chunkedRead.h src/common/io/limitRead.h
M src/meson.build
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/module/command/backupTest.c
M test/src/module/command/manifestTest.c
M test/src/module/common/ioTest.c

PostgreSQL 16 Support.

commit   : 3fc3690dd77ef5fa032abb006558184164db53ec    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Apr 2023 10:30:50 +0300    
  
committer: GitHub <[email protected]>    
date     : Thu, 27 Apr 2023 10:30:50 +0300    

Click here for diff

Add catalog version and WAL magic for PostgreSQL 16.
  

  
The GUC to force parallel mode has be renamed so update that in the tests.

M doc/xml/release.xml
M src/build/postgres/postgres.yaml
M src/postgres/interface/static.vendor.h
M src/postgres/interface/version.vendor.h
M src/postgres/version.auto.h
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessPostgres.c
M test/src/common/harnessPostgres.h
A test/src/common/harnessPostgres/harness160.c

Remove extra linefeed.

commit   : 39059dc4e7a74e6473099eefeed4e9e0efe65e60    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Apr 2023 15:00:36 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Apr 2023 15:00:36 +0300    

Click here for diff

M src/build/postgres/render.c

Avoid chown() on recovery files during restore.

commit   : 3ff88ffbb495fd669e9a361478e60192930ee0a1    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 Apr 2023 11:52:28 +0300    
  
committer: GitHub <[email protected]>    
date     : Tue, 25 Apr 2023 11:52:28 +0300    

Click here for diff

The chown() was already skipped on the files restored from the repository but the same logic was not applied to the generated recovery files, probably because chown'ing a few recovery files does not have performance implications. Use the same logic for recovery files to determined if they need to be chown'd.
  

  
Ultimately this behavior is pretty hard to test, so add a fail safe into the Posix driver that will skip chown if the permissions are already as required.

M doc/xml/release.xml
M src/command/restore/restore.c
M src/storage/posix/write.c
M test/src/module/command/restoreTest.c
M test/src/module/storage/posixTest.c

Add MacOS back to CirrusCI.

commit   : 750ab8e55c276fa0127d19961a59fec254e24c61    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Apr 2023 16:21:10 +0300    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 Apr 2023 16:21:10 +0300    

Click here for diff

9e29c01 removed MacOS testing due to breaking changes in the update to arm on the platform.  
  
Update the scripts to correctly work with the version of Homebrew deployed with the arm images.  

M .cirrus.yml

Fix missing void parameter.

commit   : 23d7d67d4961c9bb38d73ef1c409369303276846    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Apr 2023 15:08:59 +0300    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 Apr 2023 15:08:59 +0300    

Click here for diff

M src/common/stackTrace.c

Allow page header checks to be skipped.

commit   : f5e6bc26983396ec51c6579aade37db39807f3a5    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Apr 2023 13:24:12 +0300    
  
committer: GitHub <[email protected]>    
date     : Thu, 20 Apr 2023 13:24:12 +0300    

Click here for diff

These checks cause false negatives for page checksum verification when the page is encrypted because pd_upper might end up as 0 in the encrypted data. This issue is rare but reproducible given a large enough cluster.
  

  
Make these checks optional, but leave them enabled by default.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/pageChecksum.c
M src/command/backup/pageChecksum.h
M src/command/backup/protocol.c
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/postgres/interface/pageChecksum.vendor.c.inc
M src/postgres/interface/static.vendor.h
M test/src/module/command/backupTest.c

Autogenerate PostgreSQL versions.

commit   : 8240eb5da57d2950ebbd3492c163b4e0151d3cb7    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Apr 2023 17:41:27 +0300    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Apr 2023 17:41:27 +0300    

Click here for diff

This will make adding/removing versions of PostgreSQL more reliable.  

M src/build/main.c
M src/build/postgres/render.c
M src/build/postgres/render.h
A src/postgres/version.auto.h
M src/postgres/version.h
M test/src/module/build/postgresTest.c
M test/test.pl

Rename PG_VERSION_*_STR constants to PG_VERSION_*_Z.

commit   : a05bf6bb15ba7f7e4b610dbb84e87ebcd359dd38    
  
author   : David Steele <[email protected]>    
date     : Sun, 16 Apr 2023 17:32:24 +0300    
  
committer: David Steele <[email protected]>    
date     : Sun, 16 Apr 2023 17:32:24 +0300    

Click here for diff

This is more consistent with other zero-terminated string constants and also has the benefit of being shorter.  

M src/command/restore/restore.c
M src/db/db.c
M src/postgres/interface.c
M src/postgres/version.h
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/manifestTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/postgres/interfaceTest.c

Parameterize configLoad() as cfgLoadP().

commit   : 75254c9285e70913920af9ab6e9f362bc4d11d7d    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Apr 2023 16:28:29 +0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Apr 2023 16:28:29 +0400    

Click here for diff

There is one existing optional parameter and there are more to come.  

M src/config/config.intern.h
M src/config/load.c
M src/config/parse.c
M src/config/parse.h
M test/src/common/harnessConfig.c
M test/src/config/load.c
M test/src/module/command/helpTest.c
M test/src/module/config/parseTest.c

Update command/check module to recent coding standards.

commit   : d6cb3de17afff0a9c57f312455a5ca3cf202cba6    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Apr 2023 14:50:59 +0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Apr 2023 14:50:59 +0400    

Click here for diff

Add const as appropriate and remove an unneeded declaration.  

M src/command/check/check.c
M src/command/check/common.c

Update command/backup module to recent coding standards.

commit   : a9c5cd9749f6b5ae1005453b1039d91f37fb245b    
  
author   : David Steele <[email protected]>    
date     : Tue, 11 Apr 2023 12:46:00 +0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Apr 2023 12:46:00 +0400    

Click here for diff

Add const as appropriate and avoid setting results to NULL if the result will definitely be set later on.  

M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/backup/file.c
M src/command/backup/pageChecksum.c
M src/command/backup/protocol.c

Add error retry detail for HTTP retries.

commit   : f33e1cfb1600643820a1ae76bdeb15c62ac1fcc7    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Apr 2023 11:38:18 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Apr 2023 11:38:18 +0400    

Click here for diff

This should make it clearer when retries have happened and for how long.  

M doc/xml/release.xml
M src/common/io/http/request.c
M test/src/module/common/ioHttpTest.c

Improve retry error messages.

commit   : df419b34b55a15a902ff5cc1912aaebc7e023738    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Apr 2023 11:03:46 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Apr 2023 11:03:46 +0400    

Click here for diff

Centralize the code to allow it to be used in more places and update the protocol/server module to use the new code.  
  
Since the time measurements make testing difficult, also add time and errorRetry harnesses to allow specific data to be used for testing. In the case of errorRetry, the production behavior is turned off by default during testing and only enabled for the errorRetry test module.  

M src/Makefile.in
A src/common/error/retry.c
A src/common/error/retry.h
M src/common/time.h
M src/meson.build
M src/protocol/server.c
M test/define.yaml
A test/src/common/harnessErrorRetry.c
A test/src/common/harnessErrorRetry.h
A test/src/common/harnessTime.c
A test/src/common/harnessTime.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/verifyTest.c
A test/src/module/common/errorRetryTest.c
M test/src/module/protocol/protocolTest.c

Move error modules to common/error directory.

commit   : 801e396dace2a4d9f4125ef0e751c90dc0cc644a    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Apr 2023 10:38:49 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Apr 2023 10:38:49 +0400    

Click here for diff

There are already enough files to warrant a separate directory and more are coming.  
  
Also remove extraneous includes.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/Makefile.in
M src/build/error/error.yaml
M src/build/error/render.c
M src/command/exit.c
M src/common/assert.h
M src/common/crypto/common.c
M src/common/encode.c
R100 src/common/error.auto.c.inc src/common/error/error.auto.c.inc
R097 src/common/error.auto.h src/common/error/error.auto.h
R099 src/common/error.c src/common/error/error.c
R099 src/common/error.h src/common/error/error.h
M src/common/fork.c
M src/common/log.c
M src/common/memContext.c
M src/common/memContext.h
M src/common/stackTrace.c
M src/config/config.c
M src/config/parse.c
M src/main.c
M src/meson.build
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/common/harnessBackup.c
M test/src/common/harnessError.h
M test/src/common/harnessServer.c
M test/src/common/harnessTest.h
M test/src/main.c
M test/src/module/build/errorTest.c
M test/src/module/command/exitTest.c
M test/src/module/test/testTest.c
M test/src/test.c

Update 32-bit CI host to Debian 10 from Ubuntu 18.04.

commit   : ff98636e41b8ad14babe1e9fa1fdc15aad2244cd    
  
author   : David Steele <[email protected]>    
date     : Thu, 6 Apr 2023 08:22:14 +0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 6 Apr 2023 08:22:14 +0400    

Click here for diff

Ubuntu 18.04 will be EOL before the next release, so update to the oldest available Debian version.  
  
Also fix one incorrect return value type, a test cast, and adjust some test timeouts.  

M .github/workflows/test.yml
M doc/release.pl
M src/info/manifest.c
M test/container.yaml
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/src/common/harnessProtocol.c
M test/src/module/command/remoteTest.c
M test/src/module/command/restoreTest.c
M test/src/module/storage/gcsTest.c

Remove unused strOS parameter in Common::HostTest.

commit   : a9f39857cf7d8c0eda9f5f19f3d4c9497d4d47f2    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Apr 2023 11:24:53 +0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Apr 2023 11:24:53 +0400    

Click here for diff

M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M test/lib/pgBackRestTest/Common/HostTest.pm
M test/lib/pgBackRestTest/Env/Host/HostAzureTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/lib/pgBackRestTest/Env/Host/HostGcsTest.pm
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm

Sleep using nanosleep() instead of select().

commit   : 8f7f73e4af916bf1bc6a0cb72be35b7061b297df    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 17:46:25 +0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 17:46:25 +0500    

Click here for diff

This is a safer way to sleep due to select's not-portable interaction with signals.  
  
Based on https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a948e49e2ef11815be0b211723bfc5b53b7f75a8 from the PostgreSQL project.  

M src/common/time.c

Use Buffer object for page buffer in PageChecksum filter.

commit   : 2cfbee903a0a16d1fdeca52dbf6e775cdcddcc74    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 15:43:37 +0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 15:43:37 +0500    

Click here for diff

This avoids the object needing to support allocations.  

M src/command/backup/pageChecksum.c

Convert XmlNode from allocation to object.

commit   : 3dbf518c3a9df538ca110039758fbfe161e3f714    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 14:50:46 +0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 14:50:46 +0500    

Click here for diff

This makes memory accounting more accurate and objects that use Xml no longer need to allow allocations.  

M src/common/type/xml.c
M src/storage/s3/write.c

Remove allocations from common crypto module.

commit   : 047472144b5fd92425225700aa11b351044244e5    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 14:42:20 +0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 14:42:20 +0500    

Click here for diff

The allocations are better done with a struct and Buffer object.  

M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M test/src/module/common/cryptoTest.c

Cleanup for objects that do not require allocations.

commit   : 84ca7b9b2713b6d36d15d592e78942d8e1ac8a35    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 11:42:22 +0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 11:42:22 +0500    

Click here for diff

A lot of these are left over from when object interfaces required allocations (changed in f6e30736 and 9ca9c8e4). Others are likely copy/paste errors.  
  
This saves some space in the mem context and makes it clear that no allocations will be made.  

M src/common/compress/zst/compress.c
M src/common/io/tls/client.c
M src/common/io/tls/server.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/iterator.c
M src/storage/posix/read.c
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M test/src/common/harnessLog.c
M test/src/module/command/restoreTest.c
M test/src/module/performance/storageTest.c

Convert the result of zNewFmt() into an object.

commit   : fe0fd71be2e9754491cbbdcde79e7a7cc87f2e7b    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 10:00:58 +0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Mar 2023 10:00:58 +0500    

Click here for diff

The result is not intended to be freed directly so this makes memory tracking more accurate. Fix a few places where memory was leaking after a call to zNewFmt().  
  
Also update an assert to make it clearer.  

M src/common/type/buffer.c
M src/common/type/stringZ.c
M src/storage/helper.c
M test/src/module/common/typeStringTest.c
M test/src/module/storage/posixTest.c

Update Github actions to new versions.

commit   : 295b53e8453b4ac4889b62bc1c7f1d6b78b09421    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Mar 2023 16:24:43 +0600    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Mar 2023 16:24:43 +0600    

Click here for diff

These updates remove various deprecation warnings.  

M .github/workflows/lock-thread.yml
M .github/workflows/test.yml

Simplify object creation with OBJ_NEW_BEGIN() macro.

commit   : b111599badccb1c196a224e3d6ac8657b8d172a2    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Mar 2023 15:05:18 +0600    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Mar 2023 15:05:18 +0600    

Click here for diff

Eliminate the boilerplate of declaring this and assigning memory to it, which is the same for the vast majority of object creations.  
  
Keep the old version of the macro as OBJ_NEW_BASE_BEGIN() for a few exceptions in the core code and (mostly) in the tests.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/build/common/yaml.c
M src/command/backup/blockIncr.c
M src/command/backup/pageChecksum.c
M src/command/restore/blockChecksum.c
M src/command/restore/blockDelta.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/compress/zst/compress.c
M src/common/compress/zst/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/crypto/xxhash.c
M src/common/exec.c
M src/common/ini.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/chunkedRead.c
M src/common/io/client.c
M src/common/io/fdRead.c
M src/common/io/fdWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/chunk.c
M src/common/io/filter/filter.c
M src/common/io/filter/group.c
M src/common/io/filter/sink.c
M src/common/io/filter/size.c
M src/common/io/http/client.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/http/request.c
M src/common/io/http/response.c
M src/common/io/http/session.c
M src/common/io/http/url.c
M src/common/io/read.c
M src/common/io/server.c
M src/common/io/session.c
M src/common/io/socket/client.c
M src/common/io/socket/server.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/server.c
M src/common/io/tls/session.c
M src/common/io/write.c
M src/common/regExp.c
M src/common/type/blob.c
M src/common/type/buffer.c
M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/list.c
M src/common/type/object.h
M src/common/type/pack.c
M src/common/type/string.c
M src/common/type/variant.c
M src/common/type/xml.c
M src/common/wait.c
M src/config/parse.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M src/postgres/client.c
M src/protocol/client.c
M src/protocol/command.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/server.c
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/iterator.c
M src/storage/list.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/storage.c
M src/storage/write.c
M test/src/command/test/build.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/typeObjectTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/protocol/protocolTest.c

Fix command/manifest unit test title.

commit   : 91f9301b9d0026ee72875d943e0ed01b14ec887b    
  
author   : David Steele <[email protected]>    
date     : Mon, 27 Mar 2023 16:42:35 +0600    
  
committer: David Steele <[email protected]>    
date     : Mon, 27 Mar 2023 16:42:35 +0600    

Click here for diff

M test/src/module/command/manifestTest.c

Improve interface handling in remaining modules.

commit   : 9ca9c8e4c97b60a4cceef1c6f9fcb93e413e15d7    
  
author   : David Steele <[email protected]>    
date     : Mon, 27 Mar 2023 14:32:37 +0600    
  
committer: David Steele <[email protected]>    
date     : Mon, 27 Mar 2023 14:32:37 +0600    

Click here for diff

As in f6e30736, make the interface object the parent of the driver object rather than the interface being allocated directly in the driver object. Allow exceptions to this general rule for objects that need to retain ownership of their interfaces.  

M src/command/backup/blockIncr.c
M src/command/backup/pageChecksum.c
M src/command/restore/blockChecksum.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/compress/zst/compress.c
M src/common/compress/zst/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/crypto/xxhash.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/chunkedRead.c
M src/common/io/client.c
M src/common/io/client.h
M src/common/io/fdRead.c
M src/common/io/fdWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/chunk.c
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/sink.c
M src/common/io/filter/size.c
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/server.c
M src/common/io/server.h
M src/common/io/session.c
M src/common/io/session.h
M src/common/io/socket/client.c
M src/common/io/socket/server.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/server.c
M src/common/io/tls/session.c
M src/common/io/write.c
M src/common/io/write.h
M src/common/type/object.c
M src/common/type/object.h
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/storage.c
M src/storage/write.c
M test/src/module/common/ioTest.c
M test/src/module/common/typeObjectTest.c
M test/src/module/performance/storageTest.c

Fix indentation in config.yaml.

commit   : 5f001248cce08137a601ce5671aa67d26bfdd366    
  
author   : David Steele <[email protected]>    
date     : Sat, 25 Mar 2023 14:59:47 +0700    
  
committer: David Steele <[email protected]>    
date     : Sat, 25 Mar 2023 14:59:47 +0700    

Click here for diff

M src/build/config/config.yaml

Add lock module initialization.

commit   : 8ff956ad7e8608dd7582825521105c289b249810    
  
author   : David Steele <[email protected]>    
date     : Sat, 25 Mar 2023 14:07:31 +0700    
  
committer: David Steele <[email protected]>    
date     : Sat, 25 Mar 2023 14:07:31 +0700    

Click here for diff

Each call to lockAcquireP() passed enough information to initialize the lock system. This was somewhat inefficient and as locks become more complicated it will lead to more code duplication. Since a process can only take one type of lock it makes sense to do most of the initialization up front.  
  
Also reduce the log level of lockRelease() since it is only called at exit and the lock will be released in any case.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/remote/remote.c
M src/common/lock.c
M src/common/lock.h
M src/config/load.c
M test/define.yaml
M test/src/common/harnessBackup.c
M test/src/common/harnessConfig.c
A test/src/common/harnessLock.c
A test/src/common/harnessLock.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/exitTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/lockTest.c

Convert lockAcquire() to lockAcquireP().

commit   : f1caecc4ff38557cf81314738ac83d420d8bcf23    
  
author   : David Steele <[email protected]>    
date     : Fri, 24 Mar 2023 10:34:42 +0800    
  
committer: David Steele <[email protected]>    
date     : Fri, 24 Mar 2023 10:34:42 +0800    

Click here for diff

This makes a few current parameters optional and allows for more optional parameters with less code churn.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/remote/remote.c
M src/common/lock.c
M src/common/lock.h
M src/config/load.c
M test/src/common/harnessBackup.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/common/lockTest.c

Add reference filter and output to manifest command.

commit   : c8ec114c8cb392d227fa267c62d11cc6b4fb9677    
  
author   : David Steele <[email protected]>    
date     : Tue, 21 Mar 2023 12:29:45 +0800    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Mar 2023 12:29:45 +0800    

Click here for diff

This allows the file list to be filtered by reference. The reference is output when it is not the default reference for the backup.  

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/manifest/manifest.c
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M test/src/module/command/manifestTest.c

Clear buffer limit when buffer is limited to allocated size.

commit   : 1807bfcff5550bb40fa4f92dee1e15580a0b752d    
  
author   : David Steele <[email protected]>    
date     : Tue, 21 Mar 2023 12:22:47 +0800    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Mar 2023 12:22:47 +0800    

Click here for diff

This minor optimization automatically clears the limit flag when limit is set to allocated size. This has no impact on the current code but will simplify a future commit where a conditional bufLimitClear() is being used.  

M src/common/type/buffer.c
M test/src/module/common/typeBufferTest.c

Begin v2.46 development.

commit   : 5b5786c082b698e637bd44b9f06fbca5e4e4cc21    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Mar 2023 10:25:44 +0800    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Mar 2023 10:25:44 +0800    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h