pgBackRest 2.57 commit log · Life with Open Source

21 Oct 2024, 08:57 by Yogesh Sharma

`v2.54.0: Target Time for Versioned Storage`

commit   : 1e5f46b7d9626eadfcf99d08cb984a74085ee0c8    
  
author   : David Steele <[email protected]>    
date     : Mon, 21 Oct 2024 08:57:43 +0300    
  
committer: David Steele <[email protected]>    
date     : Mon, 21 Oct 2024 08:57:43 +0300

Click here for diff

NOTE TO PACKAGERS: This is last feature release to support the autoconf/make build. Please migrate to meson if you have not already done so. 2.54.X patch releases (if any) will continue to support autoconf/make.  
  
Bug Fixes:  
  
* Fix PostgreSQL query performance for large datasets. (Fixed by Thibault Vincent, David Steele. Reviewed by David Christensen, Antoine Millet. Reported by Antoine Millet.)  
  
Features:  
  
* Allow repositories on versioned storage to be read at a target time. (Reviewed by Stefan Fercot, David Christensen.)  
* Allow requested standby backup to proceed with no standby. (Reviewed by Stefan Fercot.)  
  
Improvements:  
  
* Summarize backup reference list for info command text output. (Contributed by Stefan Fercot. Reviewed by David Steele.)  
* Refresh web-id token for each S3 authentication. (Contributed by Brent Graveland. Reviewed by David Steele.)  
* Correctly display current values for indexed options in help. (Reviewed by David Christensen.)  
* Save backup.info only when contents have changed. (Reviewed by Stefan Fercot.)  
* Remove limitation on reading files in parallel during restore. (Reviewed by David Christensen.)  
* Improve SFTP error messages. (Contributed by Reid Thompson. Reviewed by David Steele.)  
  
Documentation Features:  
  
* Add performance tuning section to user guide. (Reviewed by Stefan Fercot.)  
  
Documentation Improvements:  
  
* Clarify source for data_directory. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Matthias.)  
* Better logic for deciding when a summary should be lower-cased. (Suggested by Daniel Westermann.)

M CONTRIBUTING.md
M README.md
M doc/RELEASE.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
R088 doc/xml/release/2024/2.54.xml doc/xml/release/2024/2.54.0.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

`Update CI tests for ppc64le/s390x architectures.`

commit   : 7effca0d0995844b3b3b89d568c3c8cee7b9ef1e    
  
author   : David Steele <[email protected]>    
date     : Sun, 20 Oct 2024 20:28:54 +0300    
  
committer: David Steele <[email protected]>    
date     : Sun, 20 Oct 2024 20:28:54 +0300

Click here for diff

The action suddenly broken and this appears to be the best work around according to https://github.com/uraimo/run-on-arch-action/issues/155.  
  
Unfortunately the tests take almost twice and long to run, probably because the container needs to be built from scratch.

M .github/workflows/test.yml

`Clarify source for data_directory.`

commit   : 9ba2db017348d304f96ce89592fc01f29197e136    
  
author   : Stefan Fercot <[email protected]>    
date     : Wed, 16 Oct 2024 11:41:26 +0200    
  
committer: GitHub <[email protected]>    
date     : Wed, 16 Oct 2024 11:41:26 +0200

Click here for diff

M doc/xml/release/2024/2.54.xml
M doc/xml/release/contributor.xml
M doc/xml/user-guide.xml
M src/build/help/help.xml

`Update CI containers to include newest PostgreSQL patch releases.`

commit   : f76da03eb674af8c05d90abf5c2a7f246f8c9c01    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Oct 2024 12:14:12 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Oct 2024 12:14:12 +0300

Click here for diff

M test/container.yaml
M test/lib/pgBackRestTest/Common/VmTest.pm

`Move SFTP test off of PostgreSQL version tested by 32-bit.`

commit   : bd0480ca7d6e5ee56a53a2887d893c41f1a8d132    
  
author   : David Steele <[email protected]>    
date     : Wed, 16 Oct 2024 11:07:21 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 16 Oct 2024 11:07:21 +0300

Click here for diff

There have been occasional SFTP authentication failures on 32-bit. We are planning to drop 32-bit support so it does not seem worth chasing these errors down and they are likely timing issues anyway.

M test/src/module/integration/allTest.c

`Retry socket bind.`

commit   : 65dfe2407d0c358209a01f06cd4326d0a80d1acb    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Oct 2024 10:06:40 +0300    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Oct 2024 10:06:40 +0300

Click here for diff

In the case of a rapid restart it is possible that the socket may not be immediately available, so retry until it becomes available.  
  
This is particularly useful for testing where sockets are bound and released very rapidly.

M src/common/io/socket/server.c
M test/src/module/common/ioTlsTest.c

`Render command help summaries the same as option help summaries.`

commit   : ed72c6f9a1b156b64e05a8fc5e236e6568657d0a    
  
author   : David Steele <[email protected]>    
date     : Fri, 11 Oct 2024 12:05:23 +0300    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Oct 2024 12:05:23 +0300

Click here for diff

Option help summaries do not have initial capitals (except in special cases) and final periods so it makes sense to render the command summaries the same way.  
  
Use the same function for both so they are consistent.

M src/command/help/help.c
M test/src/module/command/helpTest.c

`Better logic for deciding when a summary should be lower-cased.`

commit   : eb2f279a2998d1ed893c5d8984590ab4997a5f32    
  
author   : David Steele <[email protected]>    
date     : Fri, 11 Oct 2024 11:53:34 +0300    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Oct 2024 11:53:34 +0300

Click here for diff

The old logic would lower-case PostgreSQL which looked odd. This should be more robust for anything that looks like a proper name of acronym.

M src/command/help/help.c
M test/src/module/command/helpTest.c

`Improve SFTP error messages.`

commit   : 70bda2cfb21130eb589a87cb419fb5f22a71c78a    
  
author   : Reid Thompson <[email protected]>    
date     : Thu, 10 Oct 2024 04:17:35 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 10 Oct 2024 04:17:35 -0400

Click here for diff

The numbers by themselves weren't very informative and required looking into the libssh2_sftp.h header file for the definition.

M doc/xml/release/2024/2.54.xml
M src/storage/sftp/storage.c
M test/define.yaml
M test/src/module/storage/sftpTest.c

`Fix PostgreSQL query performance for large datasets.`

commit   : c8ccaaa755993e3fd5e87148bd0a991acb3050c3    
  
author   : Thibault VINCENT <[email protected]>    
date     : Thu, 10 Oct 2024 08:48:43 +0200    
  
committer: GitHub <[email protected]>    
date     : Thu, 10 Oct 2024 08:48:43 +0200

Click here for diff

The asynchronous logic used to implement the query timeout was misusing PQisBusy(), which caused the wait handler to throttle the consumption of command results. It could introduce a large delay on a query up to `db-timeout` because of the back-off sequence.
  

  
Following the recommendation of libpq, fix by polling the client socket for data availability and then continue consuming results and checking for command busyness.

M doc/xml/release/2024/2.54.xml
M doc/xml/release/contributor.xml
M src/postgres/client.c
M test/src/common/harnessPq.c
M test/src/common/harnessPq.h
M test/src/module/postgres/clientTest.c

`Refactor Wait object to expose remaining wait as waitRemains().`

commit   : 33fa3965616b1432e5d1d59c489302fa4891a39e    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Oct 2024 17:57:52 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Oct 2024 17:57:52 +0300

Click here for diff

This is useful for code that has its own wait mechanism, e.g. poll(), but still needs a way to track overall time elapsed.  
  
To keep it simple waitRemains() is called by waitMore().

M src/common/wait.c
M src/common/wait.h
M test/src/module/common/waitTest.c

`Allow fdReady() shim to be run only once with a specified return value.`

commit   : 77ae753ef59bd40b9867be21e8519093786c3b85    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Oct 2024 17:48:00 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Oct 2024 17:48:00 +0300

Click here for diff

This allows more targeted testing than the prior (still supported) approach.

M test/src/common/harnessFd.c
M test/src/common/harnessFd.h

`Move fd module shim to io module test.`

commit   : 6ac4e8dd78f16e93300f7747c4bc8fa8e1a0558b    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Oct 2024 10:46:45 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Oct 2024 10:46:45 +0300

Click here for diff

The shim is needed in an earlier test and this is a more logical place for it anyway.

M test/define.yaml

`Shuffle PostgreSQL versions between test containers.`

commit   : 1020bc117ab2e116abb41eebd40216726a3d2561    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Oct 2024 18:15:34 +0300    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2024 18:15:34 +0300

Click here for diff

Ubuntu 20.04 has been having consistent errors starting PostgreSQL 10 so move 9.5 to this container instead. An older version makes sense with an older distro.  
  
Also move PostgreSQL 12 from RHEL 8 since this version will be EOL soon.

M test/lib/pgBackRestTest/Common/VmTest.pm

`Use systemctl replacement to generate documentation.`

commit   : b50ad48fd44ba958a6a71ac0c5fe1b113968e577    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Oct 2024 11:01:55 +0300    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2024 11:01:55 +0300

Click here for diff

Containers are notoriously unfriendly to systemctl (really systemd) but we prefer to use systemctl to make our documentation as accurate as possible. This replacement performs all the functions of systemctl without requiring systemd, which great simplifies container configuration and allows the documentation build to work in more environments.

M doc/xml/user-guide.xml

`Use output path for temp config files during documentation build.`

commit   : 1ebdcd611d1724feedb23f9de854819ec5499c9f    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Oct 2024 10:28:01 +0300    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Oct 2024 10:28:01 +0300

Click here for diff

Putting the files in the user's home directory was too Linux-centric.

M doc/lib/pgBackRestDoc/Common/DocExecute.pm

`Correctly display current values for indexed options in help.`

commit   : b3ca2e34823b28862d999f87c4d7346707952a5b    
  
author   : David Steele <[email protected]>    
date     : Sat, 5 Oct 2024 09:41:50 +0300    
  
committer: GitHub <[email protected]>    
date     : Sat, 5 Oct 2024 09:41:50 +0300

Click here for diff

The current value for an indexed option was always for the first index, e.g. pg1-path. This is likely legacy from before indexing was added (and faithfully copied over from Perl, apparently).
  

  
Fix this by enumerating the current values in the option help and displaying <multi> in the option list when more than one value exists.

M doc/xml/release/2024/2.54.xml
M src/command/help/help.c
M test/src/module/command/helpTest.c

`Finalize catalog number for PostgreSQL 17 release.`

commit   : 047e3d0ed95d882b00e2da1640e78e3ba06e1d51    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Oct 2024 19:49:15 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Oct 2024 19:49:15 +0300

Click here for diff

M src/build/postgres/postgres.yaml
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Use `uname -m` to get architecture for RHEL package in user guide.

commit   : 9f97269678091cc43a15920cf8eab34b231ae431    
  
author   : David Steele <[email protected]>    
date     : Wed, 2 Oct 2024 18:02:15 +0300    
  
committer: David Steele <[email protected]>    
date     : Wed, 2 Oct 2024 18:02:15 +0300

Click here for diff

M doc/xml/user-guide.xml

`Reduce logging for help functions.`

commit   : d4ed1884ea5ce54e05f84ab1aacdfc0385538c06    
  
author   : David Steele <[email protected]>    
date     : Tue, 24 Sep 2024 12:33:19 +0300    
  
committer: David Steele <[email protected]>    
date     : Tue, 24 Sep 2024 12:33:19 +0300

Click here for diff

Full debug/trace logging in production is unlikely to be useful but does use space in the binary.  
  
Reduce logging to be useful for testing but not be deployed in production.

M src/command/help/help.c

`Update Cirrus CI MacOS running to Sonoma.`

commit   : cd7711f5e6306778427ced8b796da09398e08015    
  
author   : David Steele <[email protected]>    
date     : Fri, 20 Sep 2024 15:58:18 +0300    
  
committer: David Steele <[email protected]>    
date     : Fri, 20 Sep 2024 15:58:18 +0300

Click here for diff

M .cirrus.yml

`Add performance tuning section to user guide.`

commit   : 278319eba3444070123d3f0f277be9234604d937    
  
author   : David Steele <[email protected]>    
date     : Fri, 20 Sep 2024 15:49:26 +0300    
  
committer: GitHub <[email protected]>    
date     : Fri, 20 Sep 2024 15:49:26 +0300

Click here for diff

We frequently tell users to enable to these options but they are spread through the documentation and not at all obvious. Hopefully putting them in the quick start will make them more visible and also provide an easy place to link.

M doc/xml/release/2024/2.54.xml
M doc/xml/user-guide.xml

`Skip command-line options in configuration reference.`

commit   : f73c6382752ca8cad43a412a03b6b5f34cf94984    
  
author   : David Steele <[email protected]>    
date     : Fri, 20 Sep 2024 15:21:38 +0300    
  
committer: David Steele <[email protected]>    
date     : Fri, 20 Sep 2024 15:21:38 +0300

Click here for diff

Options that are only valid on the command-line should not appear in the configuration reference because it implies that they can be added to pgbackrest.conf, which is not the case.  
  
Most command-line options were already excluded because they lacked a section, but a few were slipping through.

M doc/src/command/build/reference.c
M test/src/module/doc/buildTest.c

`Allow repositories on versioned storage to be read at a target time.`

commit   : b19134dccd5d0dab87b5d45b443f481df00370b1    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Sep 2024 10:51:21 +0300    
  
committer: GitHub <[email protected]>    
date     : Tue, 17 Sep 2024 10:51:21 +0300

Click here for diff

This feature allows the archive-get, info, repo-get, repo-ls, restore, and verify commands to operate at a point-in-time on versioned buckets in Azure, GCS, and S3. This allows recovery even if a repository has been accidentally or maliciously deleted or corrupted.

M doc/xml/release/2024/2.54.xml
M doc/xml/user-guide.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/repo/ls.c
M src/common/type/xml.c
M src/common/type/xml.h
M src/config/config.auto.h
M src/config/load.c
M src/config/parse.auto.c.inc
M src/storage/azure/helper.c
M src/storage/azure/read.c
M src/storage/azure/read.h
M src/storage/azure/storage.c
M src/storage/azure/storage.h
M src/storage/gcs/helper.c
M src/storage/gcs/read.c
M src/storage/gcs/read.h
M src/storage/gcs/storage.c
M src/storage/gcs/storage.h
M src/storage/gcs/storage.intern.h
M src/storage/helper.c
M src/storage/helper.h
M src/storage/info.h
M src/storage/iterator.c
M src/storage/iterator.h
M src/storage/list.c
M src/storage/list.h
M src/storage/posix/storage.c
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/read.h
M src/storage/remote/storage.c
M src/storage/remote/storage.h
M src/storage/s3/helper.c
M src/storage/s3/read.c
M src/storage/s3/read.h
M src/storage/s3/storage.c
M src/storage/s3/storage.h
M src/storage/sftp/storage.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/define.yaml
M test/src/common/harnessHost.c
M test/src/common/harnessHost.h
M test/src/common/harnessStorage.c
A test/src/common/harnessStorageHelper.c
A test/src/common/harnessStorageHelper.h
M test/src/common/harnessTime.h
M test/src/module/command/helpTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/serverTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/config/loadTest.c
M test/src/module/integration/allTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

`Remove limitation on reading files in parallel during restore.`

commit   : 46c1297be344909e900be596a865064233da9f87    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Sep 2024 15:47:23 +0300    
  
committer: GitHub <[email protected]>    
date     : Mon, 9 Sep 2024 15:47:23 +0300

Click here for diff

This restriction prevented multiple files being read from a remote simultaneously, which was not supported by the protocol. Although the limitation only applied to remotes, it was applied in all cases for testing purposes and because we planned to fix it.
  

  
Protocol command multiplexing added in df8cbc91 allows files to be read simultaneously from a remote so this restriction is no longer required.
  

  
Note that there is a test for this condition since the prior code had coverage. It might be tricky to ensure that test doesn't go away, but in general we should have enough tests in place to ensure simultaneous reads function as expected.

M doc/xml/release/2024/2.54.xml
M src/command/restore/file.c

`Allow requested standby backup to proceed with no standby.`

commit   : a42629f87ab70ffaee8d1241eb50c5ea7154a87d    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Sep 2024 12:56:52 +0300    
  
committer: GitHub <[email protected]>    
date     : Mon, 9 Sep 2024 12:56:52 +0300

Click here for diff

Add a "prefer" value to the backup-standby option to allow the backup to proceed when no standby is found. Note that this will not help if the standby is responding but fails to sync with the primary after the backup has started.
  

  
Introduce a new option modifier, bool-like, that allows a boolean option to be converted to a string or string-id option while still allowing the option to act like a boolean on the command-line, e.g. --no-backup-standby.

M doc/xml/release/2024/2.54.xml
M src/build/config/config.yaml
M src/build/config/parse.c
M src/build/config/parse.h
M src/build/config/render.c
M src/build/help/help.xml
M src/command/backup/backup.c
M src/command/check/check.c
M src/command/stanza/common.c
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/config/parse.c
M src/db/helper.c
M src/db/helper.h
M test/src/common/harnessBackup.c
M test/src/common/harnessBackup.h
M test/src/module/build/configTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/config/parseTest.c
M test/src/module/db/dbTest.c

`Save backup.info only when contents have changed.`

commit   : b454b9e403526b1f352df964d9206c27d2397df3    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Sep 2024 11:38:42 +0300    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Sep 2024 11:38:42 +0300

Click here for diff

This prevents backup.info from being saved again when expire does not make any changes.  
  
More importantly, as we look to support versioning on object stores, it will be much easier to determine a good point-in-time to use for restore if there are no extraneous saves of backup.info.

M doc/xml/release/2024/2.54.xml
M src/info/infoBackup.c
M src/info/infoBackup.h
M test/src/module/command/expireTest.c
M test/src/module/info/infoBackupTest.c

`Remove invalid const keywords from info/infoBackup module.`

commit   : 83802ded8c5560a097757c420d0753c50d0edbb3    
  
author   : David Steele <[email protected]>    
date     : Sun, 1 Sep 2024 12:51:27 +0700    
  
committer: David Steele <[email protected]>    
date     : Sun, 1 Sep 2024 12:51:27 +0700

Click here for diff

In these functions infoBackup was marked as const even though it was modified in the function. This was allowed by the compiler because the infoBackup struct was not being directly modified but it still goes against our coding conventions.

M src/command/expire/expire.c
M src/info/infoBackup.c
M src/info/infoBackup.h

`Refresh web-id token for each S3 authentication.`

commit   : 95a4b9e10a3f7a26ebaae22fc6abd79cee14189a    
  
author   : Brent Graveland <[email protected]>    
date     : Sat, 31 Aug 2024 22:21:56 -0600    
  
committer: GitHub <[email protected]>    
date     : Sat, 31 Aug 2024 22:21:56 -0600

Click here for diff

The token file pointed to by the AWS_WEB_IDENTITY_TOKEN_FILE environment variable was read once at startup, but for long operations the token might expire before completion.
  

  
Instead read the token on each S3 authentication so the current token is always used.

M doc/xml/release/2024/2.54.xml
M src/storage/s3/helper.c
M src/storage/s3/storage.c
M src/storage/s3/storage.h
M test/src/module/storage/s3Test.c

`Refactor storage/remote module to move more repository storage tests.`

commit   : d204bac75d3020251c52c9b7892b47c5b3e7f018    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 16:23:55 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 16:23:55 +0700

Click here for diff

Move tests missed in 3b3886f3.

M test/src/module/storage/remoteTest.c

`Update comment in storage/list module.`

commit   : 0b4371f3ecead992e6140251c658e9fed56a8f37    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 16:02:04 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 16:02:04 +0700

Click here for diff

This aligns better with other comments in the area.

M src/storage/list.c

`Refactor storage/remote module to move repository storage tests.`

commit   : 3b3886f3b81776adfe3fd4970e977f56d0c2b02f    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 16:00:07 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 16:00:07 +0700

Click here for diff

Move tests that use functions that are not valid for non-Posix repositories to pg storage.  
  
This allows for a test storage driver that only implements functions required for repositories.

M test/src/module/storage/remoteTest.c

`Simplify command/repo tests.`

commit   : b3fed2cfcf4562dde52be28c35e20d7d38171564    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 15:44:43 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 15:44:43 +0700

Click here for diff

Reduce the number of tests that use links and special files. This simplifies future versioning tests.

M test/src/module/command/repoTest.c

`Add hrnSleepRemainder() to time test harness.`

commit   : 94e21893e8b43ef9080246feba5fa89d39e67f0c    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 15:39:03 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Aug 2024 15:39:03 +0700

Click here for diff

This function replaces some ad hoc implementations in the tests.

M test/src/common/harnessTime.c
M test/src/common/harnessTime.h
M test/src/module/command/backupTest.c
M test/src/module/command/expireTest.c

`Improve reporting of partial reads in server test harness.`

commit   : c40dd5ae41ae800a031715a60c91dd839b39905d    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Aug 2024 11:07:52 +0700    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Aug 2024 11:07:52 +0700

Click here for diff

Previously partial reads would be reported as an EOF which was not very helpful.  
  
Also update error formatting to make requests easier to compare.

M test/src/common/harnessServer.c

`Refactor interface/driver getters for IoRead/Write and StorageRead/Write.`

commit   : 2ce01e57d3f2f294efe9a9ac6c7d357342796a13    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Aug 2024 20:57:06 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Aug 2024 20:57:06 +0700

Click here for diff

Add getters where needed and remove the unused driver member from the StorageRead/Write objects. The new getters are only required for testing but they don't compromise the core code.

M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/restore/restore.c
M src/common/io/read.c
M src/common/io/read.intern.h
M src/common/io/write.c
M src/common/io/write.h
M src/common/io/write.intern.h
M src/storage/azure/read.c
M src/storage/azure/write.c
M src/storage/gcs/read.c
M src/storage/gcs/write.c
M src/storage/posix/read.c
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/read.h
M src/storage/read.intern.h
M src/storage/remote/read.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/write.c
M src/storage/sftp/read.c
M src/storage/sftp/write.c
M src/storage/write.c
M src/storage/write.h
M src/storage/write.intern.h
M test/src/module/common/ioTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c
M test/src/module/storage/sftpTest.c

`Improve TEST_STORAGE_GET() to allow testing of missing files.`

commit   : a71d88481c60178de1c10dac2a0d306959675999    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Aug 2024 20:34:40 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Aug 2024 20:34:40 +0700

Click here for diff

Previously a missing file would always throw an error. Now there is the option to return NULL for missing files.

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

`Improve file removal performed after each unit test.`

commit   : 4eab64fcd926c2b5741d4b44f8b9d9144bf1e3a5    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Aug 2024 20:14:32 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Aug 2024 20:14:32 +0700

Click here for diff

The rm would miss files beginning with a dot which are being used going forward for the versioning test driver.  
  
Instead use find since it will delete all files.

M test/src/common/harnessTest.c

`Merge v2.53.1 release.`

commit   : f6aaa3672b851fa8e70942f332a633f3b42babb7    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Aug 2024 11:03:24 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Aug 2024 11:03:24 +0700

Click here for diff

M CONTRIBUTING.md
M README.md
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
A doc/xml/release/2024/2.53.1.xml
M doc/xml/release/2024/2.54.xml
M test/code-count/file-type.yaml

`Move getEpoch() to cvtZToTime() and refactor.`

commit   : 2b29fe9337e9b2cbde26915c8bfb999b4010560c    
  
author   : David Steele <[email protected]>    
date     : Sat, 17 Aug 2024 12:58:55 +0700    
  
committer: David Steele <[email protected]>    
date     : Sat, 17 Aug 2024 12:58:55 +0700

Click here for diff

This function will be needed elsewhere going forward and in any case it makes sense to move this out of the restore module.  
  
Refactor to avoid dependency on a regular expression.

M src/command/restore/restore.c
M src/common/type/convert.c
M src/common/type/convert.h
M test/define.yaml
M test/src/module/command/restoreTest.c
M test/src/module/common/typeConvertTest.c

`Fix segfault on delayed connection errors.`

commit   : 60f96429b8f58827e3f2d6cd96dc1f6e1725c90c    
  
author   : David Steele <[email protected]>    
date     : Fri, 16 Aug 2024 10:56:07 +0700    
  
committer: GitHub <[email protected]>    
date     : Fri, 16 Aug 2024 10:56:07 +0700

Click here for diff

Connection errors could cause a segfault if the error was delayed enough to pass the initial call to sckClientOpenWait() and the error was instead thrown by a subsequent call to sckClientOpenWait(), which was not correctly initializing a variable required for error handling.
  

  
While this can be produced fairly easily in a test environment, I was unable to craft a unit test to hit this exact condition, probably due to timing. The new code still has full coverage and I added several comments to help prevent regressions.

M doc/xml/release/2024/2.54.xml
M src/common/io/socket/client.c

`Skip local repository duplicate check for SFTP.`

commit   : 04ef43d9ed3e1900a85f0df9475ce6dc04dc9c5f    
  
author   : Reid Thompson <[email protected]>    
date     : Tue, 13 Aug 2024 01:45:24 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 13 Aug 2024 01:45:24 -0400

Click here for diff

This check does not make sense for SFTP since the repository will never be local.

M doc/xml/release/2024/2.54.xml
M src/config/load.c
M test/src/module/config/loadTest.c

`PostgreSQL 17beta3 support.`

commit   : 80c9b3001c8f4689837d46783d348a2c839645c0    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Aug 2024 11:53:12 +0800    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Aug 2024 11:53:12 +0800

Click here for diff

This release changed the control and WAL format, which is very unusual for a beta. Update control and WAL versions/structs to match.

M doc/xml/release/2024/2.54.xml
M src/postgres/interface/version.vendor.h
M test/container.yaml
M test/src/module/postgres/interfaceTest.c

`Fix permissions when restore run as root user.`

commit   : ed9b0c260a6e1f05a17d5d2b86c274afdd2d1b05    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Aug 2024 11:43:05 +0800    
  
committer: GitHub <[email protected]>    
date     : Tue, 13 Aug 2024 11:43:05 +0800

Click here for diff

When restore was run as the root user the pg_control file would end up with root permissions. This bug was introduced in e634fd8. Fix this by directly overwriting the pg_control temp file rather than doing an atomic write that updates permissions. Also update other parameters to more closely match similar calls.
  

  
There was also an adjacent error where restore as the root user would fail if the base path did not exist. Fix this by ignoring the missing path since it will be created later and this logic is just trying to find an alternate user for permissions if the user in the manifest does not exist.

M doc/xml/release/2024/2.54.xml
M doc/xml/release/contributor.xml
M src/command/restore/restore.c
M test/src/common/harnessHost.c
M test/src/common/harnessHost.h
M test/src/module/integration/allTest.c

`Use Posix driver to complete dummy storage driver.`

commit   : 57663536493bfa917e4b1c37d221697a0ddb7a73    
  
author   : David Steele <[email protected]>    
date     : Fri, 2 Aug 2024 15:47:30 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 2 Aug 2024 15:47:30 +0700

Click here for diff

Instead of stub functions, use the Posix driver to make the dummy storage driver interface more functional.

M test/define.yaml
M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c

`Summarize backup reference list for info command text output.`

commit   : b306f83493cc859ad15c8f175fc6db2321b79340    
  
author   : Stefan Fercot <[email protected]>    
date     : Wed, 31 Jul 2024 13:53:02 +0200    
  
committer: GitHub <[email protected]>    
date     : Wed, 31 Jul 2024 13:53:02 +0200

Click here for diff

The backup reference list can be very long so it seems better to summarize the list by default for text output and keep the full list when --set is specified.

M doc/xml/release/2024/2.54.xml
M src/build/help/help.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c

`Fix typo in user guide.`

commit   : c42d484c9d76dfb2baa7b9b462870441e8ad37a0    
  
author   : Aleksey Tsalolikhin <[email protected]>    
date     : Tue, 30 Jul 2024 00:48:32 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 30 Jul 2024 00:48:32 -0500

Click here for diff

M doc/xml/user-guide.xml

`Refactor config parse to remove none command, add version/help options.`

commit   : 8d6bceb5416a702d010fd5db4be9d6dc294cb1d9    
  
author   : David Steele <[email protected]>    
date     : Tue, 23 Jul 2024 16:39:02 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Jul 2024 16:39:02 +0700

Click here for diff

The none command was a bit confusing since it was only valid when parsing failed but still needed to be added to various switches and logic. Replace with cfgInited() which should make it clearer what state configuration is in.  
  
Make the default command help and convert --version and --help to real options.  
  
Combine version and help output into a single function to simplify processing in main.  
  
Additional reformatting and a bit of refactoring.

M .github/workflows/symbol.out
M doc/src/build/config/config.yaml
M doc/src/build/help/help.xml
M doc/src/config/load.c
M doc/src/main.c
M src/build/config/config.yaml
M src/build/config/render.c
M src/build/help/help.xml
M src/command/command.c
M src/command/exit.c
M src/command/help/help.c
M src/config/config.auto.h
M src/config/config.c
M src/config/config.intern.h
M src/config/load.c
M src/config/parse.auto.c.inc
M src/config/parse.c
M src/main.c
M test/src/build/config/config.yaml
M test/src/build/help/help.xml
M test/src/common/harnessConfig.c
M test/src/common/harnessProtocol.c
M test/src/config/load.c
M test/src/main.c
M test/src/module/build/configTest.c
M test/src/module/command/exitTest.c
M test/src/module/command/helpTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c

`Simplify main() functions in core, doc, and test.`

commit   : 6c757366c2c7221a9fd571495fa2ec82b4d4da2f    
  
author   : David Steele <[email protected]>    
date     : Tue, 23 Jul 2024 11:50:12 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Jul 2024 11:50:12 +0700

Click here for diff

This refactor simplifies the main() functions and puts the more commonly run commands first.  
  
For core main() also remove code duplication in local/remote role handling.

M doc/src/main.c
M src/main.c
M test/src/main.c

`Do not allow help for command roles other than main.`

commit   : faee89206776287a2ba000c43f28ac6479d33689    
  
author   : David Steele <[email protected]>    
date     : Tue, 23 Jul 2024 11:10:54 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 23 Jul 2024 11:10:54 +0700

Click here for diff

Previously requesting help for roles other than main would result in a segfault since help.auto.c.inc does not include such help.  
  
Prevent this by erroring when a user requests help for roles other than main.  
  
This is a bug but it is unlikely to have ever been seen in the field and in any case it has a very low severity since it only impacts a corner-case for the help command.

M src/config/parse.c
M test/src/module/config/parseTest.c

`Remove internal repo-create command.`

commit   : ee70c2e26e63ee6299de5a9ca20a3d1ddea036b1    
  
author   : David Steele <[email protected]>    
date     : Mon, 22 Jul 2024 14:17:47 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 22 Jul 2024 14:17:47 +0700

Click here for diff

This command was used by the Perl integration tests to create buckets for storage types that required it. Now that the integration tests are written in C they can simply use the same code to create buckets.  
  
The command was also used in the documentation but there it seems more appropriate to use the corresponding vendor CLI.

M doc/xml/user-guide.xml
M src/Makefile.in
M src/build/config/config.yaml
M src/build/help/help.xml
D src/command/repo/create.c
D src/command/repo/create.h
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/main.c
M src/meson.build
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/common/harnessHost.c
M test/src/module/command/repoTest.c

`Update protocol and storage/remote modules to recent coding standards.`

commit   : 55ca41e13784ef5fdafdb02444969ceb1a699744    
  
author   : David Steele <[email protected]>    
date     : Mon, 22 Jul 2024 12:43:54 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 22 Jul 2024 12:43:54 +0700

Click here for diff

Add const as appropriate.

M src/protocol/client.c
M src/protocol/helper.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/server.c
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M test/src/common/harnessProtocol.c

`Protocol command multiplexing.`

commit   : df8cbc91c348bc2f746767c87ac5626460fb6f39    
  
author   : David Steele <[email protected]>    
date     : Mon, 22 Jul 2024 11:48:32 +0700    
  
committer: GitHub <[email protected]>    
date     : Mon, 22 Jul 2024 11:48:32 +0700

Click here for diff

Previously it was not possible to read or write two files at the same time on the same remote because the protocol was entirely taken over by the read or write command. Multiple reads are required to make restores efficient when a list of bundled files is being read but blocks need to be retrieved from a separate file or a different part of the same file.
  

  
Improve that situation with sessions that allow related commands to be run with shared state. Also break read/write into separate requests (rather than pushing all data at once) so they can be multiplexed.
  

  
The disadvantage for read/write is that they now require more back and forth to transfer a file. This is mitigated by sending asynchronous read/write requests to keep both server and client as busy as possible. Reads that can fit into a single buffer are optimized to transfer in a single command. Reads that transfer the entire file can also skip the close command since it is implicit on end-of-file.
  

  
These changes allow the protocol to be simplified to provide one response per request, which makes the data end message obsolete. Any data sent for the request is now added to the parameters so no data needs to be sent separately to the server outside the request parameters.
  

  
Also update the Db protocol to use the new sessions. Previously this code had tracked its own sessions.

M doc/xml/release/2024/2.54.xml
M src/Makefile.in
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/archive/get/protocol.h
M src/command/archive/push/protocol.c
M src/command/archive/push/protocol.h
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/protocol.c
M src/command/backup/protocol.h
M src/command/remote/remote.c
M src/command/restore/protocol.c
M src/command/restore/protocol.h
M src/command/restore/restore.c
M src/command/verify/protocol.c
M src/command/verify/protocol.h
M src/command/verify/verify.c
M src/config/protocol.c
M src/config/protocol.h
M src/db/db.c
M src/db/protocol.c
M src/db/protocol.h
M src/meson.build
M src/postgres/client.c
M src/postgres/client.h
M src/protocol/client.c
M src/protocol/client.h
D src/protocol/command.c
D src/protocol/command.h
M src/protocol/helper.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M src/protocol/server.c
M src/protocol/server.h
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/common/harnessDebug.h
M test/src/common/harnessProtocol.c
M test/src/module/command/backupTest.c
M test/src/module/db/dbTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

`Begin v2.54 development.`

commit   : e7f4e8d8004f609ab0edda5d7183ccb3e6c55b6a    
  
author   : David Steele <[email protected]>    
date     : Mon, 22 Jul 2024 09:42:33 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 22 Jul 2024 09:42:33 +0700

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
A doc/xml/release/2024/2.54.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml