pgBackRest 2.41 commit log

v2.40: OpenSSL 3 Support

commit   : 1ff531090b9d1eccd9107592398148a51486c9f8    
  
author   : David Steele <[email protected]>    
date     : Mon, 18 Jul 2022 09:32:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 18 Jul 2022 09:32:30 -0400    

Click here for diff

NOTE TO PACKAGERS: An experimental meson build has been added but packagers should continue to use the autoconf/make build for the foreseeable future.  
  
Improvements:  
  
* OpenSSL 3 support. (Reviewed by Stephen Frost.)  
* Create snapshot when listing contents of a path. (Reviewed by John Morris, Stephen Frost.)  
* Force target-timeline=current when restore type=immediate. (Reviewed by Stephen Frost.)  
* Truncate files during delta restore when they are larger than expected. (Reviewed by Stephen Frost.)  
* Disable incremental manifest save when resume=n. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Set backup percent complete to zero before copy start. (Contributed by Reid Thompson. Reviewed by David Steele.)  
* Use S3 IsTruncated flag to determine list continuation. (Reviewed by John Morris, Soulou. Suggested by Christian Montagne.)  
  
Documentation Bug Fixes:  
  
* Skip internal options in the configuration reference. (Reported by Francisco Miguel Biete.)  
  
Documentation Improvements:  
  
* Add link to PostgreSQL configuration in repository host section. (Reviewed by Stefan Fercot. Suggested by Julien Cigar.)  
  
Test Suite Improvements:  
  
* Add experimental Meson build. (Reviewed by Eli Schwartz, Sam Bassaly.)  
* Allow any path to be passed to the --test-path option. (Contributed by Andrey Sokolov. Reviewed by David Steele.)  
* Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG. (Contributed by Andrey Sokolov. Reviewed by David Steele.)  

M CONTRIBUTING.md
M README.md
M doc/RELEASE.md
M doc/release.pl
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Replace AC_PROG_CC_C99 configure macro with AC_PROG_CC.

commit   : 1df1e0281b94a06ff0f16622e2cbb76598d430d1    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Jul 2022 17:13:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Jul 2022 17:13:33 -0400    

Click here for diff

AC_PROG_CC_C99 is now obsolete.  
  
The AC_PROG_CC macro does not guarantee C99 compliance but we add an option to check for that a bit later.  

M src/build/aclocal.m4
M src/build/configure.ac
M src/configure

Force target-timeline=current when restore type=immediate.

commit   : 364af1635d40431276cc6fd87d6ddd689b611e09    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Jul 2022 08:26:03 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 14 Jul 2022 08:26:03 -0400    

Click here for diff

Explicitly set target timeline to "current" when type=immediate and PostgreSQL >= 12. We do this because type=immediate means there won't be any actual attempt to change timelines, but if we leave the target timeline as the default of "latest" then PostgreSQL might fail to restore because it can't reach the "latest" timeline in the repository from this backup.
  

  
This is really a PostgreSQL bug and will hopefully be addressed there, but we'll handle it here for older versions, at least until they aren't really seen in the wild any longer.
  

  
PostgreSQL < 12 defaults to "current" (but does not accept "current" as a parameter) so no need set it explicitly.

M doc/xml/release.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Create snapshot when listing contents of a path.

commit   : 75623d4583bfdaccb3a19b935fd04d744b363923    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jul 2022 17:21:39 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 8 Jul 2022 17:21:39 -0400    

Click here for diff

Previously a callback was used to list path contents and if no sort was specified then a snapshot was not required. When deleting files from the path some filesystems could omit files that still existed, which meant the path could not be removed.
  

  
Filter . out of lists in the Posix driver since this special entry was only used by test code (and filtered everywhere in the core code).
  

  
Also remove callbacks from the storage interface and replace with an iterator that should be easier to use and guarantees efficient use of the snapshots.

M doc/xml/release.xml
M src/Makefile.in
M src/command/backup/backup.c
M src/command/repo/ls.c
M src/command/restore/restore.c
A src/common/type/blob.c
A src/common/type/blob.h
M src/info/manifest.c
M src/meson.build
M src/storage/azure/storage.c
M src/storage/gcs/storage.c
A src/storage/iterator.c
A src/storage/iterator.h
A src/storage/list.c
A src/storage/list.h
M src/storage/posix/storage.c
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/define.yaml
M test/src/common/harnessStorage.c
M test/src/module/command/backupTest.c
A test/src/module/common/typeBlobTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Fix typo.

commit   : f9ac53db92574e8b499187ee4f2a7258bc2f4794    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jul 2022 14:38:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 8 Jul 2022 14:38:14 -0400    

Click here for diff

M doc/xml/release.xml

commit   : 74a4ac801db921f7b11b63b9f927228c6a6af861    
  
author   : David Steele <[email protected]>    
date     : Fri, 8 Jul 2022 11:13:55 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 8 Jul 2022 11:13:55 -0400    

Click here for diff

This should make the documentation clearer when starting from this section.

M doc/xml/release.xml
M doc/xml/user-guide.xml

Meson builds work on version 0.45.

commit   : 0eccbc8bf47073ab8ec1f13fed9865d0bd7c3635    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Jul 2022 18:17:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 Jul 2022 18:17:52 -0400    

Click here for diff

v0.45 ships with Ubuntu 18.04, which is currently the oldest distro we support. We may never do a Meson release on Ubuntu 18.04 but this allows us to start running unit tests with Meson in the meantime.  
  
Some more granular options are not available so we use buildtype in more places.  
  
The check for a in-tree autoconf/make build had to be removed since the filesystem APIs are not available.  
  
Finally, alias_target was removed. This means that full paths must be used for build targets, which does not seem too bad. For instance, test/src/test-pgbackrest must now be used as a build target instead of simple test-pgbackrest.  

M meson.build
M src/meson.build
M test/src/command/help/meson.build
M test/src/meson.build
M test/test.pl

Rename strTrunc() to strTruncIdx() and add strTrunc().

commit   : 72960bbf179b4e8051bd4fc50be57d95159e9e13    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Jul 2022 09:55:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 Jul 2022 09:55:32 -0400    

Click here for diff

strTrunc() is now shorthand for truncating to index 0. This is convenient when a string is being reused.  

M src/command/verify/verify.c
M src/common/ini.c
M src/common/type/string.c
M src/common/type/string.h
M src/config/common.c
M src/config/parse.c
M test/src/module/common/typeStringTest.c

Add comment to indicate that type is in a different info level.

commit   : cd25cec1471dde60f85404744252b675efff2522    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Jul 2022 14:34:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Jul 2022 14:34:31 -0400    

Click here for diff

M src/storage/info.h

Reorder null user/group checks in remote protocol and add tests.

commit   : 1c0bf0b15da8c0d15cc639851ff1d1bd05b51d06    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Jul 2022 14:28:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Jul 2022 14:28:40 -0400    

Click here for diff

Coverage for these checks was dependent on the order the files were read from disk, which made the tests fragile.  
  
Rearrange the checks and add a test that won't depend on order.  

M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M test/src/module/storage/remoteTest.c

Update contributor name.

commit   : 326d152a14121cb298089847a3694aa9f50bcbd2    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Jul 2022 06:58:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Jul 2022 06:58:19 -0400    

Click here for diff

M doc/xml/release.xml

Use S3 IsTruncated flag to determine list continuation.

commit   : 845d82f6825d4101e6a465a435ef056ff6ec15a2    
  
author   : David Steele <[email protected]>    
date     : Fri, 1 Jul 2022 20:32:10 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 1 Jul 2022 20:32:10 -0400    

Click here for diff

Previously we were just checking for the existence of NextContinuationToken, which the S3 documentation indicates will not be present when the list is not truncated. However, recent versions of Scality send a blank NextContinuationToken when IsTruncated is false. Sending the blank continuation token back causes Scality to send another blank continuation token and an infinite loop occurs.
  

  
Instead use IsTruncated (which is required to be present) to determine whether NextContinuationToken should be present. Error if NextContinuationToken is then missing or empty, since an empty token caused an infinite loop with the Scality server (which arguably should have errored when passed an empty token).

M doc/xml/release.xml
M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Replace hrnStorageInfoListCallback() with TEST_STORAGE_LIST() in tests.

commit   : 61ca9b58a000aa8789ef3267682519bdb15e8a6b    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Jun 2022 14:15:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Jun 2022 14:15:08 -0400    

Click here for diff

The TEST_STORAGE_LIST() macro is more robust and hides the callback mechanism from the caller.  
  
Add features to TEST_STORAGE_LIST() that hrnStorageInfoListCallback() had.  
  
Update tests to use the abbreviated type output (e.g. path/) generated by TEST_STORAGE_LIST().  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/restoreTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Generate -fmacro-prefix-map to strip relative path from __FILE__.

commit   : 59f148bf6ea8c090d4d65ce5b135a71501e8ae22    
  
author   : David Steele <[email protected]>    
date     : Sun, 26 Jun 2022 08:42:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 26 Jun 2022 08:42:43 -0400    

Click here for diff

This provides reproducible builds and minimizes the file path in debug messages, just like an in-tree make build.  
  
For test source, prefix with test/ in case there are any module name collisions.  

M meson.build

Add experimental unit test harness written in C.

commit   : f863fc98886f4d6c2803b9a2396539597d0d7be2    
  
author   : David Steele <[email protected]>    
date     : Thu, 23 Jun 2022 12:20:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 23 Jun 2022 12:20:56 -0400    

Click here for diff

Having the test harness in C will allow us to remove duplicated Perl code and test on systems where Perl support is not present.  
  
Custom harnesses and shims are currently not implemented, which means only the following tests in the common module will run: error, stack-trace, type-convert, assert-on, mem-context, time, encode, type-object, type-string, type-list, type-buffer, type-variant, reg-exp, log.  
  
The experimental test harness is being committed with partial functionality so it can be used in Windows development. The remaining features will follow as needed.  

M meson.build
M src/build/common/yaml.h
M src/build/config/main.c
M src/build/help/main.c
M src/meson.build
A test/src/build/config/config.yaml
A test/src/build/help/help.xml
A test/src/command/help/meson.build
A test/src/command/test/build.c
A test/src/command/test/build.h
A test/src/command/test/define.c
A test/src/command/test/define.h
A test/src/command/test/test.c
A test/src/command/test/test.h
M test/src/common/harnessFork.h
A test/src/config/load.c
A test/src/config/load.h
A test/src/config/meson.build
A test/src/main.c
A test/src/meson.build
M test/src/test.c

Used DEBUG instead of NDEBUG in common/stackTrace module.

commit   : b7a1b3ec2c16fde6a201f404a27d8343595900b6    
  
author   : David Steele <[email protected]>    
date     : Thu, 23 Jun 2022 10:57:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 23 Jun 2022 10:57:58 -0400    

Click here for diff

In some testing cases these might not be in sync, which causes unpredictable behavior.  

M src/common/stackTrace.h

Add cast to handle compilers (e.g. MSVC) that coerce to signed int.

commit   : 29d2f0f9fda4dbba0dc5eca6f84368a6f655bd28    
  
author   : Sam Bassaly <[email protected]>    
date     : Wed, 22 Jun 2022 15:52:30 +0200    
  
committer: GitHub <[email protected]>    
date     : Wed, 22 Jun 2022 15:52:30 +0200    

Click here for diff

MSVC changes the sign in this case, presumable because of the subtraction. Cast so that MSVC does not also trigger a mixed sign warning.

M doc/xml/release.xml
M src/common/log.c

Disable meson for all but debug builds.

commit   : 9fd85a104ad56061b028afd9057ce8c1d636d3dd    
  
author   : David Steele <[email protected]>    
date     : Tue, 21 Jun 2022 09:50:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Jun 2022 09:50:38 -0400    

Click here for diff

The meson builds are still experimental so for now the configure/make build process is preferred for release builds. This message should help prevent any automated build systems from picking up meson instead.  

M meson.build
M test/test.pl

Update meson.build include comment to be more general.

commit   : 665da12ae7f519077512e60ce3a5ec6c6a9997fd    
  
author   : David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 16:41:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 16:41:48 -0400    

Click here for diff

Eventually multiple subdirs will be included so update the comment to reflect that.  

M meson.build

Use constants rather than replacements when possible in test.c.

commit   : 5ecae90f0254ef752d8323ea5be3b99673ec46f7    
  
author   : David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 16:31:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 16:31:48 -0400    

Click here for diff

Some of the replacements that were being done already existed as constants, so use the constants instead.  
  
Also fix a minor formatting error introduced when testAdd() was renamed to hrnAdd().  

M test/src/test.c

Add strReplace().

commit   : dab1e4b6c66018c03c6e01309c41a89ef82ff5bc    
  
author   : David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 16:27:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 16:27:31 -0400    

Click here for diff

Allows substrings to be replaced with another string.  

M src/common/type/string.c
M src/common/type/string.h
M test/src/module/common/typeStringTest.c

Add higher level YAML functions.

commit   : fb9acc1c93d504d366493bc42038f15faa80300d    
  
author   : David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 16:17:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 16:17:02 -0400    

Click here for diff

These functions make parsing YAML simpler.  

M src/build/common/yaml.c
M src/build/common/yaml.h
M test/src/module/build/commonTest.c

Move protocol module from common to command.

commit   : 55bcb933eee753a0cdd13f9861a00c33d194c1e7    
  
author   : David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 11:17:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 17 Jun 2022 11:17:52 -0400    

Click here for diff

This module has dependencies on command/command so it does not make sense for it to be in the common module. Also move protocolFree() to main() since this is a very large dependency.  
  
Adjust the tests so command/exit can be tested later. This is a bit messy but will get adjusted as we improve the test harness.  

M src/Makefile.in
R098 src/common/exit.c src/command/exit.c
R094 src/common/exit.h src/command/exit.h
M src/command/server/server.c
M src/main.c
M src/meson.build
M test/code-count/file-type.yaml
M test/define.yaml
R097 test/src/module/common/exitTest.c test/src/module/command/exitTest.c
M test/src/module/command/serverTest.c

Fix continuation character spacing.

commit   : eb72a80b471145d97ac4170e3215f7c67aa83082    
  
author   : Stefan Fercot <[email protected]>    
date     : Wed, 15 Jun 2022 09:06:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 15 Jun 2022 09:06:25 -0400    

Click here for diff

M test/src/module/info/manifestTest.c

Fix hard-coded WAL size assertion in walSegmentNext().

commit   : 716bba5800d504a76d4bdc99588448824be3d83f    
  
author   : Alexey Gordeev <[email protected]>    
date     : Tue, 14 Jun 2022 17:13:22 +0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 14 Jun 2022 17:13:22 +0500    

Click here for diff

PG_WAL_SEGMENT_SIZE_DEFAULT is used to compare and check WAL size on pre-11 installations. However, there is a hard-coded assertion in walSegmentNext() which doesn't respect PG_WAL_SEGMENT_SIZE_DEFAULT.
  

  
Update the assertion to use PG_WAL_SEGMENT_SIZE_DEFAULT.

M doc/xml/release.xml
M src/command/archive/common.c

Remove storageLocal() dependency from common/lock module.

commit   : 282edda661b5f1eb189998e7db680b91a2b8a66b    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Jun 2022 17:42:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Jun 2022 17:42:00 -0400    

Click here for diff

The storage/helper module is a very heavy dependency to introduce in the common module. Creating Posix storage objects is cheap so just do that instead.  

M src/common/lock.c

Add valgrind to Dockerfile missed in a16cf5ea.

commit   : 04f5ef25faf276e51bd546199f1197b5c70003e8    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Jun 2022 17:18:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Jun 2022 17:18:32 -0400    

Click here for diff

M test/Dockerfile

Move bldEnum() to the build/common/render module.

commit   : 79443bea3634f7dd3858b6f65838e31fb9b87c3f    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Jun 2022 16:55:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Jun 2022 16:55:07 -0400    

Click here for diff

This function will be useful for other build/test modules so extern it.  
  
Also skip the first upper-case when no prefix is provided.  

M src/build/common/render.c
M src/build/common/render.h
M src/build/config/render.c
M test/src/module/build/commonTest.c

Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG.

commit   : 1a00ab10033e9ac6e6f797627567e8b91e143432    
  
author   : Andrey Sokolov <[email protected]>    
date     : Thu, 9 Jun 2022 14:34:11 +0300    
  
committer: GitHub <[email protected]>    
date     : Thu, 9 Jun 2022 14:34:11 +0300    

Click here for diff

If DEBUG is not defined then the ASSERT() macro expands to nothing. In this case the timeBegin variable is never used and a compilation error occurs.
  

  
This test should work without DEBUG defined so use CHECK() instead of ASSERT().

M doc/xml/release.xml
M src/command/archive/common.c

Add FN_NO_RETURN macro.

commit   : 0dabf88e9d85e4db2ff65ebbb14b4320e1092a8b    
  
author   : Sam Bassaly <[email protected]>    
date     : Wed, 8 Jun 2022 23:43:23 +0200    
  
committer: GitHub <[email protected]>    
date     : Wed, 8 Jun 2022 23:43:23 +0200    

Click here for diff

Change all instances of __attribute__((__noreturn__)) to a macro in meson.build / build.auto.h.in.
  

  
As compiler attributes written in the form of __attribute__ are not supported by MSVC, this is one of several commits to make the code-base more robust and allow using MSVC-style attributes later.

M doc/xml/release.xml
M meson.build
M src/build.auto.h.in
M src/common/crypto/common.h
M src/common/error.h
M src/common/io/http/request.h

Add missing build.auto.h includes.

commit   : 8babd558bc3a5b930fd096ffb306551773f1f14d    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Jun 2022 07:26:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 8 Jun 2022 07:26:15 -0400    

Click here for diff

These are required for proper building but were probably not noticed before since they are not part of the core code.  

M src/build/common/render.c
M src/build/config/main.c
M src/build/error/main.c
M src/build/help/main.c
M src/build/postgres/main.c

Update CI to use Ubuntu 22.04 and Fedora 36.

commit   : a16cf5eac7217e79d8669fab5dff1863e26208c9    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Jun 2022 16:32:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 6 Jun 2022 16:32:20 -0400    

Click here for diff

Both have newer gcc and OpenSSL 3.  
  
Fedora 36 runs horribly slow with valgrind enabled so run the valgrind tests on Ubuntu 22.04. Fedora 36 has a newer gcc so it is still worth testing on.  

M .github/workflows/test.yml
M test/Dockerfile
M test/Vagrantfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/test.pl

OpenSSL 3 support.

commit   : 08242ee6ac93b9eb9c05cc39e3db15bbafb9052c    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Jun 2022 14:47:47 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 6 Jun 2022 14:47:47 -0400    

Click here for diff

There are two changes:
  

  
* Suppress deprecation warnings so we can build with -Werror and -Wfatal-errors. At some point we'll need to migrate to the new APIs but there does not seem to be a good reason to support two sets of code right now.
  

  
* Update the handling for unexpected EOF to handle EOF or error. The error code for EOF has changed and become harder to identify, but we probably don't care whether it is an error or EOF.

M doc/xml/release.xml
M src/common/io/tls/server.c
M src/common/io/tls/session.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/ioTlsTest.c

Automatically create PostgreSQL version interfaces.

commit   : f92ce674f79e2d6056794c1af4002c01dad93aed    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Jun 2022 13:52:56 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 6 Jun 2022 13:52:56 -0400    

Click here for diff

Maintaining the version interfaces was complicated by the fact that each interface needed to be in separate compilation unit to avoid type conflicts. This also meant that various build/test files needed to be updated to add the new interfaces.
  

  
Solve these problems by auto-generating all the interfaces into a single file. This is made possible by parsing defines and types out of the header files and creating macros to rename the types. At the end of the version interface everything is undef'd. Another benefit is that the auto-generated interfaces can be static and included directly into postgres/interface.c.
  

  
Since some code generation is now always required for tests, change --no-gen to --min-gen in test.pl.
  

  
It would also make sense to auto-generate the version defines in postgres/version.h, but that will be left for a future commit.

M .cirrus.yml
M .github/workflows/test.yml
M doc/xml/release.xml
M src/.gitignore
M src/Makefile.in
A src/build/postgres/main.c
A src/build/postgres/parse.c
A src/build/postgres/parse.h
A src/build/postgres/postgres.yaml
A src/build/postgres/render.c
A src/build/postgres/render.h
M src/meson.build
M src/postgres/interface.c
D src/postgres/interface/v090.c
D src/postgres/interface/v091.c
D src/postgres/interface/v092.c
D src/postgres/interface/v093.c
D src/postgres/interface/v094.c
D src/postgres/interface/v095.c
D src/postgres/interface/v096.c
D src/postgres/interface/v100.c
D src/postgres/interface/v110.c
D src/postgres/interface/v120.c
D src/postgres/interface/v130.c
D src/postgres/interface/v140.c
D src/postgres/interface/v150.c
D src/postgres/interface/version.h
M src/postgres/interface/version.intern.h
M src/postgres/interface/version.vendor.h
A src/postgres/meson.build
M test/Vagrantfile
M test/code-count/file-type.yaml
M test/define.yaml
A test/src/module/build/postgresTest.c
M test/test.pl

Add experimental Meson build.

commit   : b8fc20d5b8e7ca003ff39f29ea6e3ba57653f55c    
  
author   : David Steele <[email protected]>    
date     : Fri, 3 Jun 2022 14:13:56 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 3 Jun 2022 14:13:56 -0400    

Click here for diff

Meson is a new build system that offers simpler syntax and superior performance to autoconf/make. In addition, Windows is supported natively.
  

  
The Meson build appears complete, but currently is used only for auto-generation of code and the host build of pgbackrest. Some container upgrades will be required before Meson can be used for container builds.
  

  
Also patch the Debian package to force autoconf/make rather than Meson.

M .github/workflows/test.yml
M CONTRIBUTING.md
M doc/xml/contributing.xml
M doc/xml/release.xml
A meson.build
A meson_options.txt
M src/build/help/main.c
A src/command/help/meson.build
A src/meson.build
M test/Dockerfile
M test/Vagrantfile
M test/ci.pl
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
A test/patch/debian-package.patch
M test/test.pl

Remove useless command/check unit test.

commit   : 148956aed8aca201aa800af321d946094b96f176    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Jun 2022 10:13:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Jun 2022 10:13:57 -0400    

Click here for diff

This test was a placeholder and did not provide any coverage, but it did give inconsistent results on different shell versions.  

M test/define.yaml
M test/src/module/command/checkTest.c

Allow any path to be passed to the --test-path option.

commit   : 29b2a54fcc74457af5c63ffb505c5fbb24bb2ade    
  
author   : Andrey Sokolov <[email protected]>    
date     : Wed, 1 Jun 2022 00:28:58 +0300    
  
committer: GitHub <[email protected]>    
date     : Wed, 1 Jun 2022 00:28:58 +0300    

Click here for diff

A hard-coded path prevented this from working correctly.

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm

Update CodeQL to version 2.

commit   : be354c489017ea84ed825f226b50d170bec1ff68    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 May 2022 16:52:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 May 2022 16:52:26 -0400    

Click here for diff

Version 1 will be deprecated on December 12, so upgrade now to avoid nasty surprises.  

M .github/workflows/test.yml

Skip stopping PostgreSQL 9.1 in real/all integration test.

commit   : 2c38c9a56f95efea42d489792751b77acfc470e4    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 May 2022 16:43:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 May 2022 16:43:18 -0400    

Click here for diff

Stopping the cluster has started consistently running out of memory on PostgreSQL 9.1. This seems to have happened after pulling in new packages at some point so it might be build related.  
  
Stopping the cluster is not critical for 9.1 so skip it.  

M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Add .inc extension to C files included in other C files.

commit   : 2feaaeaac803d6c0193477b56dbeb622fd991156    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 May 2022 16:06:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 May 2022 16:06:41 -0400    

Click here for diff

These files were never intended to be compiled on their own so the .c extension was a bit misleading. In particular Meson does not like .c files that are not intended to be compiled independently.  
  
Leave header files as is since they are already protected against being included more than once and are never expected to be compiled.  

M .gitattributes
M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/.gitignore
M src/Makefile.in
M src/build/config/render.c
M src/build/error/error.yaml
M src/build/error/render.c
M src/build/help/render.c
M src/common/crypto/hash.c
R100 src/common/crypto/md5.vendor.c src/common/crypto/md5.vendor.c.inc
R100 src/common/error.auto.c src/common/error.auto.c.inc
M src/common/error.c
R100 src/config/parse.auto.c src/config/parse.auto.c.inc
M src/config/parse.c
M src/main.c
M src/postgres/interface/page.c
R100 src/postgres/interface/pageChecksum.vendor.c src/postgres/interface/pageChecksum.vendor.c.inc
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/src/module/build/configTest.c
M test/src/module/build/errorTest.c
M test/src/module/build/helpTest.c

Run remaining Github Actions CI on Ubuntu 20.04.

commit   : cb891fa2d402ee338b72d2c0962cc5cd49eea4f1    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 May 2022 13:02:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 May 2022 13:02:51 -0400    

Click here for diff

The s390x/ppc64le tests are already running on 20.04.  

M .github/workflows/test.yml

Skip internal options in the configuration reference.

commit   : 2643050be0dc4b7363c434f58373919bd6fdac68    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 May 2022 12:36:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 May 2022 12:36:21 -0400    

Click here for diff

Most internal options were being skipped, but not in the case where an option was marked internal for a specific command.  
  
The command-line help was not affected by this issue.  

M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/xml/release.xml

Remote repo options should not be valid for the backup command.

commit   : f0aae6ceab9a95c6d63b8955abd2592ddbbce90a    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 May 2022 12:26:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 May 2022 12:26:57 -0400    

Click here for diff

Some of the remote repo options were gated by repo-local, but the rest relied on repo-host-cmd.  
  
Remove backup from the repo-host-cmd option since none of the dependent options are valid for backup.  

M src/build/config/config.yaml
M src/config/parse.auto.c

Make all repo-* options visible for stanza create/update/delete.

commit   : a902c7808d8bf651389e2479d4622b0fd49ab09d    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 May 2022 12:24:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 May 2022 12:24:21 -0400    

Click here for diff

31c7824a allowed these commands to run remotely but neglected to remove some internal flags, which prevented all the repo-* options from being visible in the documentation.  

M src/build/config/config.yaml

Disable incremental manifest save when resume=n.

commit   : 60d70fa66fc3f709914f40dd088bac1b467001f2    
  
author   : Reid Thompson <[email protected]>    
date     : Wed, 25 May 2022 18:18:37 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 25 May 2022 18:18:37 -0400    

Click here for diff

The manifest is saved on a regular basis during a backup so a failed backup can be resumed. For backups that the user has configured/invoked as not resumable, skip the incremental save of the manifest.

M doc/xml/release.xml
M src/command/backup/backup.c
M test/src/module/command/backupTest.c

Use StringId for type in cryptoHash() and cryptoHmacOne().

commit   : 8c2b3a044fa175031366fb4f676b5c1a0a319ec0    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 May 2022 15:27:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 May 2022 15:27:53 -0400    

Click here for diff

This brings hash types in line with cipher types (i.e. a StringId enum) and allows removal of some Strings.  

M src/command/archive/push/file.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/restore/file.c
M src/command/verify/file.c
M src/command/verify/verify.c
M src/common/crypto/common.h
M src/common/crypto/hash.c
M src/common/crypto/hash.h
M src/info/info.c
M src/storage/azure/storage.c
M src/storage/gcs/write.c
M src/storage/s3/storage.c
M test/src/common/harnessInfo.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/performance/storageTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Set backup percent complete to zero before copy start.

commit   : 38ad2838d553febae1b7fe0244d942dfaa9cf548    
  
author   : Reid Thompson <[email protected]>    
date     : Wed, 25 May 2022 14:20:19 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 25 May 2022 14:20:19 -0400    

Click here for diff

Waiting to write percent complete until the first file completed resulted in a period of time where the backup was running without status available to the user.
  

  
Remedy this by initializing percent complete to zero when the backup is ready to start copying files.

M doc/xml/release.xml
M src/command/backup/backup.c

Remove unused errors.

commit   : 7b1935c71005a5a928e34caf7b8b48e5d871cc66    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 May 2022 10:23:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 May 2022 10:23:43 -0400    

Click here for diff

Most of these were probably never ported from Perl to C and others became obsolete over time.  
  
Fix one error that was the wrong type.  
  
Also fix/improve some comments.  

M src/build/error/error.yaml
M src/command/restore/restore.c
M src/common/error.auto.c
M src/common/error.auto.h

Add missing static keywords.

commit   : f6f2f2e2a34c01e20bbf6c26e5e8a232a6bd232d    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 May 2022 09:42:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 May 2022 09:42:49 -0400    

Click here for diff

Add static keyword to local variables where missing.  

M src/common/stat.c
M src/common/user.c
M src/config/config.c

Truncate files during delta restore when they are larger than expected.

commit   : 7ec51e7e6279ea576ad3b4b5c2ea1fc389d08ab5    
  
author   : David Steele <[email protected]>    
date     : Tue, 24 May 2022 16:39:35 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 24 May 2022 16:39:35 -0400    

Click here for diff

Previously the behavior was to download the file from the repository when it was not exactly the same size in PGDATA. However, it may just be that the file was extended and the contents are the same up to the file size recorded in the manifest. This could also be very valuable for files that are always append only, like logs.
  

  
Change info.size to file->size in one place. Both are technically correct but file->size makes more sense.
  

  
Use the new fileName variable in a few existing places.
  

  
Also adjust some existing comments to make them clearer.

M doc/xml/release.xml
M src/command/restore/file.c
M test/src/module/command/restoreTest.c

New CI container builds for PostgreSQL 15 beta1 and minor releases.

commit   : c98baab6b56967e222b825e22e5808e17c02d025    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 May 2022 18:24:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 May 2022 18:24:09 -0400    

Click here for diff

Remove VM_OS_REPO since it is no longer required.  
  
Rebalance PostgreSQL versions for more efficient test times.  
  
Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.  

M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm

Use storagePathP() instead of cfgOptionStr() to get base repo path.

commit   : 69adb990dced3bf1d76a3e2482aad38016165d6c    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 May 2022 12:25:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 May 2022 12:25:58 -0400    

Click here for diff

cfgOptionStr() may not have the correct value if the repo is remote.  
  
Use storagePathP() instead since it can ask the remote for the correct value when required.  

M src/command/repo/common.c

Improve memory usage of mem contexts.

commit   : c7a66ac1afb9a55b049dc18493b602b7bf62bda4    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 May 2022 10:52:01 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 18 May 2022 10:52:01 -0400    

Click here for diff

Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible.
  

  
This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case.
  

  
Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed.
  

  
Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree().
  

  
Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.

M doc/xml/release.xml
M src/build/common/yaml.c
M src/command/backup/pageChecksum.c
M src/command/server/server.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/compress/zst/compress.c
M src/common/compress/zst/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/exec.c
M src/common/ini.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/fdRead.c
M src/common/io/fdWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/group.c
M src/common/io/filter/sink.c
M src/common/io/filter/size.c
M src/common/io/http/client.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/http/request.c
M src/common/io/http/response.c
M src/common/io/http/session.c
M src/common/io/http/url.c
M src/common/io/read.c
M src/common/io/socket/client.c
M src/common/io/socket/server.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/server.c
M src/common/io/tls/session.c
M src/common/io/write.c
M src/common/lock.c
M src/common/memContext.c
M src/common/memContext.h
M src/common/regExp.c
M src/common/stat.c
M src/common/type/buffer.c
M src/common/type/json.c
M src/common/type/keyValue.c
M src/common/type/list.c
M src/common/type/object.h
M src/common/type/pack.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/common/type/xml.c
M src/common/user.c
M src/common/wait.c
M src/config/parse.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M src/postgres/client.c
M src/protocol/client.c
M src/protocol/command.c
M src/protocol/helper.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/server.c
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/helper.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M test/src/common/harnessLog.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/typeObjectTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/protocol/protocolTest.c

Add additional detail to warnings when delta checksum is auto-enabled.

commit   : 83af3f1b7ae78b150849939766e2e555366b95d5    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 May 2022 08:48:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 May 2022 08:48:48 -0400    

Click here for diff

Hopefully this will help with debugging when it is not clear why delta checksum is being enabled.  

M src/info/manifest.c
M test/src/module/command/backupTest.c
M test/src/module/info/manifestTest.c

Fix RHEL container build for documentation.

commit   : 5dfd00bb6ce0d6d2ead0ba6c899c18983f9a7ac9    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 May 2022 08:18:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 May 2022 08:18:34 -0400    

Click here for diff

For some reason /lib/systemd/system/sysinit.target.wants no longer exists in the rockylinux:8 container.  
  
Create this directory explicitly in case it does not exist.  

M doc/xml/user-guide.xml

Fix comment indentation.

commit   : 5360f2ec0af08ac1755d48f234347c240ff09f4a    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 May 2022 10:50:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 May 2022 10:50:07 -0400    

Click here for diff

M src/command/backup/file.c

Udpate parse.auto.c with labels from 4dcc9df2.

commit   : b598f49dedbfb2bdd19d43dae9fa9ba654f2287c    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 May 2022 09:17:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 May 2022 09:17:00 -0400    

Click here for diff

Committed separately so it can be ignored in history/blame.  

M src/config/parse.auto.c

Add labels in parse.auto.c to make diffs easier to read.

commit   : 4dcc9df222d2ecef6721c9ecd52b05e24024cc3d    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 May 2022 09:14:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 May 2022 09:14:46 -0400    

Click here for diff

Because there is a lot of repetition in this file, changes can look very jumbled with existing data in a diff. Also, if can be hard to tell what is being modified if the diff does not show enough lines before and after.  
  
This change adds labels to the end of the line to localize the diff and make it easier to see what has been changed. Also, remove some linefeeds and make separators more consistent.  
  
The change to parse.auto.c will be committed separately so it can be ignored in history/blame.  

M src/build/config/main.c
M src/build/config/render.c
M src/build/config/render.h
M test/src/module/build/configTest.c

Update config.guess to latest version.

commit   : f5023a769d4580258a336cf26f2b293aa2341b29    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 May 2022 08:54:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 May 2022 08:54:19 -0400    

Click here for diff

M src/build/config.guess

Begin v2.40 development.

commit   : 243eef1e520bb72ae2b28a627dda323305e38862    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 May 2022 08:51:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 May 2022 08:51:37 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h