v2.49: Remove PostgreSQL 9.3 Support
commit : 3cb891e3ca49ed1166fb0b513b126412448f5b09
author : David Steele <[email protected]>
date : Mon, 27 Nov 2023 08:55:56 -0300
committer: David Steele <[email protected]>
date : Mon, 27 Nov 2023 08:55:56 -0300 Bug Fixes:
* Fix regression in retries. (Reviewed by Stephen Frost. Reported by Norman Adkins, Tanel Suurhans, Jordan English, Timothée Peignier.)
* Fix recursive path remove in SFTP storage driver. (Fixed by Reid Thompson. Reviewed by Stephen Frost. Reported by Luc.)
Improvements:
* Remove support for PostgreSQL 9.3. (Reviewed by Stephen Frost.)
Documentation Features:
* Document maintainer options. (Reviewed by Stefan Fercot.)
* Update point-in-time recovery documentation for PostgreSQL >= 13.
Test Suite Improvements:
* Allow config/load unit test to run without libssh2 installed. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Wu Ning.) M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release/2023/2.49.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml
Migrate backupFile() tests in command/backup module.
commit : 7d51228bf5227494bf50327488032221bd3d30fa
author : David Steele <[email protected]>
date : Fri, 24 Nov 2023 17:07:49 -0300
committer: David Steele <[email protected]>
date : Fri, 24 Nov 2023 17:07:49 -0300 The backupFile() tests were written before the bulk of the backup command had been migrated to C. Some of them have been migrated to the complete backup tests, but others were left because there was no way to make changes to files during a backup.
Now that we have the backup script harness introduced in 337da35a it is now possible to migrate all the tests. The new tests are better because they not only test backupFile() but all the functions upstream and downstream of it. M test/define.yaml
M test/src/common/harnessBackup.c
M test/src/common/harnessBackup.h
M test/src/module/command/backupTest.c
Add test to show behavior of bundled files truncated during backup.
commit : 337da35ab278e058ec4cdffeb4319f1a1e2cb0df
author : David Steele <[email protected]>
date : Fri, 24 Nov 2023 12:25:40 -0300
committer: David Steele <[email protected]>
date : Fri, 24 Nov 2023 12:25:40 -0300 This behavior violates an assertion but is completely possible with the current implementation. This behavior will be fixed in a future commit, but for now at least test how it works correctly and remove the assertion so the test runs without error.
Also add a new harness that allows changes during the backup to be scripted. M src/info/manifest.c
M test/define.yaml
M test/src/common/harnessBackup.c
M test/src/common/harnessBackup.h
M test/src/module/command/backupTest.c
Remove unused fields from backupJobResult() test in command/backup unit.
commit : ac78b965837440ee34f7b5bf0a039c6e9d78eeec
author : David Steele <[email protected]>
date : Sat, 18 Nov 2023 11:12:41 -0300
committer: David Steele <[email protected]>
date : Sat, 18 Nov 2023 11:12:41 -0300 These fields were not used because of the noop so it was hard to keep them up to date. Rather than attempt to do so, just remove them and add a comment to explain why they are missing. M doc/xml/release/2023/2.49.xml
M doc/xml/release/contributor.xml
M test/src/module/command/backupTest.c
Fix recursive path remove in SFTP storage driver.
commit : c4dc4665f88e9d67a3b66f44cd59d6dd4bb5107f
author : Reid Thompson <[email protected]>
date : Sat, 18 Nov 2023 08:47:58 -0500
committer: GitHub <[email protected]>
date : Sat, 18 Nov 2023 08:47:58 -0500 storageSftpPathRemove() used LIBSSH2_FX_FAILURE to determine when it was attempting to unlink a directory, but it appears that LIBSSH2_FX_PERMISSION_DENIED is also valid for this case.
Update storageSftpPathRemove() to accept either error and adjust tests.M doc/xml/release/2023/2.49.xml
M doc/xml/release/contributor.xml
M src/storage/sftp/storage.c
M test/src/module/storage/sftpTest.c
Allow config/load unit test to run without libssh2 installed.
commit : e2b734eff97e3fac32895b799cc32126013aae27
author : Reid Thompson <[email protected]>
date : Thu, 16 Nov 2023 10:50:09 -0500
committer: GitHub <[email protected]>
date : Thu, 16 Nov 2023 10:50:09 -0500 Add additional #ifdef HAVE_LIBSSH2 wrapping around tests requiring libssh2 in loadTest.c.M doc/xml/release.xml
M doc/xml/release/2023/2.49.xml
M test/src/module/config/loadTest.c
Fix storageReadRemote() to return actual read bytes.
commit : 05207bb8e4f462cd2fd5c685d43ad8911b219077
author : Georgy Shelkovy <[email protected]>
date : Wed, 15 Nov 2023 17:41:40 +0500
committer: GitHub <[email protected]>
date : Wed, 15 Nov 2023 17:41:40 +0500 All storage interface read methods should return actual read bytes. This patch refactors storageReadRemote() to eliminate duplicated code and return actual read bytes. The return value is calculated as the number of bytes written to the passed buffer.
This is technically a bug but does not express as an issue currently because this return value is not being used. It will be used in the future, though, so it needs to be fixed.M src/storage/remote/read.c
M test/src/module/storage/remoteTest.c
Remove old version conditionals from user guide.
commit : ea317df5d9af66935175f7ec87845f30be4f5d36
author : David Steele <[email protected]>
date : Sat, 11 Nov 2023 10:33:09 -0300
committer: David Steele <[email protected]>
date : Sat, 11 Nov 2023 10:33:09 -0300 The user guide does not need to build for EOL versions of PostgreSQL, so remove some conditionals used to support versions older than 12. M doc/xml/user-guide.xml
Update point-in-time recovery documentation for PostgreSQL >= 13.
commit : eb69e2ee6339047f0145b751437ac2bec82bb5ab
author : David Steele <[email protected]>
date : Fri, 10 Nov 2023 17:00:57 -0300
committer: David Steele <[email protected]>
date : Fri, 10 Nov 2023 17:00:57 -0300 PITR changed in PostgreSQL 13 to error when the recovery target is not reached. Update the documentation to work with PostgreSQL >= 13 as well as < 13.
Also update the versions built for RHEL and Debian since PostgreSQL 11 is now EOL. M doc/xml/release/2023/2.49.xml
M doc/xml/user-guide.xml
Remove support for PostgreSQL 9.3.
commit : dcf078198761a2cfe9be6b4c7f4b4fb07ef53cbf
author : David Steele <[email protected]>
date : Thu, 9 Nov 2023 12:59:12 -0300
committer: GitHub <[email protected]>
date : Thu, 9 Nov 2023 12:59:12 -0300 Per our policy to support five EOL versions of PostgreSQL, 9.3 is no longer supported by pgBackRest.
Remove all logic associated with 9.3 and update the tests.M README.md
M doc/xml/index.xml
M doc/xml/release/2023/2.49.xml
M src/build/help/help.xml
M src/build/postgres/postgres.yaml
M src/info/manifest.c
M src/postgres/interface/version.intern.h
M src/postgres/interface/version.vendor.h
M src/postgres/version.auto.h
M test/code-count/file-type.yaml
M test/container.yaml
M test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessPostgres.c
M test/src/common/harnessPostgres.h
D test/src/common/harnessPostgres/harness093.c
M test/src/common/harnessPostgres/harnessVersion.intern.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/interfaceTest.c
M test/test.pl
Fix regression in retries.
commit : fa5b2d44ad9f7591517eecaaaaeb07e9500dd51c
author : David Steele <[email protected]>
date : Thu, 9 Nov 2023 12:04:25 -0300
committer: GitHub <[email protected]>
date : Thu, 9 Nov 2023 12:04:25 -0300 5314dbf aimed to make nested Wait objects more accurate with regard to wait time but it also got rid of the "bonus" retry that was implicit in the prior implementation. This meant that if an operation used up the entire allotted timeout, it would not be retried. Object stores especially are noisy places and some amount of retry should always be attempted. So even though removing the "bonus" retry was intended, it turned out not to be a good idea.
Instead of an implicit retry, formalize two retries in the Wait object even if the wait time has expired. Any number of retries are allowed during the wait period. Also remove waitRemaining() since it is no longer needed.
Adjust tests as needed to account for the extra timeouts.
Note that there may still be an underlying issue here that is simply being masked by retries. That is, the issue expressing was that waiting for a socket to be writable was timing out and without a retry that caused a hard error. This patch does nothing to address the source of the write timeout and perhaps there is nothing we can do about it. It does seem similar to the write issue we had with our blocking TLS implementation, but it was never clear if that was a problem with TLS, the kernel, or a bug in pgBackRest itself. It cropped up after a kernel update and we switched to non-blocking TLS to address the issue (c88684e).M doc/xml/release/2023/2.49.xml
M doc/xml/release/contributor.xml
M src/common/io/socket/client.c
M src/common/wait.c
M src/common/wait.h
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/waitTest.c
M test/src/module/db/dbTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/storage/sftpTest.c
Remove unused header.
commit : 3c116e1829a0df62aaf50ce241c63d829eb438e0
author : David Steele <[email protected]>
date : Sat, 4 Nov 2023 14:43:11 -0300
committer: David Steele <[email protected]>
date : Sat, 4 Nov 2023 14:43:11 -0300 M src/common/io/fd.c
Refactor of pq shim to allow more flexible scripting.
commit : e0f5880b09624db2c9274a8a615a5309a0bddab1
author : David Steele <[email protected]>
date : Sun, 22 Oct 2023 13:55:56 -0400
committer: David Steele <[email protected]>
date : Sun, 22 Oct 2023 13:55:56 -0400 The pq scripts were pretty static which had already led to a lot of code duplication in the backup test harness.
Instead allow the scripts to be built dynamically, which allows for much more flexibility and reduces duplication. For now just make these changes in the backup harness, but they may be useful elsewhere.
While we are making big changes, also update the macro/function names to hew closer to our current harness naming conventions. M CONTRIBUTING.md
M doc/xml/contributing.xml
M test/src/common/harnessBackup.c
M test/src/common/harnessBackup.h
M test/src/common/harnessPq.c
M test/src/common/harnessPq.h
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/db/dbTest.c
M test/src/module/postgres/clientTest.c
Remove unused parameter in backupProcess().
commit : 306fdff93a072c72d6a40aab728ceeb44cc7b23f
author : David Steele <[email protected]>
date : Thu, 19 Oct 2023 10:11:27 -0400
committer: David Steele <[email protected]>
date : Thu, 19 Oct 2023 10:11:27 -0400 Also fix a comment in the same function. M src/command/backup/backup.c
Tidy and align site introduction and description.
commit : 04d92cca7eb42e0037e65701109da284b214c84f
author : David Steele <[email protected]>
date : Thu, 19 Oct 2023 09:54:34 -0400
committer: David Steele <[email protected]>
date : Thu, 19 Oct 2023 09:54:34 -0400 Make the description more concise and use it for both the site description and introduction. M doc/xml/index.xml
More efficient/compact calculations for encoded sizes.
commit : 459d59615a18c147140b1ae94a804be66541c0ab
author : David Steele <[email protected]>
date : Wed, 18 Oct 2023 18:14:32 -0400
committer: David Steele <[email protected]>
date : Wed, 18 Oct 2023 18:14:32 -0400 encodeToStrSizeBase64() is definitely more efficient (pulled from the PostgreSQL implementation).
encodeToStrSizeBase64Url() is probably about as efficient as the prior implementation but is certainly more compact.
Also add tests for zero byte encoding sizes. M src/common/encode.c
M test/src/module/common/encodeTest.c
Document maintainer options.
commit : 21c8c8a66cb06eba714bfa41bed7239af3d14f08
author : David Steele <[email protected]>
date : Sat, 14 Oct 2023 16:22:09 -0400
committer: GitHub <[email protected]>
date : Sat, 14 Oct 2023 16:22:09 -0400 Document maintainer options in a separate section with appropriate explanation and caveats.
Also make the pg-version-force option user visible now that maintainer caveats have been documented.M doc/src/command/build/reference.c
M doc/xml/release/2023/2.49.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/help/help.c
M test/src/module/command/helpTest.c
M test/src/module/doc/buildTest.c
Simplify description rendering in command and configuration reference.
commit : 81536cd486d716ebd10eb0637ef6b1b9a61d9dd0
author : David Steele <[email protected]>
date : Tue, 10 Oct 2023 16:49:05 -0400
committer: David Steele <[email protected]>
date : Tue, 10 Oct 2023 16:49:05 -0400 The reference documentation was still using a very old version of rendering from before the user guide was introduced. This was preserved in the initial C migration to reduce the diff between Perl and C for testing purposes. The old version used hard linefeeds to simulate paragraphs and reduce the amount of markup that needed to be used. In retrospect this was not a great idea.
Instead use more natural rendering that does not depend on using hard linefeeds between paragraphs. M doc/src/command/build/reference.c
M doc/xml/release/2023/2.49.xml
M test/src/module/doc/buildTest.c
Simplify section titles in configuration reference.
commit : 45abea471ecec10a351bfd9e5fbed8737c32da42
author : David Steele <[email protected]>
date : Tue, 10 Oct 2023 13:02:50 -0400
committer: David Steele <[email protected]>
date : Tue, 10 Oct 2023 13:02:50 -0400 For some reason the internal section id was included in the title. This was probably copied from another section title where it made more sense, e.g. including the option name after the title.
Also add release note missed in 1eb01622. M doc/src/command/build/reference.c
M doc/xml/release/2023/2.49.xml
M test/src/module/doc/buildTest.c
Build command and configuration reference in C.
commit : 1eb0162208b37f577d35ba775f007202b285900a
author : David Steele <[email protected]>
date : Mon, 9 Oct 2023 14:03:43 -0400
committer: David Steele <[email protected]>
date : Mon, 9 Oct 2023 14:03:43 -0400 Migrate generation of these files from help.xml to the intermediate documentation format. This allows us to share a lot of code that is already in C and remove duplicated code in Perl. More duplicate code can be removed in Perl once man generation is migrated.
Also update the unit test harness to allow testing of modules in the doc directory. M doc/doc.pl
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocRender.pm
M doc/manifest.xml
A doc/src/build/config/config.yaml
A doc/src/build/help/help.xml
A doc/src/command/build/build.c
A doc/src/command/build/build.h
A doc/src/command/build/reference.c
A doc/src/command/build/reference.h
A doc/src/command/help/meson.build
A doc/src/config/load.c
A doc/src/config/load.h
A doc/src/config/meson.build
A doc/src/main.c
A doc/src/meson.build
M doc/xml/dtd/doc.dtd
M meson.build
M src/build/help/help.xml
M test/ci.pl
M test/define.yaml
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/src/command/test/build.c
A test/src/module/doc/buildTest.c
M test/src/module/test/testTest.c
Adjust timeouts in the common/io/tls test to fix flapping coverage.
commit : 983cc1a9e3f2865bf5128b5065722fd995382ba1
author : David Steele <[email protected]>
date : Sun, 1 Oct 2023 16:32:27 -0400
committer: David Steele <[email protected]>
date : Sun, 1 Oct 2023 16:32:27 -0400 This test was failing coverage pretty regularly because the retry in tlsClientOpen() was not always being reached. Make the TLS timeouts longer to ensure reliable coverage. M test/src/module/common/ioTlsTest.c
Fix common/lock test failing with -Werror=unused-result.
commit : af4621894a20ca0333726d459e2e6409bd4ee1e0
author : Reid Thompson <[email protected]>
date : Sat, 30 Sep 2023 12:47:15 -0400
committer: GitHub <[email protected]>
date : Sat, 30 Sep 2023 12:47:15 -0400 Wrap ftruncate() in TEST_RESULT_INT() to prevent this error.M test/src/module/common/lockTest.c
Update Minio test/documentation container version.
commit : eccd9eed196ff9adc074061f26fec523eae2ba00
author : David Steele <[email protected]>
date : Sat, 30 Sep 2023 12:45:24 -0400
committer: David Steele <[email protected]>
date : Sat, 30 Sep 2023 12:45:24 -0400 M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm
Use CSS to number sections in documentation.
commit : 33ba4db9cbd334c9a10f15880e13e3fb589c3b10
author : David Steele <[email protected]>
date : Sat, 30 Sep 2023 09:40:44 -0400
committer: David Steele <[email protected]>
date : Sat, 30 Sep 2023 09:40:44 -0400 This reduces churn in the HTML when sections are added or removed from the documentation. M doc/lib/pgBackRestDoc/Html/DocHtmlPage.pm
M doc/resource/html/default.css
Parse defaults and text sections in help.xml.
commit : 1d5563288cf1839acb237a188b865ecb01fe97da
author : David Steele <[email protected]>
date : Fri, 29 Sep 2023 17:28:00 -0400
committer: David Steele <[email protected]>
date : Fri, 29 Sep 2023 17:28:00 -0400 These will be required to build documentation in C. M doc/xml/dtd/doc.dtd
M src/build/config/parse.h
M src/build/help/help.xml
M src/build/help/parse.c
M src/build/help/parse.h
M src/build/main.c
M test/src/module/build/helpTest.c
M test/src/module/command/helpTest.c
Allow documentation source file to be specified in manifest.
commit : 088026e6ffcd1b084099e11f8461fdd3895b2507
author : David Steele <[email protected]>
date : Fri, 29 Sep 2023 17:06:37 -0400
committer: David Steele <[email protected]>
date : Fri, 29 Sep 2023 17:06:37 -0400 The help source file had previously been hardcoded and now that is no longer needed.
A future commit will introduce more sources outside of the xml path. M doc/lib/pgBackRestDoc/Common/DocManifest.pm
M doc/manifest.xml
M doc/xml/dtd/manifest.dtd
Remove unused references to DocConfig and DocConfigData Perl modules.
commit : 55fda01733f2a7e71ee175345e26344a99989927
author : David Steele <[email protected]>
date : Fri, 29 Sep 2023 16:57:01 -0400
committer: David Steele <[email protected]>
date : Fri, 29 Sep 2023 16:57:01 -0400 M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
M doc/lib/pgBackRestDoc/Html/DocHtmlSite.pm
M doc/lib/pgBackRestDoc/Latex/DocLatex.pm
M doc/lib/pgBackRestDoc/Markdown/DocMarkdown.pm
M doc/lib/pgBackRestDoc/Markdown/DocMarkdownRender.pm
Add new XML functions required for building documentation.
commit : 217584a2c4134667e82b2de8032b5334214e08f1
author : David Steele <[email protected]>
date : Fri, 29 Sep 2023 16:52:08 -0400
committer: David Steele <[email protected]>
date : Fri, 29 Sep 2023 16:52:08 -0400 M src/build/common/xml.c
M src/build/common/xml.h
M test/src/module/common/typeXmlTest.c
Update config.guess and config.sub to latest versions.
commit : 8f319b6fd34f3430b5b527825bd502113f8a957d
author : David Steele <[email protected]>
date : Mon, 25 Sep 2023 09:43:30 -0400
committer: David Steele <[email protected]>
date : Mon, 25 Sep 2023 09:43:30 -0400 M src/build/config.guess
M src/build/config.sub
Begin v2.49 development.
commit : 084c8e1691f59aa412b859f9004e188383826c58
author : David Steele <[email protected]>
date : Mon, 25 Sep 2023 09:40:45 -0400
committer: David Steele <[email protected]>
date : Mon, 25 Sep 2023 09:40:45 -0400 M doc/resource/git-history.cache
M doc/xml/release.xml
A doc/xml/release/2023/2.49.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h