pgBackRest 2.46 commit log

by Yogesh Sharma
pgBackRest
CommitLog

v2.45: Block Incremental Backup (BETA)

commit   : 6ad79d16ca638b7ccdc81b5530a39007b8e01aed    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Mar 2023 09:37:23 +0800    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Mar 2023 09:37:23 +0800

Click here for diff

Bug Fixes:  
  
* Skip writing recovery.signal by default for restores of offline backups. (Reviewed by Stefan Fercot. Reported by Marcel Borger.)  
  
Features:  
  
* Block incremental backup (BETA). (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)  
  
Improvements:  
  
* Keep only one all-default group index. (Reviewed by Stefan Fercot.)  
  
Documentation Improvements:  
  
* Add explicit instructions for upgrading between 2.x versions. (Contributed by Christophe Courtois. Reviewed by David Steele.)  
* Remove references to SSH made obsolete when TLS was introduced.

M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Add beta feature and documentation for block incremental backup.

commit   : 04d224b88e73194ff1c6497dbecaaff392008dfe    
  
author   : David Steele <[email protected]>    
date     : Sat, 18 Mar 2023 20:47:03 +0800    
  
committer: David Steele <[email protected]>    
date     : Sat, 18 Mar 2023 20:47:03 +0800

Click here for diff

M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/config/parse.auto.c.inc

Updates required for new manifest command to work remotely.

commit   : 78c036efb7ebf7060949f3d2f2e6d93986734414    
  
author   : David Steele <[email protected]>    
date     : Thu, 16 Mar 2023 17:46:32 +0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 16 Mar 2023 17:46:32 +0700

Click here for diff

Additional options are required and the BlockChecksum must work remotely.

M src/build/config/config.yaml
M src/command/remote/remote.c
M src/command/restore/blockChecksum.c
M src/command/restore/blockChecksum.h
M src/config/parse.auto.c.inc
M test/src/module/command/restoreTest.c

Prototype manifest command.

commit   : 505a639f1e627b506c2e5a3351e8d4aacae9ed0b    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Mar 2023 21:56:05 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Mar 2023 21:56:05 +0700

Click here for diff

Output a manifest in text or JSON format. Neither format is complete but they cover the basics.  
  
In particular the manifest command outputs the complete block restore when the filter option is specified and the block delta when the pg option is also specified. This is non-destructive so it is safe to execute against a running cluster.

M src/Makefile.in
M src/build/config/config.yaml
M src/build/help/help.xml
A src/command/manifest/manifest.c
A src/command/manifest/manifest.h
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/main.c
M src/meson.build
M test/define.yaml
M test/src/common/harnessBackup.c
A test/src/module/command/manifestTest.c

Block incremental map fixes and improvements.

commit   : c30d3e439b77d544477e289606d30d0c8ffe301f    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Mar 2023 17:48:25 +0700    
  
committer: GitHub <[email protected]>    
date     : Tue, 14 Mar 2023 17:48:25 +0700

Click here for diff

Bug Fixes:
  

  
* Remove the distinction between maps where super block size is equal to block size and maps where they are not. In practice, maps with equal blocks are now rare and most of the optimizations can be applied directly to super blocks where the blocks are equal. This fixes a bug where a map that was created with equal size blocks and then converted to differing block sizes would generate an invalid map.
  

  
* Free reads during restore to avoid running out of file handles.
  

  
Improvements:
  

  
* Store super block sizes in the block map. This allows the final block size to be removed from the block list and provides a more optimal restore and better potential for analysis.
  

  
* Always round the super block size up to the next block size. This makes the number of blocks per super block more predictable.
  

  
* Allow super block sizes to be changed at will in the map. The first case for this is to store the reduced super block size required when the last super block is short but it could be used to dynamically change the super block size to optimize compression.
  

  
* Store a block count rather than a list of blocks in a super block. Blocks must always be sequential, though there may be an offset to the first block in a super block. This saves 11-14% on space for checksum sizes 6-7.
  

  
* In the case that all the blocks for a super block are present, and there is no offset, the block size is omitted.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/backup/blockIncr.c
M src/command/backup/blockIncr.h
M src/command/backup/blockMap.c
M src/command/backup/blockMap.h
M src/command/restore/blockDelta.c
M src/command/restore/file.c
M test/src/module/command/backupTest.c

Fix typo in blockIncrProcess().

commit   : 5c1f78d4dd62c37218285d13b09a9450eb3a588e    
  
author   : David Steele <[email protected]>    
date     : Sun, 12 Mar 2023 22:38:38 +0700    
  
committer: David Steele <[email protected]>    
date     : Sun, 12 Mar 2023 22:38:38 +0700

Click here for diff

M doc/xml/release.xml
M src/command/backup/blockIncr.c

Ensure no continuations when block size equals super block size.

commit   : 1281a6eaf8552232c8c9af4d7fc15c082a7794d9    
  
author   : David Steele <[email protected]>    
date     : Sun, 12 Mar 2023 16:21:43 +0700    
  
committer: David Steele <[email protected]>    
date     : Sun, 12 Mar 2023 16:21:43 +0700

Click here for diff

In this case each super block contains a single block so continuations are not possible.

M doc/xml/release.xml
M src/command/backup/blockMap.c

Move backup pq test script generation to backup harness.

commit   : d7704a8bc837b0be5564737e491bf5f73796ca0a    
  
author   : David Steele <[email protected]>    
date     : Sat, 11 Mar 2023 14:04:43 +0700    
  
committer: David Steele <[email protected]>    
date     : Sat, 11 Mar 2023 14:04:43 +0700

Click here for diff

This allows scripted online backups to be used by other test modules.

M test/src/common/harnessBackup.c
M test/src/common/harnessBackup.h
M test/src/module/command/backupTest.c

Fix error handling in build-code binary.

commit   : 2fffd64213b7d51b6444f61502f4225584667731    
  
author   : David Steele <[email protected]>    
date     : Fri, 10 Mar 2023 16:11:30 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 10 Mar 2023 16:11:30 +0700

Click here for diff

120a49b6 improved the error handling but due to a copy/pasto errors were being handled as asserts, which tripped an assertion in the log module.

M src/build/main.c

Add missing header.

commit   : 19f3a1d3040df56e73f19674618eb5f447dabe8d    
  
author   : David Steele <[email protected]>    
date     : Fri, 10 Mar 2023 16:08:30 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 10 Mar 2023 16:08:30 +0700

Click here for diff

M test/src/common/harnessServer.h

Add beta feature infrastructure.

commit   : 24f725212d0c1dedc0f34fe4e939e481585ff697    
  
author   : David Steele <[email protected]>    
date     : Fri, 10 Mar 2023 15:30:27 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 10 Mar 2023 15:30:27 +0700

Click here for diff

This allows options to be marked as beta, which will require that the --beta option be supplied to prevent accidental usage of a beta feature.  
  
The online and command-line documentation also show warnings when options are beta.

M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Custom/DocConfigData.pm
M src/build/config/config.yaml
M src/build/config/parse.c
M src/build/config/parse.h
M src/build/config/render.c
M src/build/help/help.xml
M src/command/help/help.c
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/config/parse.c
M src/config/parse.h
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/build/config/config.yaml
M test/src/build/help/help.xml
M test/src/common/harnessConfig.c
M test/src/module/build/configTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/restoreTest.c
M test/src/module/config/parseTest.c

Update default block size and super block values based on testing.

commit   : 6b409d049eb935146f7996b75b97744b75c411a9    
  
author   : David Steele <[email protected]>    
date     : Fri, 10 Mar 2023 14:01:38 +0700    
  
committer: David Steele <[email protected]>    
date     : Fri, 10 Mar 2023 14:01:38 +0700

Click here for diff

Block sizes are incremented when the size of the map becomes as large as a single block. This is arbitrary but it appears to give a good balance of block size vs map size.  
  
The full backup super block size is set to minimize loss of compression efficiency since most blocks in the database will likely never be modified. For diff/incr backup super blocks, a smaller size is allowable since only modified blocks are stored. The overall savings of not storing unmodified blocks offsets the small loss in compression efficiency due to the smaller super block and allows more granular fetches during restore.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/command/backup/backup.c
M src/config/parse.auto.c.inc
M test/src/module/command/backupTest.c

Rename BlockHash to BlockChecksum.

commit   : 1119a5353911d6d326edac325f530bb796a806ae    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Mar 2023 11:04:03 +0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Mar 2023 11:04:03 +0700

Click here for diff

Checksum is the generally used terminology in the code base, even when a hash is being used as a checksum.

M doc/xml/release.xml
M src/Makefile.in
R066 src/command/restore/blockHash.c src/command/restore/blockChecksum.c
R080 src/command/restore/blockHash.h src/command/restore/blockChecksum.h
M src/command/restore/blockDelta.c
M src/command/restore/blockDelta.h
M src/command/restore/file.c
M src/command/restore/file.h
M src/meson.build
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/module/command/restoreTest.c

Exclude backup set size from info for block incremental backups.

commit   : 6252c0e4485caee362edec13302a5f735a69bff4    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Mar 2023 10:30:57 +0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Mar 2023 10:30:57 +0700

Click here for diff

As calculated this size is not correct since it does not include the parts of prior block incrementals that are required to make the current block incremental valid. At best this could be approximated and the resulting values might be very confusing.  
  
For now, at least, exclude this metric for block incremental backups.

M doc/xml/release.xml
M src/command/info/info.c
M test/src/module/command/infoTest.c

Use xxHash instead of SHA-1 for block incremental checksums.

commit   : 210bed4511ae5523b170eb193b877346d787140d    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Mar 2023 10:02:04 +0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Mar 2023 10:02:04 +0700

Click here for diff

xxHash is significantly faster than SHA-1 so this helps reduce the overhead of the feature.  
  
A variable number of bytes are used from the xxHash depending on the block size with a minimum of six bytes for the smallest block size. This keeps the maps smaller while still providing enough bits to detect block changes.

M doc/xml/release.xml
M src/Makefile.in
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/backup/backup.c
M src/command/backup/blockIncr.c
M src/command/backup/blockIncr.h
M src/command/backup/blockMap.c
M src/command/backup/blockMap.h
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/protocol.c
M src/command/restore/blockDelta.c
M src/command/restore/blockDelta.h
M src/command/restore/blockHash.c
M src/command/restore/blockHash.h
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/command/restore/restore.c
A src/common/crypto/xxhash.c
A src/common/crypto/xxhash.h
A src/common/crypto/xxhash.vendor.c.inc
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/info/manifest.c
M src/info/manifest.h
M src/meson.build
M test/define.yaml
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/common/harnessManifest.c
M test/src/common/harnessManifest.h
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/info/manifestTest.c

Block-level incremental backup super blocks.

commit   : 8b5153ad210011812c8fe6db2f2ed829a5aa543a    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Mar 2023 09:39:54 +0700    
  
committer: GitHub <[email protected]>    
date     : Thu, 9 Mar 2023 09:39:54 +0700

Click here for diff

Small blocks sizes can lead to reduced compression efficiency, so allow multiple blocks to be compressed together in a super block. The disadvantage is that the super block must be read sequentially to retrieve blocks. However, different super block sizes can be used for different backup types, so the full backup super block sizes are large for compression efficiency and diff/incr are smaller for retrieval efficiency.

M doc/xml/release.xml
M src/Makefile.in
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/backup/backup.c
M src/command/backup/blockIncr.c
M src/command/backup/blockIncr.h
M src/command/backup/blockMap.c
M src/command/backup/blockMap.h
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/protocol.c
A src/command/restore/blockDelta.c
A src/command/restore/blockDelta.h
M src/command/restore/file.c
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/info/manifest.c
M src/meson.build
M test/define.yaml
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c

Add pg-version-force option for fork integration.

commit   : 740c2258e36613b1b4e76bab452d0e610a399e7f    
  
author   : Stefan Fercot <[email protected]>    
date     : Thu, 9 Mar 2023 02:23:15 +0100    
  
committer: GitHub <[email protected]>    
date     : Thu, 9 Mar 2023 02:23:15 +0100

Click here for diff

Forks may update pg_control version or WAL magic without affecting the structures that pgBackRest depends on.
  

  
This option forces pgBackRest to treat a cluster as the specified version when it cannot be automatically identified.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/archive/get/get.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/stanza/common.c
M src/command/verify/verify.c
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/db/db.c
M src/postgres/interface.c
M src/postgres/interface.h
M src/postgres/interface/version.intern.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/postgres/interfaceTest.c

Skip writing recovery.signal by default for restores of offline backups.

commit   : 2fa7e53c5d4b5a5fd79f62d39763746727a5b9be    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Mar 2023 19:05:23 +0700    
  
committer: GitHub <[email protected]>    
date     : Wed, 8 Mar 2023 19:05:23 +0700

Click here for diff

When restoring an offline backup on PostgreSQL >= 12, skip writing recovery.signal by default since this will error if the backup was made with wal_level=minimal. If the user explicitly sets the type option to something other than none, then write recovery.signal as usual since it is possible to do Point-In-Time-Recovery from an offline backup as long as wal_level was not minimal.

M doc/xml/release.xml
M src/build/help/help.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Use raw compression/encryption to bundling and block incremental backup.

commit   : 7e5adc03594a3b7a975001e80902a51df3ddd577    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Mar 2023 18:46:24 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Mar 2023 18:46:24 +0700

Click here for diff

Raw encryption was already being used for block incremental. This commit adds raw compression to block incremental where possible (see da918587).  
  
Raw compression/encryption is also added to bundling for a backup set when block incremental is enabled on the full backup. This prevents a break in backward compatibility since block incremental is not backward compatible.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/protocol.c
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/command/restore/restore.c
M src/info/manifest.c
M src/info/manifest.h
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/info/manifestTest.c

Add optional raw format for compression types.

commit   : da9185870222057b97365f5db0acf797f959ec5e    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Mar 2023 18:31:17 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Mar 2023 18:31:17 +0700

Click here for diff

Raw format saves 12 bytes of header for gzip and 4 bytes of checksum for lz4 (plus CPU overhead). This may not seem like much, but over millions of small files or incremental blocks can really add up. Even though it may be a relatively small percentage of the overall backup size it is still objectively a large amount of data.  
  
Use raw format for protocol compression to exercise the feature.  
  
Raw compression format will be added to bundling and block incremental in a followup commit.

M doc/xml/release.xml
M src/build/help/render.c
M src/command/archive/get/file.c
M src/command/archive/push/file.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/help/help.c
M src/command/restore/file.c
M src/command/verify/file.c
M src/command/verify/verify.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/compress.h
M src/common/compress/bz2/decompress.c
M src/common/compress/bz2/decompress.h
M src/common/compress/gz/compress.c
M src/common/compress/gz/compress.h
M src/common/compress/gz/decompress.c
M src/common/compress/gz/decompress.h
M src/common/compress/helper.c
M src/common/compress/helper.h
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/compress.h
M src/common/compress/lz4/decompress.c
M src/common/compress/lz4/decompress.h
M src/common/compress/zst/compress.c
M src/common/compress/zst/compress.h
M src/common/compress/zst/decompress.c
M src/common/compress/zst/decompress.h
M src/storage/remote/read.c
M src/storage/remote/write.c
M test/src/common/harnessStorage.c
M test/src/module/command/backupTest.c
M test/src/module/common/compressTest.c
M test/src/module/performance/storageTest.c
M test/src/module/storage/remoteTest.c

Improve interface handling in storage module.

commit   : f6e307365f5a55a8bb4eb694e0a11fb4f32fb6e7    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Mar 2023 11:27:41 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Mar 2023 11:27:41 +0700

Click here for diff

Make the interface object the parent of the driver object rather than the interface being allocated directly in the driver object.  
  
The prior method was more efficient when mem contexts had a much higher cost. Now mem contexts are cheap so it makes more sense to structure the objects in a way that works better with mem context auditing. This also means the mem context does not need to be stored separately since it can be extracted directly from the interface object.  
  
There are other areas that need to get the same improvement before the specialized objMoveContext() and objFreeContext() functions can be removed.

M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/read.h
M src/storage/remote/storage.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/storage.c
M src/storage/write.c
M src/storage/write.h
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c

Improved error handling in build-code binary.

commit   : 120a49b6590a549a72bfe0a6a5a22905e835b2ce    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Mar 2023 21:07:08 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 6 Mar 2023 21:07:08 +0700

Click here for diff

Show a full stack trace instead of just the error.

M src/build/main.c

Set online flag in manifest in command/restore unit tests.

commit   : 0818601c055724bde3b97d4b02d3a38b0a53899a    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Mar 2023 09:49:33 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 6 Mar 2023 09:49:33 +0700

Click here for diff

This flag does not currently affect restore behavior but it will in an upcoming commit. Set the flag here to simplify the test diff in the upcoming commit.

M test/src/module/command/restoreTest.c

Allow control version and WAL magic to be overridden in test harness.

commit   : c656669ac2354dab504d94b604e305c3b68bb493    
  
author   : David Steele <[email protected]>    
date     : Sat, 4 Mar 2023 12:50:02 +0700    
  
committer: David Steele <[email protected]>    
date     : Sat, 4 Mar 2023 12:50:02 +0700

Click here for diff

This makes it easier to write tests for invalid control version/WAL magic.  
  
Also add HRN_PG_WAL_TO_BUFFER() to simplify generation of WAL headers.

M test/src/common/harnessPostgres.c
M test/src/common/harnessPostgres.h
M test/src/common/harnessPostgres/harnessVersion.intern.h
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/verifyTest.c
M test/src/module/postgres/interfaceTest.c

Keep only one all-default group index.

commit   : 1648c133d6bc1ffd0682ae6219934d00295aa580    
  
author   : David Steele <[email protected]>    
date     : Sat, 4 Mar 2023 12:45:08 +0700    
  
committer: GitHub <[email protected]>    
date     : Sat, 4 Mar 2023 12:45:08 +0700

Click here for diff

It is possible for a group index to be created for an option that is later found to not meet dependencies. In this case all values would be default leading to a phantom group, which can be quite confusing.
  

  
Remove group indexes that are all default (except the final one) and make sure the key for the final all default group index is 1.

M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c

Fix typo and remove extraneous linefeed.

commit   : 439447977681ea8730935535e25a657334596942    
  
author   : Stefan Fercot <[email protected]>    
date     : Tue, 28 Feb 2023 08:47:51 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Feb 2023 08:47:51 +0700

Click here for diff

M src/build/config/config.yaml
M src/postgres/interface.c

Rename block incremental manifest keys.

commit   : 16ac5ee8d3b926464cd27e8f064579f1e7935f4c    
  
author   : David Steele <[email protected]>    
date     : Sun, 26 Feb 2023 16:13:44 +0700    
  
committer: David Steele <[email protected]>    
date     : Sun, 26 Feb 2023 16:13:44 +0700

Click here for diff

Since the keys need to be read/written in order, these keys make the logic a bit simpler.

M doc/xml/release.xml
M src/info/manifest.c
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/module/command/backupTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/manifestTest.c

Add repo-block-age-map and repo-block-size-map options.

commit   : a9867cb0b8e8d7abcac9b18b76e3e490c265f3cb    
  
author   : David Steele <[email protected]>    
date     : Sun, 26 Feb 2023 14:49:34 +0700    
  
committer: David Steele <[email protected]>    
date     : Sun, 26 Feb 2023 14:49:34 +0700

Click here for diff

Make these options configurable. This is primarily for testing purposes so the new options will be kept internal.

M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/help/help.xml
M src/command/backup/backup.c
M src/common/type/list.c
M src/common/type/list.h
M src/config/config.auto.h
M src/config/parse.auto.c.inc
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/typeListTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/typeTest.c

Add explicit instructions for upgrading between 2.x versions.

commit   : 15d5dcdd3bb387aab28bbe7262011c38c7d6d22c    
  
author   : Christophe Courtois <[email protected]>    
date     : Sun, 26 Feb 2023 08:41:32 +0100    
  
committer: GitHub <[email protected]>    
date     : Sun, 26 Feb 2023 08:41:32 +0100

Click here for diff

Add an explicit statement that there is nothing special to do when upgrading between 2.x versions.
  

  
Leave the previous paragraph about the default location that changed between 2.00 and 2.02, as it is more a matter of transitioning from 1.x to 2.x.

M doc/xml/release.xml
M doc/xml/user-guide.xml

Rename DeltaMap to BlockHash.

commit   : dffc933384e4327e2fd52868dd19668ed9d3ed91    
  
author   : David Steele <[email protected]>    
date     : Mon, 13 Feb 2023 09:17:30 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 13 Feb 2023 09:17:30 +0700

Click here for diff

This more accurately describes what the object does.

M doc/xml/release.xml
M src/Makefile.in
R083 src/command/restore/deltaMap.c src/command/restore/blockHash.c
R084 src/command/restore/deltaMap.h src/command/restore/blockHash.h
M src/command/restore/file.c
M src/command/restore/file.h
M src/meson.build
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/module/command/restoreTest.c

Consistently declare block incremental size as size_t.

commit   : 779efe0d7a54340141e01597279e9ab1d0bd0773    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Feb 2023 13:01:56 +0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Feb 2023 13:01:56 +0700

Click here for diff

The block is often allocated in memory so size_t makes more sense than uint64_t.

M doc/xml/release.xml
M src/command/backup/blockIncr.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/protocol.c
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/info/manifest.c
M src/info/manifest.h
M test/src/common/harnessManifest.h

Remove parameter list from deltaMapNew().

commit   : d520816acf9ecb01ef805cecbc9de7e8d0a9f468    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Feb 2023 08:11:05 +0700    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Feb 2023 08:11:05 +0700

Click here for diff

Since this filter cannot be used remotely (yet) there is no reason to create a parameter list.

M doc/xml/release.xml
M src/command/restore/deltaMap.c

Improve IoChunkedRead end-of-file handling.

commit   : 3feed389a2199454db68e446851323498b45db20    
  
author   : David Steele <[email protected]>    
date     : Wed, 8 Feb 2023 22:34:23 +0700    
  
committer: David Steele <[email protected]>    
date     : Wed, 8 Feb 2023 22:34:23 +0700

Click here for diff

Determine end-of-file earlier to improve throughput.  
  
Also clean up some comments and formatting.

M doc/xml/release.xml
M src/common/io/chunkedRead.c

Add block incremental to real/all test output.

commit   : 089fae035bf0a9778fff0527ea3cbb113625a403    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Feb 2023 14:09:50 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Feb 2023 14:09:50 +0700

Click here for diff

M doc/xml/release.xml
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Check for stray execute permissions in test.pl --code-format.

commit   : 31cad5e09e1910fc9d36d18306524aabca3ff36b    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Feb 2023 12:57:04 +0700    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Feb 2023 12:57:04 +0700

Click here for diff

Sometimes these can get set while updating permissions from with a VM or container.  
  
Also fix a few permissions that were not correct.

M src/config/parse.auto.c.inc
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/test.pl
M test/uncrustify.cfg

Fix contributors in release notes.

commit   : 8e7e9d36a133f2ec01f4864b5b0c80344888b737    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 Jan 2023 21:28:28 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 Jan 2023 21:28:28 +0700

Click here for diff

M doc/xml/release.xml

Remove references to SSH made obsolete when TLS was introduced.

commit   : c5907a2e7193ca8f79cf56dca23e0df118e686c3    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 Jan 2023 08:28:32 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 Jan 2023 08:28:32 +0700

Click here for diff

Also remove details about SSH compression that are not helpful.

M doc/xml/release.xml
M src/build/help/help.xml
M test/src/module/command/helpTest.c

Update comments missed in copy/paste.

commit   : 9ae6b6ef0f344b49f6153bf842ac815b0fcb9ea3    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 Jan 2023 08:04:09 +0700    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 Jan 2023 08:04:09 +0700

Click here for diff

M src/command/backup/blockIncr.c
M src/command/backup/pageChecksum.c
M src/command/restore/deltaMap.c
M src/common/io/filter/chunk.c

Use uncrustify for code formatting.

commit   : ce0ea2cfab17e6770de3897133183af891df943c    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 12:22:41 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 12:22:41 +0700

Click here for diff

The code is not completely reflowed yet so there are some cases that uncrustify will not catch. The formatting will be improved over time.  
  
Some block of code require special formatting so have been surrounded with the {uncrustify-off}/{uncrustify-on} markers. These exceptions should be kept to a minimum.  
  
Add --code-format (to reformat code) and --code-format-check (to check formatting) to test.pl.  
  
Add a CI test that will check code formatting. Code must be correctly formatted before it can be merge to integration.  
  
Add documentation to the coding standards for code formatting.

A .git-blame-ignore-revs
M .github/workflows/test.yml
M CODING.md
M doc/xml/coding.xml
M src/build/common/render.h
M src/common/encode.c
M src/common/io/tls/server.c
M src/common/memContext.c
M src/common/type/stringId.c
M src/info/info.h
M test/Dockerfile
M test/Vagrantfile
M test/ci.pl
M test/src/module/build/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/repoTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typePackTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/s3Test.c
M test/test.pl
A test/uncrustify.cfg

Reformat code with uncrustify.

commit   : d4070c90641a61fa3cb1169f3bd53067193bab4e    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 11:55:54 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 11:55:54 +0700

Click here for diff

uncrustify has been configured to be as close to the current format as possible but the following changes were required:  
  
* Break long struct initializiers out of function calls.  
* Bit fields get extra spacing.  
* Strings that continue from the previous line no longer indented.  
* Ternary operators that do not fit on a single line moved to the next line first.  
* Align under parens for multi-line if statements.  
* Macros in header #if blocks are no longer indented.  
* Purposeful lack of function indentation in tests has been removed.  
  
Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward.  
  
Support code for uncrustify will be in a followup commit.

M src/build/config/parse.c
M src/build/config/parse.h
M src/build/config/render.c
M src/build/error/parse.c
M src/build/help/parse.c
M src/build/postgres/parse.c
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/blockMap.c
M src/command/backup/file.c
M src/command/check/common.c
M src/command/info/info.c
M src/command/restore/file.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/verify/verify.c
M src/common/assert.h
M src/common/compress/helper.c
M src/common/compress/lz4/compress.c
M src/common/crypto/hash.h
M src/common/debug.h
M src/common/error.c
M src/common/error.h
M src/common/io/http/client.h
M src/common/io/http/common.c
M src/common/io/http/request.h
M src/common/io/http/response.c
M src/common/io/socket/client.h
M src/common/io/socket/server.h
M src/common/io/tls/client.h
M src/common/io/tls/server.h
M src/common/log.h
M src/common/macro.h
M src/common/memContext.c
M src/common/memContext.h
M src/common/stackTrace.h
M src/common/time.c
M src/common/type/blob.h
M src/common/type/pack.h
M src/common/type/string.h
M src/config/load.c
M src/config/parse.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/manifest.c
M src/info/manifest.h
M src/postgres/interface.c
M src/postgres/interface.h
M src/protocol/client.c
M src/protocol/helper.h
M src/storage/azure/storage.c
M src/storage/azure/storage.intern.h
M src/storage/azure/write.c
M src/storage/gcs/storage.c
M src/storage/gcs/storage.intern.h
M src/storage/gcs/write.c
M src/storage/helper.h
M src/storage/posix/storage.c
M src/storage/s3/helper.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/storage.c
M test/src/command/test/define.c
M test/src/command/test/test.c
M test/src/common/harnessDebug.h
M test/src/common/harnessFork.h
M test/src/common/harnessLog.h
M test/src/common/harnessManifest.h
M test/src/common/harnessPostgres.c
M test/src/common/harnessPq.h
M test/src/common/harnessStackTrace.h
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/module/build/configTest.c
M test/src/module/command/annotateTest.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/commandTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/db/dbTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c
M test/src/module/test/testTest.c
M test/src/test.c

Fix formatting errors.

commit   : b2202c36d9e7c4557ac37087757df80193d516b5    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 11:16:31 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 11:16:31 +0700

Click here for diff

Errors in our current (manually-maintained) code format discovered by uncrustify.

M src/build/common/yaml.c
M src/build/common/yaml.h
M src/command/annotate/annotate.c
M src/command/archive/common.c
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/blockIncr.c
M src/command/backup/blockMap.c
M src/command/backup/file.c
M src/command/backup/pageChecksum.c
M src/command/check/check.c
M src/command/check/common.c
M src/command/exit.h
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/repo/create.c
M src/command/repo/get.c
M src/command/restore/restore.c
M src/command/server/ping.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/command/stanza/upgrade.c
M src/command/verify/verify.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/helper.c
M src/common/compress/helper.h
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/cipherBlock.h
M src/common/crypto/common.c
M src/common/crypto/hash.c
M src/common/debug.c
M src/common/encode.c
M src/common/error.c
M src/common/exec.c
M src/common/ini.c
M src/common/io/filter/group.c
M src/common/io/http/session.h
M src/common/io/http/url.c
M src/common/io/http/url.h
M src/common/io/socket/client.c
M src/common/io/socket/common.c
M src/common/io/socket/server.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/common.c
M src/common/io/tls/server.c
M src/common/io/tls/session.c
M src/common/memContext.c
M src/common/stackTrace.c
M src/common/type/blob.c
M src/common/type/buffer.c
M src/common/type/convert.h
M src/common/type/json.c
M src/common/type/list.c
M src/common/type/string.c
M src/common/type/xml.c
M src/config/config.c
M src/config/load.c
M src/config/parse.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/infoPg.h
M src/info/manifest.c
M src/info/manifest.h
M src/postgres/client.h
M src/postgres/interface/version.vendor.h
M src/protocol/command.c
M src/storage/azure/storage.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/helper.c
M src/storage/iterator.c
M src/storage/read.intern.h
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/read.h
M src/storage/s3/helper.c
M src/storage/s3/read.c
M src/storage/s3/read.h
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.h
M test/src/command/test/build.h
M test/src/command/test/define.h
M test/src/common/harnessBackup.c
M test/src/common/harnessConfig.c
M test/src/common/harnessError.c
M test/src/common/harnessLog.c
M test/src/common/harnessManifest.h
M test/src/common/harnessPack.c
M test/src/common/harnessPostgres/harnessVersion.intern.h
M test/src/common/harnessPq.c
M test/src/common/harnessProtocol.c
M test/src/common/harnessServer.c
M test/src/common/harnessStorage.c
M test/src/common/harnessTest.c
M test/src/module/build/commonTest.c
M test/src/module/build/helpTest.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/remoteTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/serverTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/execTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/timeTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Refactor loop limit in removeExpiredArchive().

commit   : a96837d2eda4b7497e74954b24d9323f33465c50    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 10:55:14 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 10:55:14 +0700

Click here for diff

This simplifies the formatting, which was a bit tortured. Also, there is no need to check the limit on each iteration so this is a bit more efficient.

M src/command/expire/expire.c

Add FN_PRINTF() macro.

commit   : f91af305845aa1e99d830a7a6addaee058b518d1    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 10:51:48 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 10:51:48 +0700

Click here for diff

Bring the format(printf) attribute in line with the FN_NO_RETURN and FN_INLINE_ALWAYS macros.  
  
This is simpler to read and can be customized for different compilers.

M meson.build
M src/build.auto.h.in
M src/common/error.h
M src/common/log.h
M src/common/stackTrace.c
M src/common/type/json.h
M src/common/type/string.h
M src/common/type/stringList.h
M src/common/type/stringStatic.h
M src/common/type/stringZ.h
M test/src/common/harnessConfig.h

Refactor stackTraceToZ() to avoid multiple versions of the function.

commit   : f8ea51bf8f4de61fe3541e2f2701b9dccb5ede70    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 10:27:17 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 10:27:17 +0700

Click here for diff

stackTraceToZ() was split this way in c8264291 to allow complete coverage. 0becb6da added a shim to improve coveage but missed simplifying the function.

M src/common/stackTrace.c
M test/src/module/common/stackTraceTest.c

Update config.guess and config.sub to latest versions.

commit   : 0d8e08f69c2ece7a060bd30dff0b375d013fe2fd    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 09:29:07 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 09:29:07 +0700

Click here for diff

M src/build/config.guess
M src/build/config.sub

Begin v2.45 development.

commit   : 240312110c5b1c07deb200b06755a886b723485b    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 09:27:04 +0700    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Jan 2023 09:27:04 +0700

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M meson.build
M src/build/configure.ac
M src/configure
M src/version.h