pgBackRest 2.36 commit log

v2.35: Binary Protocol

commit   : 3787cf78032cb928f27044c13b6f50d57bf06d20    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 06:52:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Aug 2021 06:52:51 -0400    

Click here for diff

IMPORTANT NOTE: The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.  
  
Bug Fixes:  
  
* Detect errors in S3 multi-part upload finalize. (Reviewed by Cynthia Shang, Marco Montagna. Reported by Marco Montagna, Lev Kokotov, Anderson A. Mallmann.)  
* Fix detection of circular symlinks. (Reviewed by Stefan Fercot. Reported by Rohit Raveendran.)  
* Only pass selected repo options to the remote. (Reviewed by David Christensen, Cynthia Shang. Reported by Greg Sabino Mullane, David Christensen.)  
  
Improvements:  
  
* Binary protocol. (Reviewed by Cynthia Shang.)  
* Automatically create data directory on restore. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Chris Bandy.)  
* Allow restore --type=lsn. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang. Suggested by James Coleman.)  
* Change level of backup/restore copied file logging to detail. (Reviewed by Stefan Fercot. Suggested by Jens Wilke.)  
* Loop while waiting for checkpoint LSN to reach replay LSN. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Fatih Mencutekin.)  
* Log backup file total and restore size/file total. (Reviewed by Cynthia Shang.)  
  
Documentation Bug Fixes:  
  
* Fix incorrect host names in user guide. (Reviewed by Stefan Fercot. Reported by Greg Sabino Mullane.)  
  
Documentation Improvements:  
  
* Update contributing documentation and add pull request template. (Contributed by Cynthia Shang. Reviewed by David Steele.)  
* Rearrange backup documentation in user guide. (Reviewed by Cynthia Shang.)  
* Clarify restore --type behavior in command reference. (Contributed by Cynthia Shang. Reviewed by David Steele.)  
* Fix documentation and comment typos. (Contributed by Eric Radman. Reviewed by David Steele.)  
  
Test Suite Improvements:  
  
* Add check for test path inside repo path. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)  
* Add CodeQL static code analysis. (Reviewed by Cynthia Shang.)  
* Update tests to use standard patterns. (Contributed by Cynthia Shang. Reviewed by David Steele.)  

M CONTRIBUTING.md
M README.md
M doc/RELEASE.md
M doc/release.pl
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Update containers for new PostgreSQL releases.

commit   : ad5e06399008118d81f41fb103ef2e980343899b    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 17:25:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 17:25:57 -0400    

Click here for diff

Also add a package that is required for older Debian versions.  

M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Rename stringz.h to stringZ.h.

commit   : 590e75996891ff1328a4a8f58bf833f4b34b6bf6    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 11:57:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 11:57:37 -0400    

Click here for diff

This is more consistent with our file naming conventions.  

M src/common/debug.h
M src/common/type/string.c
R100 src/common/type/stringz.h src/common/type/stringZ.h
M test/code-count/file-type.yaml
M test/src/common/harnessPack.c

Add instructions for dev builds to stress test in user guide.

commit   : 0354144c900eb52d0fa9a9166704d225a968eab5    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 11:35:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 11:35:49 -0400    

Click here for diff

M doc/xml/user-guide.xml

Rearrange MemContext struct members to save space on 64-bit.

commit   : 9b45df70573d54c3a1aa81848642460a9ddc5b28    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 11:12:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 11:12:56 -0400    

Click here for diff

On 64-bit systems this saves a bit of space (10%) due to alignment. 32-bit systems see no benefit.  
  
Also add tests for the individual struct sizes.  

M src/common/memContext.c
M test/src/module/common/memContextTest.c

Mark contextTop static.

commit   : bab7a01f9971b7b45e384141f1777fc86853937e    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 11:00:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Aug 2021 11:00:17 -0400    

Click here for diff

This variable is not used outside of the memContext module so no need for it to be extern'd.  

M src/common/memContext.c

Change ASSERT() to CHECK() in infoLoad().

commit   : bb1ca1d3099db2ccb5cdd11a90a9d62ecabc009b    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 13:32:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 13:32:16 -0400    

Click here for diff

Coverity complains that "Argument loaded of ASSERT() has a side effect because the variable is volatile. The containing function might work differently in a non-debug build."  
  
It does not look like this is a real issue, but a CHECK() here is not too expensive for production so change it to silence Coverity.  
  
Also fix a typo in the comment.  

M src/info/info.c

Remove dead loop in verifyArchive().

commit   : 9ddfcd6a6de5cfd8acf1470216365d6951a692b8    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 13:23:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 13:23:24 -0400    

Click here for diff

This loop has been dead since the code was initially committed in ad79932b. It looks like it was used at one point but became dead when the enclosing if-else was added during development.  
  
Found by Coverity.  

M src/command/verify/verify.c

Add variables for build paths in the user guide.

commit   : 56ad42765311dd399b3b76025ac639c98bcd6027    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 10:52:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 10:52:35 -0400    

Click here for diff

This eliminates repetition of the build path so it can be changed more easily.  
  
Also create the build path explicitly rather than suggest that the user do it.  

M doc/xml/user-guide.xml

Reduce memory requirements of stress test section in the user guide.

commit   : 944a9e8ff10747a4d15187649fcc84308b7d0db9    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 10:17:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 10:17:10 -0400    

Click here for diff

The standby memory was set to 1024mb in 86a651f9 to compensate for a memory leak in restore. The leak has been fixed (or at least mitigated) in e1e6e475 and 4fb6384f so the memory can be reduced to 512mb, the same as the primary.  

M doc/xml/user-guide.xml

Fix infinite loop in protocolServerProcess() on error.

commit   : d69a5963587136b91abb318aa56efc7249c14a64    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 09:48:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 09:48:37 -0400    

Click here for diff

The error was written to the client and then another command read. If the write did not fail then the loop would never exit.  
  
Instead exit on any error that is not raised by the command handler as we can pretty safely assume this is an unrecoverable protocol error. The command handler might throw a protocol error itself, but this should be caught in the next read or write in the main loop.  

M src/protocol/server.c
M test/src/module/protocol/protocolTest.c

Fix EOF behavior of ioReadSmall() to match ioRead().

commit   : 9a6afe3fc16fcb0685613308e47c19f6b11e7e35    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 09:32:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 09:32:20 -0400    

Click here for diff

If the buffer was not full at EOF then ioReadSmall() would get stuck in an infinite loop. Instead, return on EOF even if the buffer is not full.  
  
This is not an issue in released versions since ioReadSmall() is not being used.  
  
Also fix a comment typo.  

M src/common/io/read.c
M test/src/module/common/ioTest.c

Fix more memory leaks introduced by the binary protocol in 6a1c0337.

commit   : 4fb6384f10f6cd26c422ab9b24feb5ae0ebeeee9    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 08:18:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 08:18:11 -0400    

Click here for diff

Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.  

M doc/xml/release.xml
M src/protocol/command.c
M src/protocol/parallel.c

Add temp mem context reset blocks in job processing loops.

commit   : e1e6e47535db09ea20b5f1826ea9e143ac8d614b    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 08:07:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Aug 2021 08:07:41 -0400    

Click here for diff

Restore definitely needed to be doing cleanup, just as backup does. The archive-get, archive-push, and verify loop did not seem to be a significant source of leaks but that could change in the future so add resets.  
  
Add temp mem context blocks in the job callbacks where they were missing.  
  
Also switch to the prior context when creating a job, if possible, to save a move.  

M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/restore/restore.c
M src/command/verify/verify.c

Update config/parse test to use standard patterns.

commit   : eca2fc69589cd44f7c1a43206553984adde5cfb1    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 12 Aug 2021 12:38:07 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 12 Aug 2021 12:38:07 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/config/parseTest.c

Update protocol/protocol test to use standard patterns.

commit   : e17865a03a5451b82a101557d400ff223d104383    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 12 Aug 2021 11:57:17 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 12 Aug 2021 11:57:17 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/protocol/protocolTest.c

Add optional stress test section to the user guide.

commit   : 86a651f9b03890af4f79076600e983bba624f8c5    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Aug 2021 07:49:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Aug 2021 07:49:59 -0400    

Click here for diff

This is intended to provide pre-release stress-testing. Include container memory limits to help check for memory leaks.  
  
Also add parallelism to make for faster builds.  

M doc/RELEASE.md
M doc/xml/user-guide.xml

Log backup file total and restore size/file total.

commit   : a0bdfa436c9e1c2723be905ebe5cef7809a26244    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Aug 2021 13:39:36 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 11 Aug 2021 13:39:36 -0400    

Click here for diff

The backup size was a bit off because it did not include any files (e.g. backup_label, WAL files) that were added to the manifest after the main copy. To fix this move the log message to the very end of the backup.
  

  
Add size/file total log message to restore since it did not exist before.

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/restore/restore.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c

Rearrange backup documentation in user guide.

commit   : 6ab18dc0fa42885aea8eb9405ab0a943c2d49a71    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Aug 2021 12:59:25 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 11 Aug 2021 12:59:25 -0400    

Click here for diff

Remove the "Automatic Stop Option" section since it only applies to PostgreSQL <= 9.6, which will soon be EOL. Since we no longer build the user guide for PostgreSQL < 10 this section was no longer being tested. The stop-auto option is still documented in the reference.
  

  
Move the "Fast Start Option" to "Quick Start - Perform Backup". This is a commonly-used option so it makes sense to mention it earlier. This also makes the backups run more quickly. In the worst case, backups in "Quick Start - Perform Backup" could take minutes to start
  

  
Move the "Archive Timeout" section to "Quick Start - Perform Backup" since it is the last section in "Backup".

M doc/xml/release.xml
M doc/xml/user-guide.xml

Fix use after free introduced by the binary protocol in 6a1c0337.

commit   : f716cb6f4f9765baa03f71b5563786d79d9f04a2    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Aug 2021 14:22:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Aug 2021 14:22:38 -0400    

Click here for diff

The user and group were stored in a temp reset mem context so they could get freed if there were enough files to trigger the reset in storageRemoteInfoList().  
  
Allocate user and group in a mem context provided by the caller to prevent them being freed prematurely.  

M doc/xml/release.xml
M src/storage/remote/storage.c

Add additional checks to performance/storage test.

commit   : 6789ec420ed375663469efcd5957be8d3746efad    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Aug 2021 10:37:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Aug 2021 10:37:37 -0400    

Click here for diff

The storageInfoList() test was broken by 54c4eb0c when the remote was changed to use writeable storage. Since the test driver was being injected into the wrong location, new default storage was created and the test effectively did nothing but still "succeeded".  
  
To prevent this type of regression, add checks to ensure the expected test driver is being used and the callback runs the expected number of times.  

M test/src/module/performance/storageTest.c

Protect against mixed FUNCTION_TEST* and FUNCTION_LOG* macros.

commit   : 0eae815ef4185bbf8bfa27856b531cc08db69d4b    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Aug 2021 17:29:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Aug 2021 17:29:09 -0400    

Click here for diff

If the macros are mixed then the debug stack may not be cleaned up correctly. Add variables to ensure that the macros cannot be mixed.  
  
Fix cases where the macros were mixed and add one missing semicolon.  

M src/common/debug.h
M src/info/infoPg.c
M src/storage/remote/protocol.c

Cleanup inherited clients in protocol test harness.

commit   : c8492aac41f29d8a0c29d7b70d1a5bf4b370965f    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Aug 2021 17:11:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Aug 2021 17:11:39 -0400    

Click here for diff

Cleanup all clients inherited from the parent process so they cannot be accidentally used to send messages to servers that do not belong to this process.  
  
We need to do this carefully so that exit commands are not sent and processes are not terminated, so clear the mem context callback on each object before freeing it.  

M test/src/common/harnessProtocol.c
M test/src/module/storage/remoteTest.c

commit   : 71b654fc29497319862fe01c3917bbc353cc6dd9    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 9 Aug 2021 16:56:06 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 9 Aug 2021 16:56:06 -0400    

Click here for diff

Removed colon from example titles to fix links, fixed test.yml link, and updated the example for the parent/child test process to use the latest macros instead of sleep().

M CONTRIBUTING.md
M doc/xml/contributing.xml
M doc/xml/release.xml

Update db/db test to use standard patterns.

commit   : f653b59664e757a0e3da63cc8462deafe2d9e0ef    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 9 Aug 2021 16:35:48 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 9 Aug 2021 16:35:48 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/db/dbTest.c

Fix memory leaks introduced by the binary protocol in 6a1c0337.

commit   : 2d5cbfc73142fa99df609aa23b64db5342e73c88    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Aug 2021 09:28:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Aug 2021 09:28:17 -0400    

Click here for diff

Additional buffers were being allocated for the protocol messages but not being freed.  
  
Most of the allocations were fairly harness, but storageRemoteOpenReadProtocol() and storageWriteRemote() were problematic because they were allocating (but not freeing) buffers equal to the transfer size of the file. Depending on compression, this could be a lot of memory. Though the memory was freed after each file transfer the aggregate of memory used during parallel processing could overwhelm systems with constrained memory.  
  
Also allocate larger initial buffers in storageRemoteOpenReadProtocol() and storageWriteRemote() so a reallocation is not needed.  

M doc/xml/release.xml
M src/protocol/client.c
M src/protocol/server.c
M src/storage/remote/protocol.c
M src/storage/remote/write.c

Fix test added for lstComparatorZ() in 9abf6a27.

commit   : 33775e53c9160b7ca41c8bcfbf28517f92aff841    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 Aug 2021 11:26:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 Aug 2021 11:26:12 -0400    

Click here for diff

strcmp() returns < 0 and > 0 but these are not guaranteed to be -1 and 1.  

M test/src/module/common/typeListTest.c

Add linefeeds around content even when not in pretty mode.

commit   : 06791284618e8840724caa5e65fa535ea0438b61    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 Aug 2021 09:48:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 Aug 2021 09:48:55 -0400    

Click here for diff

This makes the generated HTML much more readable in diffs because a single word change will not change a line with potentially many tags.  
  
The output is now slightly larger because of the extra linefeeds.  

M doc/lib/pgBackRestDoc/Html/DocHtmlBuilder.pm

Only pass selected repo options to the remote.

commit   : 935de1e7a06d60415c16cad137875b49346337c6    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Aug 2021 14:29:26 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 5 Aug 2021 14:29:26 -0400    

Click here for diff

Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster.
  

  
Also fix an outdated comment.

M doc/xml/release.xml
M src/protocol/helper.c
M test/src/module/protocol/protocolTest.c

Fix incorrect host names in user guide.

commit   : 155ef85013088a1150a4c15b04af36420d06e9c3    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 Aug 2021 14:48:22 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 3 Aug 2021 14:48:22 -0400    

Click here for diff

pg1 was incorrectly used instead of {[host-pg1]} which meant the wrong host name was displayed.
  

  
Also, the install block was installing packages to the build host no matter which host was specified.

M doc/xml/release.xml
M doc/xml/user-guide.xml

Fix typo.

commit   : 07f0a5901f244ccd8f81a07dc95e45744d7e2a55    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 3 Aug 2021 11:48:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 Aug 2021 11:48:58 -0400    

Click here for diff

M src/Makefile.in

Migrate error code generation to C.

commit   : 74c0c44fc8f2ed04653f1486e421c5d5c9d1b0b0    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 18:32:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 18:32:11 -0400    

Click here for diff

Parse src/build/error.yaml and write to src/config/error.auto.h and src/config/error.auto.c.  

D build/lib/pgBackRestBuild/Error/Build.pm
D build/lib/pgBackRestBuild/Error/Data.pm
M src/Makefile.in
R100 build/error.yaml src/build/error/error.yaml
A src/build/error/main.c
A src/build/error/parse.c
A src/build/error/parse.h
A src/build/error/render.c
A src/build/error/render.h
M src/common/error.auto.c
M src/common/error.auto.h
M test/define.yaml
A test/src/module/build/errorTest.c
M test/test.pl

Update error.yaml comment to remove Perl reference.

commit   : aaa9136820d59d17fa81714d7fda56aa84ae31f3    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 18:16:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 18:16:57 -0400    

Click here for diff

M build/error.yaml

Move bldStrId() into a C file.

commit   : 930fee3a0c94944242e5582a5d1c97908b049c81    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 17:49:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 17:49:05 -0400    

Click here for diff

This function was included in a header but not declared inline, so linker errors happened when the header was included into more than one file.  
  
Because of the setjmp() in TRY_BEGIN() it can't be inlined so put it in a C file.  
  
Also add some missing headers.  

M src/Makefile.in
A src/build/common/render.c
M src/build/common/render.h
M test/define.yaml
M test/src/module/build/commonTest.c

Improve comments.

commit   : 13de3615678290c820345eadf67cf4cedfaee222    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 17:41:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 17:41:40 -0400    

Click here for diff

M src/build/config/render.c
M src/build/config/render.h

Increase timeouts in storage/remote test.

commit   : 7542eadc9f073cbcb5e1cb1cb33fe451560bd31d    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 14:48:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 14:48:31 -0400    

Click here for diff

There have been intermittent failures on f33 (with coverage) but not on u16 (without coverage).  
  
Reproducing this reliably has been very difficult, so just try increasing the timeouts. This is based on the observation that tests with coverage take longer than tests without, which may lead the f33 tests to fail if CI is running slower than usual.  
  
This will not increase the runtime of the test unless there is an error.  

M test/src/module/storage/remoteTest.c

Clean src path in test.pl before attempting vpath builds.

commit   : 4e71c077f1be9665ae7570a8a230e4735c4dd592    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 10:00:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 10:00:42 -0400    

Click here for diff

If configure/make has been run in the src path it can conflict with tests, which may require different build options.  
  
Also add a comment when rebuilding for code generation.  

M src/Makefile.in
M test/test.pl

Fix rsync include list removed from test.pl in 9ee9b1fa.

commit   : c192ec4561fa2f870152b31e034243862d2eed82    
  
author   : David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 08:52:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 2 Aug 2021 08:52:49 -0400    

Click here for diff

This caused the include list to be ignored and all files to be rsync'd, which worked but took a much longer.  

M test/test.pl

Remove unused struct member noted in f3f0c64a.

commit   : 2dfca8a61f8305bb94afaf1791af37833e8b0795    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Jul 2021 18:15:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Jul 2021 18:15:07 -0400    

Click here for diff

This variable is used to build the final command role lists, but is not needed after parsing completes.  

M src/build/config/parse.c
M src/build/config/parse.h

Add lstComparatorZ().

commit   : 9abf6a2709e9c1a2cc1c246d8307fd2e2b3d566a    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Jul 2021 18:08:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Jul 2021 18:08:04 -0400    

Click here for diff

Works just like lstComparatorStr() but with zero-terminated strings.  

M src/common/type/list.c
M src/common/type/list.h
M test/src/module/common/typeListTest.c

Add line and column to build yaml error messages.

commit   : 8bca6946b4771a0d438fe1239ca8cc0a8604efca    
  
author   : David Steele <[email protected]>    
date     : Fri, 30 Jul 2021 17:51:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 30 Jul 2021 17:51:56 -0400    

Click here for diff

This makes it much easier to debug errors in the yaml files.  

M src/build/common/yaml.c
M src/build/common/yaml.h
M test/src/module/build/commonTest.c

Remove generated config.auto.c file.

commit   : b47a07b8b9178d2241a9ae06d1a9732337554d21    
  
author   : David Steele <[email protected]>    
date     : Thu, 29 Jul 2021 14:40:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 29 Jul 2021 14:40:30 -0400    

Click here for diff

This file duplicated the command list that already exists in parse.auto.c.  
  
Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/Makefile.in
M src/build/config/render.c
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/command/help/help.c
D src/config/config.auto.c
M src/config/config.c
M src/config/config.h
M src/config/config.intern.h
M src/config/exec.c
M src/config/load.c
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M src/db/helper.c
M src/protocol/helper.c
M test/code-count/file-type.yaml
M test/src/common/harnessConfig.c
M test/src/common/harnessPq.h
M test/src/module/build/configTest.c
M test/src/module/config/parseTest.c

Add check for test path inside repo path.

commit   : e32f9e146b11389e3d16feba9665360b73737d66    
  
author   : David Steele <[email protected]>    
date     : Thu, 29 Jul 2021 11:09:51 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 29 Jul 2021 11:09:51 -0400    

Click here for diff

If the test path is inside the repo path then it can cause strange issues during testing because the entire repo path is duplicated into the test path so that all tests see a consistent view of the repo.
  

  
Another solution might be to pick a better test path name and exclude it from the rsync, but this fix at least addresses the immediate issue.

M .cirrus.yml
M doc/xml/release.xml
M test/test.pl

Remove code made obsolete by f3f0c64a.

commit   : 58cc470b3eae5eeb956be6ce6e809e9110ca2dd8    
  
author   : David Steele <[email protected]>    
date     : Thu, 29 Jul 2021 08:14:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 29 Jul 2021 08:14:17 -0400    

Click here for diff

The code in DocExecute.pm appears to have been obsolete for a long time.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm

Complete migration of config code generation to C.

commit   : f3f0c64a78e2ac425567d792df204e493ad7bb20    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Jul 2021 19:59:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Jul 2021 19:59:23 -0400    

Click here for diff

This was started in c5ae047e but did not include generation of parse.auto.c.  
  
The parser has also been improved with better errors and multiple passes to reduce dependency on ordering and produce and cleaner output.  
  
Option order resolution now includes cycle detection.  

M build/lib/pgBackRestBuild/Config/BuildHelp.pm
D build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/Makefile.in
M src/build/common/render.h
M src/build/config/main.c
M src/build/config/parse.c
M src/build/config/parse.h
M src/build/config/render.c
M src/config/parse.auto.c
M test/code-count/file-type.yaml
M test/src/module/build/configTest.c
M test/test.pl

Fix miscalculation in time option command defaults.

commit   : 7517c5fef8bc43464acf961aded8d5ad3f20b52f    
  
author   : David Steele <[email protected]>    
date     : Wed, 28 Jul 2021 18:53:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 28 Jul 2021 18:53:10 -0400    

Click here for diff

These defaults were not getting the multiplier so the timeouts were much lower than expected.  
  
Since PostgreSQL retries get/push this was probably not a big deal, but it could be critical in the future for a different time value.  

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/config/parse.auto.c

Updates to build/config/config.yaml.

commit   : 9f3c9496358a88d8ad44a0ff1e6dde265345a9c6    
  
author   : David Steele <[email protected]>    
date     : Tue, 27 Jul 2021 14:49:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 27 Jul 2021 14:49:43 -0400    

Click here for diff

Fix booleans that were set to y rather than true and defaults that were set to true instead of 1. Perl was tolerant of these but C is not.  
  
Reorder repo-storage-verify-tls to satisfy inheritance ordering for the dependency.  

M src/build/config/config.yaml

Use consistent separator comment spacing in parse.auto.c.

commit   : 8ca89fb41f85386666242f6c8fbc7d35db9b96ca    
  
author   : David Steele <[email protected]>    
date     : Mon, 26 Jul 2021 14:38:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Jul 2021 14:38:55 -0400    

Click here for diff

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/config/parse.auto.c

Automatically generate option value StringIds in make build-config.

commit   : 52d4574460455975e8929108080dc89bba6cbc7e    
  
author   : David Steele <[email protected]>    
date     : Mon, 26 Jul 2021 12:02:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 26 Jul 2021 12:02:31 -0400    

Click here for diff

Remove strIdGenerate() since bldStrId() performs the same function without cluttering the core code. Since bldStrId() is intended to work in non-debug builds, move the validity checks for input strings out of the DEBUG block.  
  
StringIds are generated as 5/6 bit, whichever is most efficient, for each option value. cfgOptionStrIdInternal() has been updated for this logic.  

M src/build/common/render.h
M src/build/config/render.c
M src/common/type/stringId.c
M src/common/type/stringId.h
M src/config/config.auto.h
M src/config/config.c
M src/config/config.h
M src/storage/azure/storage.h
M src/storage/cifs/storage.h
M src/storage/gcs/storage.h
M src/storage/posix/storage.h
M src/storage/remote/storage.h
M test/src/module/build/configTest.c
M test/src/module/common/typeStringTest.c

Update config tests (except parse) to use standard patterns.

commit   : eeaab6a3d192fc788c0f74aadbf4b3de5da52531    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 23 Jul 2021 16:18:50 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 23 Jul 2021 16:18:50 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/config/execTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/protocolTest.c

Refactor cmdLocal()/cmdRemote() to accept a ProtocolServer object.

commit   : 947c7a84cb37d09ab925785ca7670c970e32b8a3    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Jul 2021 10:35:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Jul 2021 10:35:36 -0400    

Click here for diff

This allows a local/remote to be started independently of server initialization, which will be useful for implementing new transport types, e.g. TLS.  
  
Also remove some dead code in localTest.c.  

M src/command/local/local.c
M src/command/local/local.h
M src/command/remote/remote.c
M src/command/remote/remote.h
M src/main.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c

Move noop from protocolClientNew() to local/remote initialization.

commit   : 0999de0279fabf35af80396a9a8d35546ec02d53    
  
author   : David Steele <[email protected]>    
date     : Fri, 23 Jul 2021 08:32:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 23 Jul 2021 08:32:30 -0400    

Click here for diff

protocolServerNew() does not automatically process a noop so this made the handshake in the constructors asymmetric. This made testing a bit confusing since an extra noop was needed when cmdLocal()/cmdRemote() were not called (since they processed the noop sent by protocolClientNew()).  
  
The extra noops also make complex protocol negotiation (coming in a future commit) more complicated and slower because of the additional round trips.  

M src/protocol/client.c
M src/protocol/helper.c
M test/src/module/command/remoteTest.c
M test/src/module/protocol/protocolTest.c

Update storage tests to use standard patterns.

commit   : 4c9ddf5ef25d0f441e20e918f57c1ec79218cfdb    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 22 Jul 2021 18:17:08 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 22 Jul 2021 18:17:08 -0400    

Click here for diff

The storage tests were not modified to the HRN_STORAGE_* nor TEST_STORAGE_* macros as these test are testing the storage drivers.
  

  
Note that posixTest.c removed an extraneous #endif // TEST_CONTAINER_REQUIRED and #ifdef TEST_CONTAINER_REQUIRED.
  

  
This PR includes all files in the storage/* test directory, namely: azureTest.c, cifsTest.c, gcsTest.c, posixTest.c, remoteTest.c, s3Test.c

M doc/xml/release.xml
M test/src/module/storage/azureTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Update db test module error for newer libpq versions.

commit   : 3a37482de90e132770bf76514baeef907f0c5d80    
  
author   : David Steele <[email protected]>    
date     : Thu, 22 Jul 2021 15:54:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 22 Jul 2021 15:54:30 -0400    

Click here for diff

PostgreSQL 14 libpq throws a more detailed error, so adjust the regular expression to handle the old and the new errors.  

M test/src/module/db/dbTest.c

Fix typo.

commit   : eb98b8d2db94488a47df5727aff6d456869cbaae    
  
author   : David Steele <[email protected]>    
date     : Wed, 21 Jul 2021 13:19:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 21 Jul 2021 13:19:09 -0400    

Click here for diff

M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Update contributing documentation and add pull request template.

commit   : e3d05e2b4aa80149219b6d49a3136b34fa268127    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 21 Jul 2021 13:14:06 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 21 Jul 2021 13:14:06 -0400    

Click here for diff

A .github/pull_request_template.md
M CONTRIBUTING.md
M doc/xml/contributing.xml
M doc/xml/release.xml

Update Vagrantfile box version.

commit   : 46992acd800f7d4fed055a9236f536f2b0f1b0a7    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Jul 2021 15:43:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Jul 2021 15:43:48 -0400    

Click here for diff

M test/Vagrantfile

Update command/restore test to use standard patterns.

commit   : 49925087644c76df005828eb27c24f503a09b20b    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 20 Jul 2021 13:30:17 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 20 Jul 2021 13:30:17 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/command/restoreTest.c

Remove test.pl --smart, --dev, and --dev-test options.

commit   : 9ee9b1fad6fc5aa8f34b0b80d0a9aa8d7e166c40    
  
author   : David Steele <[email protected]>    
date     : Tue, 20 Jul 2021 12:01:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 20 Jul 2021 12:01:10 -0400    

Click here for diff

--smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it.  
  
Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/build/common/render.h
M src/build/config/render.c
M test/test.pl

Allow NULL path in HRN_STORAGE_PATH_REMOVE() macro.

commit   : 1522cb4ed2c493739dfac797fd4cd20e94cbc5ec    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 19 Jul 2021 15:20:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 19 Jul 2021 15:20:43 -0400    

Click here for diff

Update command/backup test to pass NULL where appropriate.  

M test/src/common/harnessStorage.c
M test/src/module/command/backupTest.c

Update command/repo test to use standard patterns.

commit   : a6cdf5b22f0eed53f33bbd84478af0542c81af7d    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 19 Jul 2021 10:47:49 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 19 Jul 2021 10:47:49 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/command/repoTest.c

Update command/local and command/remote tests to use standard patterns.

commit   : ebe5848494d4e37a9c86f1fafb5b3453c1f425a6    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 19 Jul 2021 09:51:11 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 19 Jul 2021 09:51:11 -0400    

Click here for diff

Note that the logging output display of a parent/child test may look jumbled on some systems since the child and parent are attempting to log information at the same time. This is not an issue with the actual test, rather a harness issue that would be beyond the scope of this project to fix.

M doc/xml/release.xml
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c

Partial migration of config code generation to C.

commit   : c5ae047e763f01a520286f9f4de7a85afba53942    
  
author   : David Steele <[email protected]>    
date     : Sun, 18 Jul 2021 19:02:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 18 Jul 2021 19:02:01 -0400    

Click here for diff

Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c.  
  
This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges.  
  
The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.  

M .cirrus.yml
M .github/workflows/test.yml
M CONTRIBUTING.md
D build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildHelp.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M doc/xml/contributing.xml
M doc/xml/user-guide.xml
M src/.gitignore
M src/Makefile.in
A src/build/common/render.h
A src/build/common/yaml.c
A src/build/common/yaml.h
A src/build/config/main.c
A src/build/config/parse.c
A src/build/config/parse.h
A src/build/config/render.c
A src/build/config/render.h
M src/build/configure.ac
M src/config/config.auto.c
M src/config/config.auto.h
M src/configure
M test/Vagrantfile
M test/ci.pl
M test/define.yaml
M test/lib/pgBackRestTest/Common/JobTest.pm
A test/src/module/build/commonTest.c
A test/src/module/build/configTest.c
M test/test.pl

Update order of configuration options.

commit   : 18ffec72da7f44b08343310d2a3b72d11ed5f121    
  
author   : David Steele <[email protected]>    
date     : Sun, 18 Jul 2021 18:31:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 18 Jul 2021 18:31:43 -0400    

Click here for diff

The upcoming C parser requires that options be in dependency order, i.e. inherited options must come before the options that inherit them.  
  
This change has been separated to demonstrate that it does no affect the auto-generated code.  

M src/build/config/config.yaml

Update test container OS versions.

commit   : 6397d735359b459a7c0c094226f266fe39234318    
  
author   : David Steele <[email protected]>    
date     : Sun, 18 Jul 2021 17:29:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 18 Jul 2021 17:29:20 -0400    

Click here for diff

Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial.  
Update Ubuntu 18.04 to 20.04.  
Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up.  
  
Add --no-install-recommends to apt-get commands to save a bit of time and space.  
  
Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small.  
  
Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.  

M .github/workflows/test.yml
M CONTRIBUTING.md
M doc/release.pl
M doc/xml/contributing.xml
M doc/xml/user-guide.xml
M src/protocol/parallel.c
M test/Dockerfile
M test/Vagrantfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
A test/src/valgrind.suppress.co7
D test/src/valgrind.suppress.u12
A test/src/valgrind.suppress.u16
M test/test.pl

Remove test.pl --vm-host option.

commit   : 81602f1593927d34f5a0f547e2a633ac224a4b3d    
  
author   : David Steele <[email protected]>    
date     : Sat, 17 Jul 2021 11:09:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 17 Jul 2021 11:09:53 -0400    

Click here for diff

This option was once used to optimize bin builds but has been defunct for some time.  

M test/ci.pl
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/test.pl

Allow NULL path in TEST_STORAGE_LIST() macro.

commit   : d57299ada8e4e275ed3ee18ca4d7f6d264542154    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 16 Jul 2021 13:07:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 16 Jul 2021 13:07:16 -0400    

Click here for diff

M test/src/common/harnessStorage.c

Update command/help tests to use standard patterns.

commit   : 117ffe88965fd60ce3dcb328e9d28503870af05a    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 15 Jul 2021 17:51:48 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 15 Jul 2021 17:51:48 -0400    

Click here for diff

In the first test (helpRenderSplitSize) added test for empty list and in that and some other tests, the test comment was updated to clarify a bit more what the actual tests is trying to accomplish.
  

  
Note that help test parameters can only use the harnessConfig system when testing option values that have been set since options passed to the help command are not "set" options.

M doc/xml/release.xml
M test/src/module/command/helpTest.c

Update command/backup tests to use standard patterns.

commit   : 4ad0bbda53b34688146f14d130cf0e53c8dd650f    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 15 Jul 2021 17:00:20 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 15 Jul 2021 17:00:20 -0400    

Click here for diff

Includes backup and backupCommon tests.
  

  
Some tests in backupTest were split out where they were originally combined into a single boolean check - which made it difficult to determine which part of the conditional failed.
  

  
String values were also removed where they were no longer needed.

M doc/xml/release.xml
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c

Allow NULL path in HRN_STORAGE_PATH_CREATE() macro.

commit   : cdc89fc9759f1ac37938219b76c89e1c3a82ba0a    
  
author   : David Steele <[email protected]>    
date     : Thu, 15 Jul 2021 16:05:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 15 Jul 2021 16:05:32 -0400    

Click here for diff

M test/src/common/harnessStorage.c

Add contributors missed in 8e187cdb.

commit   : 25508a48e0d3b0a3eabc6d1fe858b9c3dc12dc86    
  
author   : David Steele <[email protected]>    
date     : Thu, 15 Jul 2021 14:34:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 15 Jul 2021 14:34:47 -0400    

Click here for diff

M doc/xml/release.xml

Add noParentCreate parameter to HRN_STORAGE_PATH_CREATE() macro.

commit   : c0feca4673834867d9ea3dbd0434d02a77ed73db    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 15 Jul 2021 13:31:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 15 Jul 2021 13:31:54 -0400    

Click here for diff

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Remove last vestiges of key replacement missed in b270253a.

commit   : 4a6ca54b47b3a56a96b2c92c6e7267fe24997ce9    
  
author   : David Steele <[email protected]>    
date     : Thu, 15 Jul 2021 13:19:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 15 Jul 2021 13:19:49 -0400    

Click here for diff

M test/src/common/harnessTest.h
M test/src/module/command/backupTest.c

Allow restore --type=lsn.

commit   : 563d117967ab9770f9f646c6186d0bc37121611f    
  
author   : Stefan Fercot <[email protected]>    
date     : Thu, 15 Jul 2021 19:02:41 +0200    
  
committer: GitHub <[email protected]>    
date     : Thu, 15 Jul 2021 19:02:41 +0200    

Click here for diff

The LSN recovery target was added in PostgreSQL 10 but did not get added to pgBackRest.

M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/build/config/config.yaml
M src/command/help/help.auto.c
M src/config/config.auto.h
M src/config/parse.auto.c
M test/src/module/command/restoreTest.c
M test/src/module/config/parseTest.c

Loop while waiting for checkpoint LSN to reach replay LSN.

commit   : 8fac1ac50d28557d162fbb251fbbdcc3c9fb950c    
  
author   : Stefan Fercot <[email protected]>    
date     : Thu, 15 Jul 2021 13:45:06 +0200    
  
committer: GitHub <[email protected]>    
date     : Thu, 15 Jul 2021 13:45:06 +0200    

Click here for diff

It is possible for the checkpoint LSN to lag slightly behind the replay LSN until pg_control has been updated.
  

  
Add a loop to keep checking rather than failing when the checkpoint LSN has not been updated.

M doc/xml/release.xml
M src/db/db.c
M test/src/common/harnessPq.h
M test/src/module/db/dbTest.c

Clarify restore --type behavior in command reference.

commit   : a6691c6f615f90195aeb6c1c6657a6afd79a10d3    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 14 Jul 2021 16:06:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Jul 2021 16:06:42 -0400    

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c
M test/src/module/command/helpTest.c

Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros.

commit   : d791bb7298b3ef775f77ad79b28185e255037b09    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Jul 2021 14:31:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Jul 2021 14:31:57 -0400    

Click here for diff

This removes a lot of boiler plate where every instance needs to create these interfaces.  
  
Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes.  
  
In both cases update the tests with the new macros.  

M test/define.yaml
M test/src/common/harnessFork.h
M test/src/common/harnessServer.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/s3Test.c

Error in TEST_STORAGE_LIST() when path does not exist.

commit   : be7b8a485b8f36751347e370c7221c40f0e686fd    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 13 Jul 2021 14:28:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Jul 2021 14:28:58 -0400    

Click here for diff

M test/src/common/harnessStorage.c

Improve HRN_FORK*() macros.

commit   : 1ace1ac938ca555273c150fd3d3b402d75e74ce0    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Jul 2021 14:22:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Jul 2021 14:22:53 -0400    

Click here for diff

Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults.  
  
Add _FD() to macros that retrieve file descriptors to make their purpose clearer.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M test/src/common/harnessFork.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/execTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/forkTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Rename HARNESS_FORK*() macros to HRN_FORK*().

commit   : 76cfbf833d113d6251fb8ffe76239d580dcaf947    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Jul 2021 11:58:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Jul 2021 11:58:23 -0400    

Click here for diff

This matches the new pattern for harness macro naming and is shorter.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M test/src/common/harnessFork.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/execTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/forkTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/gcsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Add ioFdReadNewOpen() and ioFdWriteNewOpen().

commit   : d6797009f8e316e5dc778d42db609f7bd81225ee    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Jul 2021 11:13:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Jul 2021 11:13:57 -0400    

Click here for diff

These functions construct and open in one call, which allows them to be used as function parameters.  

M src/command/local/local.c
M src/command/remote/remote.c
M src/common/exec.c
M src/common/io/fdRead.h
M src/common/io/fdWrite.h
M src/common/io/socket/session.c
M test/src/common/harnessProtocol.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/common/ioTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c

Fix typo.

commit   : f70ddc16d7a12a41f3b757424b73d1f244400d16    
  
author   : Stefan Fercot <[email protected]>    
date     : Tue, 13 Jul 2021 07:09:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Jul 2021 07:09:55 -0400    

Click here for diff

M src/db/db.c

Add compressType parameter to TEST_STORAGE_GET() macro.

commit   : 26dd9188568481815e0e47f19c2acd32d79d9f84    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 12 Jul 2021 15:36:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 12 Jul 2021 15:36:52 -0400    

Click here for diff

Also remove duplicate extension output in HRN_STORAGE_PUT().  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Add cipher parameters to TEST_STORAGE_GET() macro.

commit   : f2ec4e00a4c4c388a8f1b094381fd566a3dc9aa7    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 9 Jul 2021 15:59:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Jul 2021 15:59:17 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Add CodeQL static code analysis.

commit   : 3c8819e10f9eff89195566e7e02f349373f5c4c7    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Jul 2021 14:16:10 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 9 Jul 2021 14:16:10 -0400    

Click here for diff

Also fix some minor issues identified, specifically using gmtime_r()/localtime_r() vs gmtime()/localtime().

M .github/workflows/test.yml
M doc/xml/release.xml
M src/command/backup/common.c
M src/command/info/info.c
M src/command/restore/restore.c
M src/common/io/http/common.c
M src/common/log.c
M src/common/type/convert.c
M src/storage/s3/storage.c

Change level of backup/restore copied file logging to detail.

commit   : 849ab343aa10d66fbbd27cad407eff981c186d06    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Jul 2021 13:50:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Jul 2021 13:50:35 -0400    

Click here for diff

The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.  

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/restore/restore.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c

Fix flapping coverage in storage/remote test.

commit   : a7a041e2412b598ccc01c37123dd6dc13d46cae9    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Jul 2021 11:18:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Jul 2021 11:18:45 -0400    

Click here for diff

The protocol does not put an end message on exit so there was a race between the main process exiting and the remote processes exiting. If the main process exited first then the remote processes might not write coverage data causing coverage to fail.  
  
Fix by calling exit explicitly at the end of the test and update the harness to put an end message so the exits are synchronized.  

M test/src/common/harnessProtocol.c
M test/src/module/storage/remoteTest.c

Make pgControlFromBuffer() static.

commit   : 3369440eef7eb67062de6d0c150f999b342f49b3    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Jul 2021 09:07:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Jul 2021 09:07:41 -0400    

Click here for diff

This function is no longer used externally.  

M src/postgres/interface.c
M src/postgres/interface.h

Update command/control and command/command tests to use standard patterns.

commit   : 8bb0b28455d59ef884eab7ab25a91f988adbab81    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 8 Jul 2021 16:34:11 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 8 Jul 2021 16:34:11 -0400    

Click here for diff

In the commandTest the HRN_STORAGE_REMOVE replacement uses .errorOnMissing when the code being tested added the file. The reason for this is 3 fold:
  

  
1. to ensure that an inadvertent typo in the path/file name does not go undetected,
  
2. to ensure that nothing else has removed the file prior to the call, and
  
3. consistency
  

  
Also, added "stanza" to comment when a stanza stop file is removed vs an "all" stop file.

M doc/xml/release.xml
M test/src/module/command/commandTest.c
M test/src/module/command/controlTest.c

Detect errors in S3 multi-part upload finalize.

commit   : 8e1807cdbe08a6e059ff79220247cf4b3c058979    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Jul 2021 13:06:52 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 8 Jul 2021 13:06:52 -0400    

Click here for diff

Multi-part upload may fail despite returning an HTTP success code. Check for the ETag field in the result and if not present consider the upload to have failed. This will trigger a retry at the local job level.

M doc/xml/release.xml
M src/storage/s3/write.c
M test/src/module/storage/s3Test.c

Fix detection of circular symlinks.

commit   : 43b874628c86ec4fe297dbd5cab5bd3f62b8639b    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Jul 2021 11:30:23 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 8 Jul 2021 11:30:23 -0400    

Click here for diff

Links were followed before they were checked for validity so a circular link would send the manifest build into endless recursion leading to a crash. Fix by moving the recursion after the link check.
  

  
Note that this issue has existed since the C migration and was not introduced by the refactor in eba013b.

M doc/xml/release.xml
M src/info/manifest.c
M test/src/module/info/manifestTest.c

Increase harness log level to detail in command/backup test.

commit   : 683933563333f39852676eb44110e7eab91b63fb    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Jul 2021 11:17:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Jul 2021 11:17:13 -0400    

Click here for diff

Also clean up some unneeded calls to harnessLogLevelReset().  

M test/src/module/command/backupTest.c

Automatically create data directory on restore.

commit   : 056a3070e0dba299389c7b5049878d40dd91ec9a    
  
author   : Stefan Fercot <[email protected]>    
date     : Thu, 8 Jul 2021 16:33:19 +0200    
  
committer: GitHub <[email protected]>    
date     : Thu, 8 Jul 2021 16:33:19 +0200    

Click here for diff

Data directory creation was added during the C migration, but creation of the base data directory (PGDATA) was prevented by a check migrated from Perl.
  

  
Remove the check and update tests to create the data directory at least once.

M doc/xml/release.xml
M src/command/restore/restore.c
M test/src/module/command/restoreTest.c

Allow NULL path in HRN_STORAGE_MODE() macro.

commit   : 34e51ee7b6968d254e0b8c6a0970764ca5cdaa42    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Jul 2021 09:25:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Jul 2021 09:25:42 -0400    

Click here for diff

M test/src/common/harnessStorage.c

Update command/archive tests to use standard patterns.

commit   : 62e8d97af03e7628d5710b17aad1cc31f83d7e69    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 2 Jul 2021 12:22:11 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 2 Jul 2021 12:22:11 -0400    

Click here for diff

Includes archiveCommon, archiveGet and archivePush.
  

  
Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).

M doc/xml/release.xml
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c

Fix documentation and comment typos.

commit   : 23bdc3deb62a622713064df3b0777110ebba8723    
  
author   : Eric Radman <[email protected]>    
date     : Thu, 1 Jul 2021 10:50:03 -0500    
  
committer: GitHub <[email protected]>    
date     : Thu, 1 Jul 2021 10:50:03 -0500    

Click here for diff

Identified using `ag -l | igor`.

M doc/lib/pgBackRestDoc/Common/Doc.pm
M doc/lib/pgBackRestDoc/Latex/DocLatex.pm
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/backup/backup.c
M src/command/expire/expire.c
M src/command/info/info.c
M src/command/repo/ls.c
M src/command/verify/verify.c
M src/common/compress/helper.h
M src/common/memContext.c
M src/common/type/pack.c
M src/common/type/string.h
M src/config/load.c
M src/info/manifest.c
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/src/test.c

Update info/manifest test to use standard patterns.

commit   : 6a90d469094ee2697ae4c932349756a54136799d    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 30 Jun 2021 08:51:11 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 30 Jun 2021 08:51:11 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/info/manifestTest.c

Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants.

commit   : 39c1c1012013f456e951fc149b0f88cbc5dcbcc0    
  
author   : David Steele <[email protected]>    
date     : Mon, 28 Jun 2021 13:28:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 28 Jun 2021 13:28:28 -0400    

Click here for diff

It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M test/src/common/harnessConfig.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c
M test/src/module/db/dbTest.c
M test/src/module/storage/remoteTest.c

Add timeModified parameter to HRN_STORAGE_PUT() macro.

commit   : c6208113bfce9d637ca3fd0b35ab28a819c1a83e    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 28 Jun 2021 12:14:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 28 Jun 2021 12:14:26 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Replace defines with string constants in unit tests.

commit   : 065a8c8454d2acdd72c0c71a24d6b8bc09669e18    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 28 Jun 2021 10:58:27 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 28 Jun 2021 10:58:27 -0400    

Click here for diff

For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes.
  

  
Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).

M doc/xml/release.xml
M test/src/module/command/checkTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/command/verifyTest.c

Update info/infoBackup test to use standard patterns.

commit   : fb2d111dcd75021313a64a348959482339bf7f2a    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 24 Jun 2021 16:11:14 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 24 Jun 2021 16:11:14 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/info/infoBackupTest.c

Add HRN_STORAGE_COPY() macro.

commit   : d0f3a3f2af3f66c899ded98b62695547fd267081    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 24 Jun 2021 14:01:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Jun 2021 14:01:03 -0400    

Click here for diff

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Add expression parameter to TEST_STORAGE_LIST() macro.

commit   : 50c129ae9d161aba080a0d2d1b012e6358ac3e40    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 24 Jun 2021 13:57:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 Jun 2021 13:57:55 -0400    

Click here for diff

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Binary protocol.

commit   : 6a1c0337ddee9900dda58b7d2fa74d82c0586ed2    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 Jun 2021 13:31:16 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 24 Jun 2021 13:31:16 -0400    

Click here for diff

Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol.
  

  
There are a number advantages:
  

  
* The pack type is more compact than JSON and are more efficient to render/parse.
  
* Packs are more strictly typed than JSON.
  
* Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state.
  
* There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files.
  

  
Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C.
  

  
There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/archive/get/protocol.h
M src/command/archive/push/protocol.c
M src/command/archive/push/protocol.h
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/protocol.c
M src/command/backup/protocol.h
M src/command/remote/remote.c
M src/command/restore/protocol.c
M src/command/restore/protocol.h
M src/command/restore/restore.c
M src/command/verify/protocol.c
M src/command/verify/protocol.h
M src/command/verify/verify.c
M src/common/type/pack.h
M src/config/protocol.c
M src/config/protocol.h
M src/db/db.c
M src/db/protocol.c
M src/db/protocol.h
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.c
M src/protocol/command.h
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M src/protocol/server.c
M src/protocol/server.h
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M test/define.yaml
M test/src/module/command/backupTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

Update command/check test to use standard patterns.

commit   : bffb43ea3fe00f445a2fc3c44a07f105cee65b4b    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 24 Jun 2021 12:23:09 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 24 Jun 2021 12:23:09 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/command/checkTest.c

Update command/info test to use standard patterns.

commit   : 17c9ed0ef0380fb14fba9a5299b6618ca8c92c2e    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 24 Jun 2021 08:55:44 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 24 Jun 2021 08:55:44 -0400    

Click here for diff

Some tests had to be reordered or updated, as follows:
  

  
* Reordered tests at line 317 and 331 to avoid unnecessary file removal.
  

  
* Change "stanza found" test at line 1735 to reflect real-life scenario. Originally this test had the cipher-pass environment key set up which caused the RepoGrp to be 2 but with no valid repo path. This resulted in the repo loops executing for the repo2 but since the path was not defined, the tests just reported "none" for cipher which is incorrect since the repo IS encrypted.
  

  
* Moved order of HRN_CFG_LOAD in some tests when able to avoid using storageTest.

M doc/xml/release.xml
M test/src/module/command/infoTest.c

Update command/verify test to use standard patterns.

commit   : ff95eddc369597c76a9ea70ffbada493ed4d1951    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 23 Jun 2021 18:09:46 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Jun 2021 18:09:46 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/command/verifyTest.c

Update info/infoArchive test to use standard patterns.

commit   : 8dd882badee76b038dc8b494a1112eb3e18ef678    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 23 Jun 2021 16:20:47 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Jun 2021 16:20:47 -0400    

Click here for diff

M doc/xml/release.xml
M test/src/module/info/infoArchiveTest.c

Clear error when a CATCH() block finishes.

commit   : e697f5705a04ef431aa93d63764b12ff98d696c6    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 Jun 2021 13:02:19 -0400    
  
committer: GitHub <[email protected]>    
date     : Wed, 23 Jun 2021 13:02:19 -0400    

Click here for diff

It is better to clear errors after the catch block completes rather than leave them set until the next error. This also make is possible to tell when a error is currently being handled, which a function further down the stack might use to modify its behavior. Currently this is only useful in testing, but clearing the error seems like a good idea in general.
  

  
Two places used errors outside the CATCH() block. Mem context cleanup now uses a FINALLY() which is a better implementation anyway. The error handling in main() now calls exitSafe() from withing the CATCH() block.

M doc/xml/release.xml
M src/common/error.c
M src/common/error.h
M src/common/memContext.c
M src/main.c
M test/src/module/common/errorTest.c

Add mode parameter to HRN_STORAGE_PUT() macro.

commit   : 7ba5aef7633fde709eafdfa7b6fed736493250c0    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 23 Jun 2021 12:07:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 23 Jun 2021 12:07:19 -0400    

Click here for diff

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h

Update command/stanza test to use standard patterns.

commit   : 03021c6a17f1374e84ef42614fa1dd2a6be4b64d    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 14 Jun 2021 13:28:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 14 Jun 2021 13:28:27 -0400    

Click here for diff

No core code changes; only changes for stanza-create, stanza-upgrade, and stanza-delete command unit tests.  

M doc/xml/release.xml
M test/src/module/command/stanzaTest.c

Ignore mismatched close requests in the pq shim during error processing.

commit   : 80d63a5e8bc6e5e6d6e4d75b349a44f2aac70954    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Jun 2021 18:00:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Jun 2021 18:00:59 -0400    

Click here for diff

M test/src/common/harnessPq.c

Improve error handling in db test module.

commit   : fba043717935babe084ebd8e9e1b7352dc9ff679    
  
author   : David Steele <[email protected]>    
date     : Sat, 12 Jun 2021 17:48:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 12 Jun 2021 17:48:35 -0400    

Click here for diff

Errors could cause segfaults since server objects were freed immediately but client destructors were run much later.  

M test/src/module/db/dbTest.c

Update pg-path options in db test module to be valid.

commit   : 0a65e59b5591718e46a4c331fa5301c41d49155d    
  
author   : David Steele <[email protected]>    
date     : Fri, 11 Jun 2021 19:16:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 11 Jun 2021 19:16:25 -0400    

Click here for diff

This allows files to be saved in the paths, which would have failed when they were invalid.  

M test/src/module/db/dbTest.c

Add mode, Pack, and StringId to the Pack type.

commit   : 97155bad860010514f06d651ef8103355b346ab8    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Jun 2021 12:40:55 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Jun 2021 12:40:55 -0400    

Click here for diff

Add StringList, which is not a primitive type but rather an array of String types.  
  
Also update pckWriteToLog() to work after pckWriteEnd(), i.e. this->tagStackTop is NULL.  

M src/common/type/pack.c
M src/common/type/pack.h
M test/src/common/harnessPack.c
M test/src/module/common/typePackTest.c

Add pckReadMove() and pckWriteMove().

commit   : 85a55bd40181bad943e224619fc936b046b2c8c1    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Jun 2021 09:25:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Jun 2021 09:25:57 -0400    

Click here for diff

Move a PackRead or PackWrite object to a new mem context.  
  
Also note that these functions may not work as expected with pack objects created by pckReadNewBuf() and pckWriteNewBuf() since the pack object does not have ownership of the passed buffer and cannot move it.  

M src/common/type/pack.c
M src/common/type/pack.h
M test/src/module/common/typePackTest.c

Skip sleep in sleepMSec() when sleep time is zero.

commit   : 788f7c1f30408b6eb3148cb288b67838e6cb718b    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Jun 2021 09:24:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Jun 2021 09:24:42 -0400    

Click here for diff

There is no point in sleeping when there is no sleep time.  
  
The advantage is that callers do not need to perform the check themselves.  

M src/common/time.c
M test/src/module/common/timeTest.c

Add error test harness/shim.

commit   : 5e1a8e6895ec59005a3974ac63ecbbc6832d5244    
  
author   : David Steele <[email protected]>    
date     : Thu, 10 Jun 2021 09:21:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 10 Jun 2021 09:21:15 -0400    

Click here for diff

The hrnErrorThrowP() macro allows errors with specified fields to be generated, which simplifies testing.  
  
Update the common/exit test to use the new macro.  

M test/define.yaml
A test/src/common/harnessError.c
A test/src/common/harnessError.h
M test/src/module/common/exitTest.c

Add HRN_STORAGE_MOVE() macro.

commit   : c5897007c4cb420c1e4111c9f9c14e1e706ec5e9    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 9 Jun 2021 12:41:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Jun 2021 12:41:23 -0400    

Click here for diff

Update command/expire test to show how it is used.  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/expireTest.c

Use lower-case names from Azure identifiers in integration tests.

commit   : 15dfbf4c1d2fd33daa1a89a644683e80eeae4194    
  
author   : David Steele <[email protected]>    
date     : Wed, 9 Jun 2021 12:28:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 9 Jun 2021 12:28:40 -0400    

Click here for diff

Azurite, which is used for testing, did not enforce this before so the capital letters were not a problem. Now Azurite enforces the same rules as Azure so use lower-case identifiers instead.  
  
These names were only used in integration tests so there was no production impact.  

M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Env/Host/HostAzureTest.pm

Add optional remove to TEST_STORAGE_EXISTS().

commit   : c6a8528e3158de3e4ccdba00c7af6d47cc28d073    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Jun 2021 14:51:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Jun 2021 14:51:23 -0400    

Click here for diff

This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before.  
  
Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.  

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/expireTest.c

Add support for more Pack types.

commit   : 4a075b725287aafe12bb2add89ae4b224dafe2bc    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Jun 2021 12:55:00 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 8 Jun 2021 12:55:00 -0400    

Click here for diff

Since the pack type was stored in 4 bits, only 15 values were allowed (0 was reserved).
  

  
Allow virtually unlimited types by storing type info in a base-128 encoded integer following the tag when the type bits in the tag are set to 0xF.
  

  
Also separate the type IDs used in the pack (PackTypeMap) from those presented to the user (PackType). The prior PackType enum exposed implementation details to the user, e.g. pckTypeUnknown.

M build/lib/pgBackRestBuild/Config/BuildHelp.pm
M doc/xml/release.xml
M src/command/help/help.auto.c
M src/common/type/pack.c
M src/common/type/pack.h
M test/src/common/harnessPack.c
M test/src/module/common/typePackTest.c

Update command/expire test to use standard patterns.

commit   : 3f9fbc3c24eab48e7ae2871484ea01273049e601    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 8 Jun 2021 11:00:28 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 8 Jun 2021 11:00:28 -0400    

Click here for diff

The way tests are written has evolved over time. Update the command/expire test to use the new test patterns.

M doc/xml/release.xml
M test/src/module/command/expireTest.c

Fix comments.

commit   : 12f81f8933b31d528a04b3640f407c9ff7ce9c45    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Jun 2021 09:56:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Jun 2021 09:56:24 -0400    

Click here for diff

The extern'd function should not have duplicated the comment in the header. The static helper needed a comment.  

M src/db/helper.c

Rename param structs to match function names.

commit   : 4ccb42bb7a84d17bd708b1a4c117037738ef1f6f    
  
author   : David Steele <[email protected]>    
date     : Tue, 8 Jun 2021 08:44:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 8 Jun 2021 08:44:10 -0400    

Click here for diff

The functions were named with short integer representations (e.g. I32) but the param structs were using longer ones, e.g. UInt32. Shorten the integer representations in the param structs to match.  
  
Also rename pckReadUInt64Internal() to pckReadU64Internal() for the same reason.  

M src/common/type/pack.c
M src/common/type/pack.h
M test/src/module/common/typePackTest.c

Switch order of remote startup in storage/remote test.

commit   : df8276f59f4271a80bcf4f0ca13089864d9b95cb    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 11:53:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 11:53:25 -0400    

Click here for diff

The pg storage must be started before the repo storage to set the max remotes allowed to 2. The protocol helper expects all remotes to have the same type so we are cheating here a bit, but without this ordering the second remote will never be sent an explicit exit and may not save coverage data.  

M test/src/module/storage/remoteTest.c

Remove parse.auto.h.

commit   : a76cb57c80026e713886bc3205e9ef60fd1e4070    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 08:48:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 08:48:09 -0400    

Click here for diff

At one time there was a lot more in this header but it got merged with the enums and constants in config.auto.c, leaving only the ConfigOptionType enum.  
  
Auto-generating the ConfigOptionType enum is not very useful since new option types require code in parse.c.  

M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/config/config.intern.h
D src/config/parse.auto.h
M src/config/parse.h

Remove unused command string constants.

commit   : 7a449f0e49df4c03bad8a2c468a3fd5adb1e5732    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 08:18:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 08:18:02 -0400    

Click here for diff

M build/lib/pgBackRestBuild/Config/Build.pm
M src/config/config.auto.c
M src/config/config.auto.h

Handle NULL VariantList in JSON.

commit   : a607750be1775fa9f74164f8d3768536b86387f6    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 07:46:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 07:46:51 -0400    

Click here for diff

This worked if the Variant was NULL but not if the Variant contained a NULL.  

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Update config.guess and config.sub to latest versions.

commit   : aaca4df983e280f8b2357ba923496f4618c72713    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 07:36:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 07:36:56 -0400    

Click here for diff

M src/build/config.guess
M src/build/config.sub

Begin v2.35 development.

commit   : a7b1d2b9d00014172edd963229ca4d40a3855da7    
  
author   : David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 07:32:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 7 Jun 2021 07:32:34 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h