pgBackRest 2.27 commit log

v2.26: Non-blocking TLS

commit   : 2352ecf7c0db8d12cda4759709b4cd570e1b5de4    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 15:44:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 15:44:31 -0400    

Click here for diff

Bug Fixes:  
  
* Remove empty subexpression from manifest regular expression. MacOS was not happy about this though other platforms seemed to work fine. (Fixed by David Raftis.)  
  
Improvements:  
  
* Non-blocking TLS implementation. (Reviewed by Slava Moudry, Cynthia Shang, Stephen Frost.)  
* Only limit backup copy size for WAL-logged files. The prior behavior could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup. (Reviewed by Cynthia Shang.)  
* TCP keep-alive options are configurable. (Suggested by Marc Cousin.)  
* Add io-timeout option.  

M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Specify that the io-timeout option is measured in seconds.

commit   : 93e4fe0199bdff3de34eee92b6ace99dab1acb98    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 20 Apr 2020 13:11:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 13:11:34 -0400    

Click here for diff

M doc/xml/reference.xml
M src/config/define.auto.c

Remove stray copy of LibCAuto.pm.

commit   : 1c319e051b26930da69bef8eebbe1414db820a1a    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 11:49:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 11:49:11 -0400    

Click here for diff

This was removed in 79cfd3ae but accidentally added back in 2fa69af8, probably due to an incorrectly resolved merge conflict.  

D test/lib/pgBackRestTest/LibCAuto.pm

Clear error queue before each SSL_*() call.

commit   : 4b28f79c92b497eb336ab9b180f029a6bffda5c1    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 11:08:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 11:08:58 -0400    

Click here for diff

The documentation recommends clearing the error queue before each SSL_*() call.  
  
Since we always check the results of SSL_*() for errors instead of blindly calling SSL_get_error() it's not clear this makes any difference, but it still seems like a good idea to be sure there are no stray errors in the queue.  

M src/common/io/tls/session.c

Add expire to commands that support io-timeout.

commit   : 6f41ce1182c3417f4a7844cbe5de5875c27e4363    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 09:45:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Apr 2020 09:45:08 -0400    

Click here for diff

The command list was not complete. Copy the command list from buffer-size to make sure all commands are included.  

M build/lib/pgBackRestBuild/Config/Data.pm
M src/config/define.auto.c

Add sck-block option.

commit   : 21a1cf9a2f18cc294474efda064a2ae08cc8d18d    
  
author   : David Steele <[email protected]>    
date     : Fri, 17 Apr 2020 09:55:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 17 Apr 2020 09:55:40 -0400    

Click here for diff

This option puts sockets into blocking mode.  
  
It is only intended for internal use and is not included in the test matrix.  

M build/lib/pgBackRestBuild/Config/Data.pm
M src/common/io/socket/common.c
M src/common/io/socket/common.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M test/src/module/common/ioTlsTest.c
M test/src/module/config/loadTest.c
M test/src/test.c

Add io-timeout option.

commit   : 5d25e508aeb8febd509f26225425b391fe45ad06    
  
author   : David Steele <[email protected]>    
date     : Fri, 17 Apr 2020 09:18:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 17 Apr 2020 09:18:52 -0400    

Click here for diff

Timeout used for connections and read/write operations.  
  
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/repo/put.c
M src/common/io/io.c
M src/common/io/io.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/storage/helper.c
M src/storage/s3/storage.h
M test/src/module/command/helpTest.c
M test/src/module/common/ioTest.c
M test/src/module/config/loadTest.c

Non-blocking TLS implementation.

commit   : c88684e2bf32cef8cc185cca4ef213457b03fa62    
  
author   : David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 16:05:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 16:05:44 -0400    

Click here for diff

The prior blocking implementation seemed to be prone to locking up on some (especially recent) kernel versions. Since we were unable to reproduce the issue in a development environment we can only speculate as to the cause, but there is a good chance that blocking sockets were the issue or contributed to the issue.  
  
So move to a non-blocking implementation to hopefully clear up these issues. Testing in production environments that were prone to locking shows that the approach is promising and at the very least not a regression.  
  
The main differences from the blocking version are the non-blocking connect() implementation and handling of WANT_READ/WANT_WRITE retries for all SSL*() functions.  
  
Timeouts in the tests needed to be increased because socket connect() and TLS SSL_connect() were not included in the timeout before. The tests don't run any slower, though. In fact, all platforms but Ubuntu 12.04 worked fine with the shorter timeouts.  

M doc/xml/release.xml
M src/common/io/socket/client.c
M src/common/io/socket/common.c
M src/common/io/socket/common.h
M src/common/io/tls/session.c
M test/define.yaml
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/s3Test.c

Use poll() instead of select() for monitoring socket read/write ready.

commit   : 2260a7512aab7a50473466cc524aabb2a5002edf    
  
author   : David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 15:02:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 15:02:33 -0400    

Click here for diff

select() is a bit old-fashioned and cumbersome to use. Since the select() code needed to be modified to handle write ready this seems like a good time to upgrade to poll().  
  
poll() has been around for a long time so there doesn't seem to be any need to provide a fallback to select().  
  
Also change the error on timeout from FileReadError to ProtocolError. This works better for read vs. write and failure to poll() is indicative of a protocol error or unexpected EOF.  

M doc/xml/release.xml
M src/common/io/socket/common.c
M src/common/io/socket/common.h
M src/common/io/socket/session.c
M src/common/io/socket/session.h
M src/common/io/tls/session.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c

Only limit backup copy size for WAL-logged files.

commit   : e5e81d3839648617266285ca519f76835b79f611    
  
author   : David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 14:48:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 14:48:16 -0400    

Click here for diff

The prior behavior introduced in dcddf3a5 could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup since they are copied via tmp files and could change size during the backup.  
  
In general it seems safer to limit this feature to WAL-logged files which will be reconstructed during recovery.  

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/protocol.c
M test/src/module/command/backupTest.c

Add Cirrus CI and Vagrant configuration for FreeBSD and MacOS.

commit   : 85f4eef55e62dcaf05705ce3123cb04b59c2a1f4    
  
author   : David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 14:13:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 14:13:53 -0400    

Click here for diff

Building on these platforms gives us better coverage for our build code. Cirrus CI was chosen because it is the only service that supports FreeBSD (that we could find).  
  
The FreedBSD configuration for Vagrant is currently just enough to perform a build.  
  
The MacOS configuration is not actually for Vagrant (yet) but does show the steps needed to setup the build environment on MacOS.  

A .cirrus.yml
M test/Vagrantfile

Make existing Vagrantfile configuration the default.

commit   : 09fc69f3e8a0f8ccf6ce95b5d7e59b68b00d14ff    
  
author   : David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 13:58:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 13:58:46 -0400    

Click here for diff

This allows us to add new configurations mostly without changing the behavior of vagrant from the command line, i.e. vagrant up and vagrant ssh will continue to bring up the default configuration.  
  
However, vagrant destroy -f will remove all configurations. That's really only a change in behavior if more than one configuration is running, which is not currently possible.  

M test/Vagrantfile

Fix headers for FreeBSD/MacOS.

commit   : 34d7338604b423b858e315db58a7a0976ad30e17    
  
author   : David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 11:50:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 16 Apr 2020 11:50:35 -0400    

Click here for diff

It looks like Linux is tolerant of the BSD headers so remove the conditionals.  
  
Eventually it might be a good idea to include these based on configure rules but that seems over overkill for now.  

M src/common/io/socket/client.c
M src/common/io/socket/common.c

Classify all .h files as C.

commit   : 492180837ee654bb5052bbaca3d04111ee89a4c6    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 18:03:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 18:03:41 -0400    

Click here for diff

Github classifies many C header files as C++, perhaps because they don't contain anything definitively C-like.  
  
Add an override to ensure .h files are always classified as C since this project contains no C++.  

A .gitattributes

Remove empty subexpression from manifest regular expression.

commit   : 0c07c5111fb73821abed678166cc7bfb132f2285    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:33:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:33:43 -0400    

Click here for diff

MacOS was not happy about this though other platforms seemed to work fine.  
  
This was likely a typo or a relic from refactoring.  

M doc/xml/release.xml
M src/info/manifest.c

Remove incorrect Github id.

commit   : 8fb0a8fd6ed7453202ca6bee6516bb5489f9a555    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:28:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:28:43 -0400    

Click here for diff

This was a copy-pasto from another contributor entry.  

M doc/xml/release.xml

Fix typo.

commit   : 02ebba02b29739b0ff7a1c8e3833c3ff8732041c    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 14 Apr 2020 15:23:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:23:19 -0400    

Click here for diff

M test/src/common/harnessTls.c

Use SocketSession/TlsSession for test servers.

commit   : 9ffa2c618fe54710a37448302a0efa1368a041f0    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:22:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:22:49 -0400    

Click here for diff

A session looks much the same whether it is initiated from the client or the server, so use the session objects to implement the TLS, HTTP, and S3 test servers.  
  
For TLS, at least, there are some differences between client and server sessions so add a client/server type to SocketSession to determine how the session was initiated.  
  
Aside from reducing code duplication, the main advantage is that the test server will now timeout rather than hanging indefinitely when less input that expected is received.  

M doc/xml/release.xml
M src/common/io/socket/client.c
M src/common/io/socket/session.c
M src/common/io/socket/session.h
M src/common/io/tls/session.c
M src/common/io/tls/session.h
M test/src/common/harnessTls.c
M test/src/module/common/ioTlsTest.c

Always throw error when OpenSSL returns SSL_ERROR_SYSCALL.

commit   : 71fb28bf3f7387a225c8a32b810772f3f2b09820    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:20:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:20:50 -0400    

Click here for diff

Previously an error was only thrown when errno was set but in practice this is usually not the case. This may have something to do with getting errno late but attempts to get it earlier have not been successful. It appears that errno usually gets cleared and spot research seems to indicate that other users have similar issues.  
  
An error at this point indicates unexpected EOF so it seems better to just throw an error all the time and be consistent.  
  
To test this properly our test server needs to call SSL_shutdown() except when the client expects this error.  

M doc/xml/release.xml
M src/common/io/tls/session.c
M test/src/common/harnessTls.c
M test/src/common/harnessTls.h
M test/src/module/common/ioTlsTest.c

Split session functionality of TlsClient out into TlsSession.

commit   : 9f2d647badb666c31ba25c4df8c7294331287a23    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:02:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 15:02:18 -0400    

Click here for diff

This abstraction allows the session code to be shared between the TLS client and (upcoming) server code.  
  
Session management is no longer implemented in TlsClient so the HttpClient was updated to free and create sessions as needed. No test changes were required for HttpClient so the functionality should be unchanged.  
  
Mechanical changes to the TLS tests were required to use TlsSession where appropriate rather than TlsClient. There should be no change in functionality other than how sessions are managed, i.e. using tlsClientOpen()/tlsSessionFree() rather than just tlsClientOpen().  

M doc/xml/release.xml
M src/Makefile.in
M src/common/io/http/client.c
M src/common/io/socket/client.c
M src/common/io/socket/client.h
M src/common/io/socket/session.c
M src/common/io/socket/session.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
A src/common/io/tls/session.c
A src/common/io/tls/session.h
A src/common/io/tls/session.intern.h
M test/define.yaml
M test/src/module/common/ioTlsTest.c

Update contributing.xml to use Ubuntu 18.04.

commit   : c9481bb95f1edaa93b26f8fce767950e00b40543    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 14:51:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 14:51:14 -0400    

Click here for diff

Ubuntu 19.04 was EOL in January and has been removed from the repository servers.  
  
It may be best to stick to LTS versions from now on.  

M CONTRIBUTING.md
M doc/xml/contributing.xml

Use __noreturn_ on error functions when coverage testing.

commit   : f03d1b5b7b880b25a19a087ec5e29e3393e2ce43    
  
author   : David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 11:43:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 14 Apr 2020 11:43:50 -0400    

Click here for diff

The errorInternalThrowSys*() functions were marked as returning during coverage testing even when they had no possibility to return, i.e. the error parameter was set to constant true. This meant the compiler would treat the functions as returning even when they would not.  
  
Instead create completely separate functions for coverage to use for THROW_ON_SYS_ERROR*() that can return and leave the regular functions marked __noreturn__.  

M doc/xml/release.xml
M src/common/error.c
M src/common/error.h
M test/src/module/common/errorTest.c

Split session functionality of SocketClient out into SocketSession.

commit   : b7d8d61526e1ed7e617953fc36736d89f9295252    
  
author   : David Steele <[email protected]>    
date     : Mon, 13 Apr 2020 16:59:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 13 Apr 2020 16:59:02 -0400    

Click here for diff

This abstraction allows the session code to be shared between the socket client and (upcoming) server code. There should no difference in how the code works -- only the organization has changed. Note that no changes to the tests were required.  
  
This same abstraction will be required for TlsClient but that will be done in a separate commit because it requires test changes.  

M doc/xml/release.xml
M src/Makefile.in
M src/common/io/socket/client.c
M src/common/io/socket/client.h
A src/common/io/socket/session.c
A src/common/io/socket/session.h
M src/common/io/tls/client.c
M test/define.yaml

Remove extraneous assignment.

commit   : b5347070af5e3e5b9d6a44555aed0212b5cd022a    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 13 Apr 2020 16:44:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 13 Apr 2020 16:44:29 -0400    

Click here for diff

This is likely leftover cruft from a prior refactor.  

M src/common/io/tls/client.c

Improve comments and fix typos in the storage interface.

commit   : 9d2223b18881bf862d6ca5da7a2ad28b86421fb1    
  
author   : David Steele <[email protected]>    
date     : Mon, 13 Apr 2020 14:57:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 13 Apr 2020 14:57:01 -0400    

Click here for diff

M src/storage/storage.intern.h

Correct option name in test.pl help.

commit   : 310d42ca35431bb196dfc81300b7d8dfa96bbd35    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 13 Apr 2020 14:51:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 13 Apr 2020 14:51:04 -0400    

Click here for diff

M test/test.pl

Use standard HARNESS_FORK*() macros to fork test servers.

commit   : 71ce6375571bf9253b7aebad5ee44774fb3cbb07    
  
author   : David Steele <[email protected]>    
date     : Sun, 12 Apr 2020 09:01:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 12 Apr 2020 09:01:41 -0400    

Click here for diff

These forks were done in a custom way (not sure why) and lack the capability of the standard macros for the parent to wait for child exit.  
  
This mean that the server would continue to run after the tests were complete and that multiple servers could run at once. This caused subtle timing and connection issues that required larger timeouts to resolve.  
  
Don't change the timeouts here since they need to be adjusted in future commits anyway.  

M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/s3Test.c

Add comment.

commit   : 42246401b8805c14d8b2a6938dfef152d2905d70    
  
author   : David Steele <[email protected]>    
date     : Sat, 11 Apr 2020 16:24:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 11 Apr 2020 16:24:45 -0400    

Click here for diff

M src/common/io/http/client.c

Suppress uncoverable error in common/io-tls test module.

commit   : 674b65886face9b2f7966c4f9ff1dc321b90e61c    
  
author   : David Steele <[email protected]>    
date     : Sat, 11 Apr 2020 16:21:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 11 Apr 2020 16:21:03 -0400    

Click here for diff

It is pretty much impossible for a static IP to not resolve to an address but in theory the error could catch other conditions so it seems best to keep it.  

M test/src/module/common/ioTlsTest.c

Fix typo.

commit   : 9a751c2d98f27c4e270d6d47c0262cad68aee68f    
  
author   : David Steele <[email protected]>    
date     : Fri, 10 Apr 2020 12:56:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 10 Apr 2020 12:56:28 -0400    

Click here for diff

M doc/xml/user-guide.xml

Update contributing documentation and remove test/README.md.

commit   : 63b9f46a34ce20c172394a625858d807b1e2a5a9    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 9 Apr 2020 18:25:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Apr 2020 18:25:25 -0400    

Click here for diff

When the Vagrant file was updated to use pgbackrest/ vs /backrest/ as the location for executing tests and building the documentation, parts of the contributing.xml (and hence the CONTRIBUTING.md) were not updated since some parts of the document are not actually executed when the CONTRIBUTING.md is built from contributing.xml: those parts that are executed were updated but those parts that are not executed were not.  
  
This commit fixes the contributing.xml issue but also removes test/README.md as its contents were out of date and redundant given that they are covered in CONTRIBUTING.md.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
D test/README.md

Allow zero wait times in Wait object.

commit   : 55c3a3c8d3d58c4b02327be39b3cf080f3805358    
  
author   : David Steele <[email protected]>    
date     : Thu, 9 Apr 2020 18:13:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 9 Apr 2020 18:13:35 -0400    

Click here for diff

This limitation forced extra logic in cases where zero wait times were needed.  
  
Remove the limitation and the extra logic in cases where zero wait times are possible.  

M src/command/archive/common.c
M src/common/io/http/client.c
M src/common/io/socket/client.c
M src/common/io/tls/client.c
M src/common/lock.c
M src/common/wait.c
M src/storage/storage.c
M test/src/module/common/waitTest.c

Add some debugging to tests that fork servers.

commit   : c292e8957ded825483f08060ce8857ad935637fd    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 14:56:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 14:56:09 -0400    

Click here for diff

Help identify whether errors are happening in the forked server or the main test by showing the line number where the server was forked off in the stack trace.  

M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/s3Test.c

Reset line number in TEST_ERROR*() macros.

commit   : 099bd85ed7cdca017fbfae4d12134a00fcf2236b    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 14:42:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 14:42:05 -0400    

Click here for diff

If these are not reset then an error not wrapped in a TEST_ERROR*() macro may show the line number of the previous error in a stack trace, which is confusing.  
  
It is better for the line number to be unreported than wrong.  

M test/src/common/harnessTest.h

Add waitRemaining() to get remaining wait time.

commit   : 627b49535222bf8408b2593b633a79c8eba3b832    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 14:35:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 14:35:36 -0400    

Click here for diff

This can be used to set timeouts so they do not wait longer than needed.  

M src/common/wait.c
M src/common/wait.h
M test/src/module/common/waitTest.c

Allow default process id to be specified in logInit().

commit   : ac3cfa4c9c31a00fb9f88ad65430cd0c550fee49    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 14:23:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 14:23:31 -0400    

Click here for diff

The default process id was previously always 0 but there are cases where it is useful to be able to set the default.  
  
Currently the only use case is for testing but the upcoming server code will also make use of it.  

M src/common/log.c
M src/common/log.h
M src/config/load.c
M src/config/parse.c
M test/src/common/harnessFork.h
M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/module/command/archiveGetTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/logTest.c
M test/src/module/storage/s3Test.c

Add ideator to release note.

commit   : 2ec379fa40142462fd018a385affa14346ca76e7    
  
author   : David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 11:10:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 7 Apr 2020 11:10:58 -0400    

Click here for diff

M doc/xml/release.xml

Simplify storage driver info and list functions.

commit   : 5e55d588507e7948dbb48bb8d9427ab4d4503c20    
  
author   : David Steele <[email protected]>    
date     : Mon, 6 Apr 2020 16:09:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 6 Apr 2020 16:09:18 -0400    

Click here for diff

The storage driver requires two list functions to be implemented, list and infoList. But the former is a subset of the latter so implementing both in every driver is wasteful. The reason both exist is that in Posix it is cheaper to get a list of names than it is to stat files to get size, time, etc. In S3 these operations are equivalent.  
  
Introduce storageInfoLevelType to determine the amount of information required by the caller. That way Posix can work efficiently and all drivers can return only the data required which saves some bandwidth. The storageList() and storageInfoList() functions remain in the storage interface since they are useful -- the only change is simplifying the drivers with no external impact.  
  
Note that since list() accepted an expression infoList() must now do so. Checking the expression is optional for the driver but can be used to limit results or save IO costs.  
  
Similarly, exists() and pathExists() are just specialized forms of info() so adapt them to call info() instead.  

M doc/xml/release.xml
M src/storage/info.h
M src/storage/posix/storage.c
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/storage.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/define.yaml
M test/src/common/harnessStorage.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/controlTest.c
M test/src/module/config/parseTest.c
M test/src/module/performance/storageTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Fix issue with THROW*_SYS_ERROR_CODE*() using wrong errno.

commit   : 7679f8f88622cd14d442e69d133bd108783cdd80    
  
author   : David Steele <[email protected]>    
date     : Sat, 4 Apr 2020 13:59:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 4 Apr 2020 13:59:50 -0400    

Click here for diff

When DEBUG_COVERAGE was defined errno was being used instead of the value being passed. This apparently worked by happenstance in the single existing usage but it won't work in general.  

M src/common/error.c
M src/common/error.h

Standardize some section names in headers.

commit   : f984aec66508354e8537b3f8c226f0e574592306    
  
author   : David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 18:15:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 18:15:32 -0400    

Click here for diff

It's better to start out with plural forms rather than flip back and forth as functions are added and subtracted. So, use "Constructors" instead of "Constructor".  
  
Use "Getters/Setters" rather than "Getters" or "Setters" to avoid similar churn.  

M src/common/compress/gz/compress.h
M src/common/compress/gz/decompress.h
M src/common/compress/lz4/decompress.h
M src/common/exec.h
M src/common/ini.h
M src/common/io/bufferRead.h
M src/common/io/bufferWrite.h
M src/common/io/filter/buffer.h
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.h
M src/common/io/filter/sink.h
M src/common/io/filter/size.h
M src/common/io/handleRead.h
M src/common/io/handleWrite.h
M src/common/io/http/cache.h
M src/common/io/http/client.h
M src/common/io/http/header.h
M src/common/io/http/query.h
M src/common/io/read.intern.h
M src/common/io/socket/client.h
M src/common/io/tls/client.h
M src/common/io/write.intern.h
M src/common/regExp.c
M src/common/regExp.h
M src/common/type/buffer.h
M src/common/type/list.h
M src/common/type/mcv.h
M src/common/type/stringList.h
M src/common/type/variant.h
M src/common/wait.h
M src/db/db.h
M src/info/info.h
M src/info/infoArchive.h
M src/info/infoPg.h
M src/postgres/client.h
M src/protocol/client.h
M src/protocol/command.h
M src/protocol/helper.c
M src/protocol/helper.h
M src/protocol/parallel.h
M src/protocol/parallelJob.h
M src/protocol/server.h
M src/storage/cifs/storage.h
M src/storage/posix/read.h
M src/storage/posix/storage.h
M src/storage/posix/storage.intern.h
M src/storage/posix/write.h
M src/storage/read.h
M src/storage/read.intern.h
M src/storage/remote/read.h
M src/storage/remote/storage.h
M src/storage/remote/write.h
M src/storage/s3/read.h
M src/storage/s3/storage.h
M src/storage/s3/write.h
M src/storage/storage.h
M src/storage/storage.intern.h
M src/storage/write.h
M src/storage/write.intern.h
M test/src/common/harnessLog.h
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTls.h

Move extern function comments to headers.

commit   : 1aca2cc90290ffefcc95530dba55e19313754219    
  
author   : David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 18:01:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 18:01:28 -0400    

Click here for diff

This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them).  
  
Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation.  
  
Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left.  
  
Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc.  
  
A few function prototypes were found that not longer had an implementation. These were removed, but there may be more.  
  
The coding document has been updated to reflect this policy, which is not new but has never been documented.  

M CODING.md
M doc/xml/coding.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/file.c
M src/command/archive/get/file.h
M src/command/archive/get/get.c
M src/command/archive/get/get.h
M src/command/archive/get/protocol.c
M src/command/archive/get/protocol.h
M src/command/archive/push/file.c
M src/command/archive/push/file.h
M src/command/archive/push/protocol.c
M src/command/archive/push/protocol.h
M src/command/archive/push/push.c
M src/command/archive/push/push.h
M src/command/backup/backup.c
M src/command/backup/backup.h
M src/command/backup/common.c
M src/command/backup/common.h
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/pageChecksum.c
M src/command/backup/pageChecksum.h
M src/command/backup/protocol.c
M src/command/backup/protocol.h
M src/command/check/check.c
M src/command/check/check.h
M src/command/check/common.c
M src/command/check/common.h
M src/command/command.c
M src/command/command.h
M src/command/control/common.c
M src/command/control/common.h
M src/command/control/start.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/expire/expire.h
M src/command/help/help.c
M src/command/help/help.h
M src/command/info/info.c
M src/command/info/info.h
M src/command/local/local.c
M src/command/local/local.h
M src/command/remote/remote.c
M src/command/remote/remote.h
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/command/restore/protocol.h
M src/command/restore/restore.c
M src/command/restore/restore.h
M src/command/stanza/common.c
M src/command/stanza/common.h
M src/command/stanza/create.c
M src/command/stanza/create.h
M src/command/stanza/delete.c
M src/command/stanza/delete.h
M src/command/stanza/upgrade.c
M src/command/stanza/upgrade.h
M src/common/compress/gz/common.c
M src/common/compress/gz/common.h
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/common.c
M src/common/compress/lz4/common.h
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/cipherBlock.h
M src/common/crypto/common.c
M src/common/crypto/common.h
M src/common/crypto/hash.c
M src/common/crypto/hash.h
M src/common/debug.c
M src/common/debug.h
M src/common/encode.c
M src/common/encode.h
M src/common/encode/base64.c
M src/common/encode/base64.h
M src/common/error.c
M src/common/error.h
M src/common/exec.c
M src/common/exec.h
M src/common/exit.c
M src/common/exit.h
M src/common/fork.c
M src/common/fork.h
M src/common/ini.c
M src/common/ini.h
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/sink.c
M src/common/io/filter/size.c
M src/common/io/handleRead.c
M src/common/io/handleWrite.c
M src/common/io/handleWrite.h
M src/common/io/http/cache.c
M src/common/io/http/cache.h
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/common.c
M src/common/io/http/common.h
M src/common/io/http/header.c
M src/common/io/http/header.h
M src/common/io/http/query.c
M src/common/io/http/query.h
M src/common/io/io.c
M src/common/io/io.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/tls/client.c
M src/common/io/write.c
M src/common/io/write.h
M src/common/lock.c
M src/common/lock.h
M src/common/log.c
M src/common/log.h
M src/common/regExp.c
M src/common/regExp.h
M src/common/stackTrace.c
M src/common/stackTrace.h
M src/common/time.c
M src/common/time.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/convert.c
M src/common/type/convert.h
M src/common/type/json.c
M src/common/type/json.h
M src/common/type/keyValue.c
M src/common/type/keyValue.h
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/mcv.c
M src/common/type/mcv.h
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/common/type/variantList.c
M src/common/type/variantList.h
M src/common/type/xml.c
M src/common/type/xml.h
M src/common/user.c
M src/common/wait.c
M src/common/wait.h
M src/config/config.c
M src/config/config.h
M src/config/define.c
M src/config/define.h
M src/config/load.c
M src/config/load.h
M src/config/parse.c
M src/config/parse.h
M src/config/protocol.c
M src/config/protocol.h
M src/db/db.c
M src/db/db.h
M src/db/helper.c
M src/db/helper.h
M src/db/protocol.c
M src/db/protocol.h
M src/info/info.c
M src/info/info.h
M src/info/infoArchive.c
M src/info/infoArchive.h
M src/info/infoBackup.c
M src/info/infoBackup.h
M src/info/infoPg.c
M src/info/infoPg.h
M src/info/manifest.c
M src/info/manifest.h
M src/postgres/client.c
M src/postgres/client.h
M src/postgres/interface.c
M src/postgres/interface.h
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.c
M src/protocol/command.h
M src/protocol/helper.c
M src/protocol/helper.h
M src/protocol/parallel.c
M src/protocol/parallel.h
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M src/protocol/server.c
M src/protocol/server.h
M src/storage/cifs/storage.c
M src/storage/helper.c
M src/storage/helper.h
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/read.h
M src/storage/remote/protocol.c
M src/storage/remote/protocol.h
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M src/storage/write.c
M src/storage/write.h
M test/src/module/common/typeObjectTest.c

Add/remove dependent backups during backup.info reconstruct.

commit   : 3fbde30c6fac4c752a3a1f47bbf309571b123b7a    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 3 Apr 2020 13:25:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 13:25:38 -0400    

Click here for diff

Prior to performing a backup or expiring backups, the backup.info file is validated by reconstructing it from the backups in the repository. When a backup had already been removed from the repo, it was removed from the backup.info file but its dependents were not.  
  
Now, the dependent backups will also be removed from backup.info and only backups in the repo that have their full dependency chain will be added to backup.info if they are missing.  

M src/info/infoBackup.c
M test/src/module/info/infoBackupTest.c

Remove storageRead() and storageWriteDriver().

commit   : f3ae74b0d62a7b85e1452d46dd60aafbeb88226d    
  
author   : David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 08:38:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 08:38:28 -0400    

Click here for diff

These functions were only being used in the tests. This usage likely dates to before the include directive was available in define.yaml.  

M src/storage/read.c
M src/storage/read.intern.h
M src/storage/write.c
M src/storage/write.intern.h
M test/define.yaml
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Update RHEL package location.

commit   : 1214f1d70b5a34bd648892519777c6bd3b33619d    
  
author   : David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 08:18:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 3 Apr 2020 08:18:07 -0400    

Click here for diff

This changed upstream so update the file paths.  

M test/test.pl

Clean up const usage in bufPtr() and bufRemainsPtr().

commit   : 713211d89fd341e5b39b5273c4cb0cea495c245d    
  
author   : David Steele <[email protected]>    
date     : Thu, 2 Apr 2020 17:25:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 2 Apr 2020 17:25:49 -0400    

Click here for diff

These functions accepted const Buffer objects and returned non-const pointers which is definitely not a good idea. Add bufPtrConst() to handle cases where only a const return value is needed and update call sites.  
  
Use UNCONSTIFY() in cases where library code out of our control requires a non-const pointer. This includes the already-documented exception in command/backup/pageChecksum and input buffers in the gzCompress and gzDecompress filters.  

M src/command/backup/pageChecksum.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/io/handleWrite.c
M src/common/io/tls/client.c
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/string.c
M src/common/type/xml.c
M src/postgres/interface.c
M src/storage/posix/write.c
M test/src/module/common/ioTest.c
M test/src/module/storage/posixTest.c

Add UNCONSTIFY() macro.

commit   : 76b88a3cd58cb69ba8660247ed0959707c12401c    
  
author   : David Steele <[email protected]>    
date     : Thu, 2 Apr 2020 16:58:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 2 Apr 2020 16:58:38 -0400    

Click here for diff

Allows casting const-ness away from an expression, but doesn't allow changing the type. Enforcement of the latter currently only works for gcc-like compilers.  
  
Note that it is not safe to cast const-ness away if the result will ever be modified (it would be undefined behavior). Doing so can cause compiler mis-optimizations or runtime crashes (by modifying read-only memory). It is only safe to use when the result will not be modified, but API design or language restrictions prevent you from declaring that (e.g. because a function returns both const and non-const variables).  
  
Note that this only works in function scope, not for global variables (it would be nice, but not trivial, to improve that).  
  
UNCONSTIFY() requires static assert which is a feature in its own right.  

M src/build.auto.h.in
M src/build/configure.ac
M src/common/macro.h
M src/configure
M test/lib/pgBackRestTest/Common/JobTest.pm

Remove unused getters in common/io/write module.

commit   : 78beb16d6f30b2de561b391579c30b2811a94cd3    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 19:33:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 19:33:11 -0400    

Click here for diff

These were probably added for symmetry with IoRead but we would prefer to remove those getters.  
  
So, just remove the equivalents in IoWrite.  

M src/common/io/write.c
M src/common/io/write.intern.h
M test/src/module/common/ioTest.c

Remove unused exports from pgBackRestBuild::Config::Data.

commit   : f19c59ab38452bd4d247aa3a073c5c525b3b3cbe    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 18:03:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 18:03:16 -0400    

Click here for diff

These were once used heavily in the Perl code but now only a few of them are used in the documentation.  
  
Remove the unused exports as there is generally no need create them for new commands/options.  

M build/lib/pgBackRestBuild/Config/Data.pm

Enable FD_CLOEXEC.

commit   : 3aedcd17760e677058e679ffaa7500ce57e42805    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 17:20:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 17:20:47 -0400    

Click here for diff

PostgreSQL enables this option when available which seems like a good idea since we also do not share connections between processes.  
  
Note that as in PostgreSQL there is no way to disable this option.  

M src/common/io/socket/common.c
M test/src/module/common/ioTlsTest.c

Enable TCP_NODELAY.

commit   : 967f2c0d7f56702059f433c248c1d2640dc26fde    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 16:56:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 16:56:15 -0400    

Click here for diff

PostgreSQL enables this option when available which seems like a good idea since we also buffer transmissions.  
  
Note that as in PostgreSQL there is no way to disable this option.  

M src/common/io/socket/common.c
M test/src/module/common/ioTlsTest.c

Fix typo.

commit   : a1a0a23c6a086bb9f95ebdd280cca4c4f7ce683d    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 16:51:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 16:51:29 -0400    

Click here for diff

M test/src/module/common/ioTlsTest.c

Rename tcp-keep-alive option to sck-keep-alive.

commit   : 789e364e6b35260a5805f1643761cd7f24b2bfdc    
  
author   : David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 15:44:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 1 Apr 2020 15:44:51 -0400    

Click here for diff

This is really a socket option so the new name is clearer.  
  
Since common/io/socket/tcp will contains a mix of options it makes sense to rename it to socket and cascade name changes as needed.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M src/Makefile.in
M src/common/io/socket/client.c
R062 src/common/io/socket/tcp.c src/common/io/socket/common.c
R063 src/common/io/socket/tcp.h src/common/io/socket/common.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M test/define.yaml
M test/src/module/command/helpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/config/loadTest.c
M test/src/test.c

TCP keep-alive options are configurable.

commit   : 5c6fb88bef242c071fd02671798b296bd65fa47f    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 Mar 2020 18:13:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 Mar 2020 18:13:11 -0400    

Click here for diff

Prior to 2.25 the individual TCP keep-alive options were not being configured due to a missing header. In 2.25 they were being configured incorrectly due to a disconnect between the timeout specified in ms and what was expected by the TCP options, i.e. seconds.  
  
Instead make the TCP keep-alive options directly configurable, with correct units and better testing. Keep-alive is enabled by default (though it can be defaulted to the system setting instead) and the rest of the options are not set by default. This is in line with what PostgreSQL does, though PostgreSQL does not allow keep-alive to be defaulted.  
  
Also move configuration of TCP options before connect() as PostgreSQL does.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M src/Makefile.in
M src/common/io/socket/client.c
A src/common/io/socket/tcp.c
A src/common/io/socket/tcp.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M test/define.yaml
M test/src/module/command/helpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/config/loadTest.c
M test/src/test.c

Add SocketClient object.

commit   : 8989118cc656eeb3458404a4a9beff059c0c984c    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 Mar 2020 12:43:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 Mar 2020 12:43:29 -0400    

Click here for diff

This functionality was embedded into TlsClient but that was starting to get unwieldy.  
  
Add SocketClient to contain all socket-related client functionality.  

M src/Makefile.in
M src/command/command.c
M src/common/io/http/client.c
A src/common/io/socket/client.c
A src/common/io/socket/client.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M test/define.yaml
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/src/module/command/commandTest.c
M test/src/module/common/ioTlsTest.c

Add OBJECT_DEFINE_GET() macro to define object getters.

commit   : abb76398079fdfb7842bbd1f83fe46e33a5121b5    
  
author   : David Steele <[email protected]>    
date     : Tue, 31 Mar 2020 11:10:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 31 Mar 2020 11:10:26 -0400    

Click here for diff

Simple object getters involve a lot of boilerplate code so create a macro to simplify implementing them.  

M src/common/type/object.h

Move common/object.h to common/type/object.h.

commit   : da43db354343d629029fd78ceefecaa57bf58c73    
  
author   : David Steele <[email protected]>    
date     : Mon, 30 Mar 2020 20:52:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 30 Mar 2020 20:52:57 -0400    

Click here for diff

This header does not contain a type but is used to define types so this seems like a better location.  

M src/command/backup/pageChecksum.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/exec.c
M src/common/ini.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.c
M src/common/io/filter/group.c
M src/common/io/filter/sink.c
M src/common/io/filter/size.c
M src/common/io/handleRead.c
M src/common/io/handleWrite.c
M src/common/io/http/cache.c
M src/common/io/http/client.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/read.c
M src/common/io/tls/client.c
M src/common/io/write.c
M src/common/macro.h
M src/common/regExp.c
M src/common/type/buffer.c
M src/common/type/keyValue.c
M src/common/type/list.c
M src/common/type/mcv.c
R099 src/common/object.h src/common/type/object.h
M src/common/type/xml.c
M src/common/wait.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M src/postgres/client.c
M src/protocol/client.c
M src/protocol/command.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/server.c
M src/storage/posix/read.c
M src/storage/posix/storage.intern.h
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/storage.c
M src/storage/write.c
M test/code-count/file-type.yaml
M test/define.yaml
R100 test/src/module/common/objectTest.c test/src/module/common/typeObjectTest.c
M test/src/module/performance/storageTest.c

Add storage filter performance test.

commit   : a29e25a845bed5d3b635766dd3ea7ca6a793ac47    
  
author   : David Steele <[email protected]>    
date     : Sun, 29 Mar 2020 21:25:48 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 29 Mar 2020 21:25:48 -0400    

Click here for diff

This test allows the important storage filters to be benchmarked by MiB/s.  

M test/define.yaml
M test/src/module/performance/storageTest.c

Remove Debian package patch now that it has been merged upstream.

commit   : 1e0b0c9344c3cdb7e11ff67820a681bba1c1c05d    
  
author   : David Steele <[email protected]>    
date     : Sun, 29 Mar 2020 09:17:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 29 Mar 2020 09:17:39 -0400    

Click here for diff

D test/patch/debian-package.patch

Add time since last test started to test output.

commit   : bf7b989103d6804362b4fb0994b96f13f5cc0dd8    
  
author   : David Steele <[email protected]>    
date     : Sat, 28 Mar 2020 18:25:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 28 Mar 2020 18:25:51 -0400    

Click here for diff

This makes it easier to see the timing of each step in the test.  

M test/src/common/harnessTest.c

Add TEST_TITLE_FMT() macro.

commit   : 7e519e72d516dc03f60c25252928a24de8cc6e58    
  
author   : David Steele <[email protected]>    
date     : Sat, 28 Mar 2020 18:20:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 28 Mar 2020 18:20:29 -0400    

Click here for diff

M test/src/common/harnessTest.h

Fix typos.

commit   : e1c72f6f97e9020874e1f31c97d1e1ff7269104d    
  
author   : Adrian Vondendriesch <[email protected]>    
date     : Sat, 28 Mar 2020 17:48:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 28 Mar 2020 17:48:57 -0400    

Click here for diff

M doc/xml/reference.xml
M src/command/restore/restore.c
M src/config/define.auto.c
M test/src/common/harnessStorage.h
M test/src/module/command/restoreTest.c

Restore prefix/exec_prefix Makefile variables removed in 237a3da4.

commit   : af734d69f4fc32502ae9b5baa7ba2ee244d0462c    
  
author   : Marc Cousin <[email protected]>    
date     : Fri, 27 Mar 2020 08:22:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 27 Mar 2020 08:22:58 -0400    

Click here for diff

M src/Makefile.in

Update packages required for each CI job.

commit   : 4b1d995bbacac8baebd8a71045d87a2126ad851b    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 22:20:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 22:20:09 -0400    

Click here for diff

Documentation builds and tests have only a few packages in common so rearrange packages to save some time and clarify dependencies.  
  
Remove the libperl-dev package which became obsolete when the LibC module was removed in 79cfd3ae.  
  
Add a few comments for good measure.  

M test/travis.pl

Rearrange Travis CI jobs based on current runtimes.

commit   : 65285ff5b28f81a5e223337f93fe127d2eed2872    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 22:00:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 22:00:40 -0400    

Click here for diff

The runtimes of the tests have change with recent updates and it is generally best to order the slowest tests first.  

M .travis.yml

Add comments to make the purpose of each CI job clearer.

commit   : 0f742204e84c1a9529085fcfad61318e1c1a1842    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 21:39:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 21:39:40 -0400    

Click here for diff

M .travis.yml

Updates for key changes in travis.yaml.

commit   : 09d26d54ceb42a1f14de4ed59541f602b7c99ee6    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 21:16:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 21:16:21 -0400    

Click here for diff

The format has evolved over time and the old keys were showing warnings during builds.  

M .travis.yml

Add performance/storage test.

commit   : 3d255dce3c83fe0de49ab5a5ff3d71151e0761b3    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 21:05:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 21:05:36 -0400    

Click here for diff

The primary purpose of this test (currently) is to measure the performance of storageRemoteInfoList(), which is critical for building a manifest when the PostgreSQL host is remote.  
  
The starting baseline of 1 million files is perhaps a bit aggressive but it seems very likely to blow up if there are performance regressions.  

M test/define.yaml
A test/src/module/performance/storageTest.c

Increase baseline of the performance/type test.

commit   : b64755d6352c9e3e4ac2508b47e2a223f755afe5    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 20:52:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 20:52:05 -0400    

Click here for diff

Recent performance improvements allow increasing the baseline of this test.  
  
In general it is best if the baseline is large enough to cause the test to blow up if there are performance regressions.  

M test/src/module/performance/typeTest.c

Add --no-performance to test.pl to suppress performance tests.

commit   : 50cf7370ee171053225a5d4511fd186da2c4c61e    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 20:36:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 20:36:09 -0400    

Click here for diff

Performance tests do not need to be run on all platforms. Using vm=none to run performance tests seems best ... for performance.  

M .travis.yml
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/test.pl

Remove RHEL package patch now that it has been merged upstream.

commit   : 24e03e1320fcc19059599c136de293183cb77f43    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 20:25:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 20:25:42 -0400    

Click here for diff

D test/patch/rhel-package.patch

Begin v2.26 development.

commit   : fa8642237146bc1b79fd57271219e0e9aa07882e    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 17:20:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 17:20:58 -0400    

Click here for diff

M doc/RELEASE.md
M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h