pgBackRest 2.30 commit log

by Yogesh Sharma
pgBackRest
CommitLog

v2.29: Auto S3 Credentials on AWS

commit   : 1df7d3eee5baa38504478fafa27de2b1b0fce724    
  
author   : David Steele <[email protected]>    
date     : Mon, 31 Aug 2020 07:28:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 31 Aug 2020 07:28:22 -0400

Click here for diff

Bug Fixes:  
  
* Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. (Reviewed by Cynthia Shang. Reported by argdenis.)  
* Fix issue with = character in file or database names. (Reviewed by Bastian Wegge, Cynthia Shang. Reported by Brad Nicholson, Bastian Wegge.)  
  
Features:  
  
* Automatically retrieve temporary S3 credentials on AWS instances. (Contributed by David Steele, Stephen Frost. Reviewed by Cynthia Shang, David Youatt, Aleš Zelený, Jeanette Bromage.)  
* Add archive-mode option to disable archiving on restore. (Reviewed by Stephen Frost. Suggested by Stephen Frost.)  
  
Improvements:  
  
* PostgreSQL 13 beta3 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.  
* Asynchronous list/remove for S3/Azure storage. (Reviewed by Cynthia Shang, Stephen Frost.)  
* Improve memory usage of unlogged relation detection in manifest build. (Reviewed by Cynthia Shang, Stephen Frost, Brad Nicholson, Oscar. Suggested by Oscar, Brad Nicholson.)  
* Proactively close file descriptors after forking async process. (Reviewed by Stephen Frost, Cynthia Shang.)  
* Delay backup remote connection close until after archive check. (Contributed by Floris van Nee. Reviewed by David Steele.)  
* Improve detailed error output. (Reviewed by Cynthia Shang.)  
* Improve TLS error reporting. (Reviewed by Cynthia Shang, Stephen Frost.)  
  
Documentation Bug Fixes:  
  
* Add none to compress-type option reference and fix example. (Reported by Ugo Bellavance, Don Seiler.)  
* Add missing azure type in repo-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)  
* Fix typo in repo-cipher-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)  
  
Documentation Improvements:  
  
* Clarify that expire must be run regularly when expire-auto is disabled. (Reviewed by Douglas J Hunley. Suggested by Douglas J Hunley.)

M CONTRIBUTING.md
M README.md
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Add none to compress-type option reference and fix example.

commit   : 14e1fd10ca74ff9c6cdf32c66d374efb09067919    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Aug 2020 10:59:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Aug 2020 10:59:04 -0400

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/define.auto.c

Add archive-mode option to disable archiving on restore.

commit   : 8c2960fab39e3eda96f791d8a7b1f88f8dedd4f9    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 Aug 2020 15:05:41 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 25 Aug 2020 15:05:41 -0400

Click here for diff

When restoring a cluster that will be promoted but is not intended to be the new primary, it is important to disable archiving to avoid polluting the repository with useless WAL. This option makes disabling archiving a bit easier.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/restore/restore.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M test/src/module/command/helpTest.c
M test/src/module/command/restoreTest.c

Fix typo.

commit   : 1812725c8e0cdd7335efdd9909c204e11c01c200    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 Aug 2020 12:50:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 Aug 2020 12:50:06 -0400

Click here for diff

M src/command/restore/restore.c

Automatically retrieve temporary S3 credentials on AWS instances.

commit   : 851f2e814e1d733989c0330e7efef3def2cace80    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 Aug 2020 10:38:49 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 25 Aug 2020 10:38:49 -0400

Click here for diff

Automatically retrieve the role and temporary credentials for S3 when the AWS instance is associated with an IAM role. Credentials are automatically updated when they are <= 5 minutes from expiring.
  

  
Basic configuration is to set repo1-s3-key-type=auto. repo1-s3-role can be used to set a specific role, otherwise it will be retrieved automatically.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/storage/helper.c
M src/storage/s3/storage.c
M src/storage/s3/storage.h
M test/src/module/command/helpTest.c
M test/src/module/storage/s3Test.c

Improve detailed error output.

commit   : 1d2b8ae7bb1ff0621229514acff587793276a123    
  
author   : David Steele <[email protected]>    
date     : Mon, 24 Aug 2020 14:51:08 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 24 Aug 2020 14:51:08 -0400

Click here for diff

Add more info (command, version, options) to asserts, and errors when debug logging is enabled. This won't cover all cases but might mean we get more info in some circumstances.

M doc/xml/release.xml
M src/common/exit.c
M test/src/module/common/exitTest.c

Consolidate variables into structs in common/stackTrace module.

commit   : 7fba1f0bbef7072964eb4ad00275fbff00f94d50    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Aug 2020 16:12:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 Aug 2020 16:12:44 -0400

Click here for diff

This makes it easier to determine which variables are used in which parts of the module and reduces the chance of name conflicts while unit testing.

M src/common/stackTrace.c
M test/src/module/common/stackTraceTest.c

Move backTraceState variable inside #ifdef WITH_BACKTRACE.

commit   : 2a73bd6cbf89135d652d5cff1c63095858068a6d    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Aug 2020 11:45:54 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 Aug 2020 11:45:54 -0400

Click here for diff

This variable is only used when WITH_BACKTRACE is defined so there's no need to declare it otherwise.

M src/common/stackTrace.c

Remove invalid call to stackTraceTestFileLineSet().

commit   : 0e8e5881d8859945760aa873b53a4b39ac26dfc3    
  
author   : David Steele <[email protected]>    
date     : Fri, 21 Aug 2020 11:43:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 21 Aug 2020 11:43:26 -0400

Click here for diff

When testing the common/stack-trace module it is important not to call this test function since the trace stack is empty and it will cause a buffer under run.  
  
Instead use a macro that is only defined under the correct circumstances and add an assert() to catch future regressions.

M src/common/stackTrace.c
M test/src/common/harnessDebug.h
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h

Rearrange members in StackTraceData struct to save memory.

commit   : 25ee913932a9de1ca2a57d0b7c37309504af8a1f    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 15:53:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 15:53:01 -0400

Click here for diff

This arrangement is more efficient on 64-bit systems due to alignment. There is no difference on 32-bit systems.

M src/common/stackTrace.c
M test/src/module/common/stackTraceTest.c

Remove unused parameter from cmdBegin().

commit   : 859b8a50fdacad0ffa92210e0e8a3c98a2eba1b7    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 14:16:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 14:16:36 -0400

Click here for diff

M src/command/command.c
M src/command/command.h
M src/config/load.c
M src/main.c
M test/src/module/command/commandTest.c

Refactor option logging into a general-purpose function.

commit   : fccca0d716de2be92ddd5c7d8f532dcad41fd828    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 14:11:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 14:11:40 -0400

Click here for diff

M src/command/command.c
M src/command/command.h
M test/src/module/command/commandTest.c

Add general-purpose statistics collector.

commit   : 959f77cd6a3be9dffa049d9b25901c72a7c947bb    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 14:04:26 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 20 Aug 2020 14:04:26 -0400

Click here for diff

Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome.
  

  
Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually.
  

  
No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible.
  

  
The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.

M doc/xml/release.xml
M src/Makefile.in
M src/command/command.c
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/request.c
M src/common/io/http/response.c
M src/common/io/socket/client.c
M src/common/io/socket/client.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
A src/common/stat.c
A src/common/stat.h
M src/main.c
M test/define.yaml
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/src/module/command/commandTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
A test/src/module/common/statTest.c
M test/src/module/performance/typeTest.c
M test/src/test.c

Asynchronous list/remove for S3/Azure storage.

commit   : 53f8e7a9cf842745282cde66b99268b500073776    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 12:12:21 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 20 Aug 2020 12:12:21 -0400

Click here for diff

Improve the performance of list/delete operations by using async requests.
  

  
It's questionable whether this will have any impact on Azure deletes since they are sent one at a time with little work done in between, but it doesn't hurt to try.

M doc/xml/release.xml
M src/storage/azure/storage.c
M src/storage/s3/storage.c

Disable s390x builds on Travis-CI (again).

commit   : 9876d7096d35496226fdbab788af9499f1b085d3    
  
author   : David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 11:24:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 20 Aug 2020 11:24:27 -0400

Click here for diff

s390x currently is too unreliable to be part of required CI builds. They can sometimes take hours or days to get queued which makes getting commits through our workflow very time-consuming.

M .travis.yml

Add user-agent to HTTP requests.

commit   : de0f8c2654ce5a924e3b93959a26c39930f02b60    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 Aug 2020 10:01:24 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 18 Aug 2020 10:01:24 -0400

Click here for diff

M doc/xml/release.xml
M src/common/io/http/request.c
M test/src/module/common/ioHttpTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Enable s390x builds on Travis-CI.

commit   : 4391497a05b0d645c5ed472f0b139ce545ece5e9    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 Aug 2020 08:17:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Aug 2020 08:17:13 -0400

Click here for diff

These were disabled in 6fc25c7a because s390x builds were not being queued. It looks like they are working again.

M .travis.yml

Update *((Type *)lstGet()) pattern to *(Type *)lstGet().

commit   : d935af5e598e27f0bff58f34f60a81d432d0491e    
  
author   : David Steele <[email protected]>    
date     : Tue, 18 Aug 2020 07:26:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 18 Aug 2020 07:26:02 -0400

Click here for diff

This pattern has extra parens and was only used in a few places so replace it with the general *(Type *)lstGet() pattern.

M src/info/infoBackup.c
M src/info/infoPg.c
M test/src/module/common/typeListTest.c

Add support for HTTP/1.0.

commit   : fbee6ec170a5f4629fa0d40494cc89c40cfa934c    
  
author   : David Steele <[email protected]>    
date     : Fri, 14 Aug 2020 13:11:33 -0400    
  
committer: GitHub <[email protected]>    
date     : Fri, 14 Aug 2020 13:11:33 -0400

Click here for diff

HTTP/1.0 connections are closed by default after a single response. Other than that, treat 1.0 the same as 1.1.
  

  
HTTP/1.0 allows different date formats that we can't parse but for now, at least, we don't need any date headers from 1.0 requests.

M doc/xml/release.xml
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/common/io/http/response.c
M test/src/module/common/ioHttpTest.c

PostgreSQL 13 beta3 support.

commit   : 6bb111c1700d34bed23ca65518d4b10970aa8554    
  
author   : David Steele <[email protected]>    
date     : Fri, 14 Aug 2020 10:08:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 14 Aug 2020 10:08:25 -0400

Click here for diff

M doc/xml/release.xml
M src/postgres/interface/version.vendor.h
M test/container.yaml

Add missing azure type in repo-type option reference.

commit   : afcc4d193dea3dfa13b56035e49fbc55a259e936    
  
author   : Don Seiler <[email protected]>    
date     : Tue, 11 Aug 2020 13:38:38 -0500    
  
committer: GitHub <[email protected]>    
date     : Tue, 11 Aug 2020 13:38:38 -0500

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/define.auto.c

Fix typo in repo-cipher-type option reference.

commit   : f40c7b65fa4e9b10eaad687300007a4510766a4b    
  
author   : Don Seiler <[email protected]>    
date     : Tue, 11 Aug 2020 10:41:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 11 Aug 2020 10:41:06 -0400

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/define.auto.c

Improve server test harness to support multiple protocols/connections.

commit   : 65858d12dd67f3bf6983ad5c8521edb168da004b    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Aug 2020 16:16:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 10 Aug 2020 16:16:11 -0400

Click here for diff

The prior implementation only supported a single connection on TLS. This is not flexible enough for complex testing scenarios which might require multiple simultaneous connections on different protocols.  
  
Allow multiple simultaneous connections and add plain sockets as a protocol option. Rename the functions used for server scripting to hrnServerScript*() to make it clear they are related. Improve error messages when less input is received by the server than expected.  
  
Also, do a bit of cleanup and add more comments.

M test/code-count/file-type.yaml
R052 test/src/common/harnessTls.c test/src/common/harnessServer.c
A test/src/common/harnessServer.h
D test/src/common/harnessTls.h
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Implement IoClient/IoSession interfaces for SocketClient/SocketSession.

commit   : 7fdbd94e39ff6a97d00f1aa4a40c02c0b7b2ef4c    
  
author   : David Steele <[email protected]>    
date     : Mon, 10 Aug 2020 16:03:38 -0400    
  
committer: GitHub <[email protected]>    
date     : Mon, 10 Aug 2020 16:03:38 -0400

Click here for diff

Following up on 111d33c, implement the new interfaces for socket client/session. Now HTTP objects can be used over TLS or plain sockets.
  

  
This required adding ioSessionFd() and ioSessionRole() to provide the functionality of sckSessionFd() and sckSessionType(). sckClientHost() and sckClientPort don't make sense in a generic interface so they were replaced with ioSessionName().

M doc/xml/release.xml
M src/command/command.c
M src/common/io/client.c
M src/common/io/client.h
M src/common/io/client.intern.h
M src/common/io/session.c
M src/common/io/session.h
M src/common/io/session.intern.h
M src/common/io/socket/client.c
M src/common/io/socket/client.h
M src/common/io/socket/session.c
M src/common/io/socket/session.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M src/common/io/tls/session.c
M src/common/io/tls/session.h
M src/storage/azure/storage.c
M src/storage/s3/storage.c
M test/src/common/harnessTls.c
M test/src/module/command/commandTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/s3Test.c

Delay backup remote connection close until after archive check.

commit   : 54c3c39645b84867c3e4bc04c6ad07a2da7f5f2d    
  
author   : Floris van Nee <[email protected]>    
date     : Mon, 10 Aug 2020 17:35:09 +0200    
  
committer: GitHub <[email protected]>    
date     : Mon, 10 Aug 2020 17:35:09 +0200

Click here for diff

Only close the remote connection after verifying that the WAL files have been received. This is necessary if the archive_command on the PostgreSQL host is conditional, i.e. archiving only happens while a backup lock is held, to ensure all WAL segments are archived.

M doc/xml/release.xml
M src/command/backup/backup.c

Remove deleteMax parameter from storageS3New().

commit   : 63ad4786af704cf20670f67271444c04a3cc0e32    
  
author   : David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 16:06:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 16:06:05 -0400

Click here for diff

This was included primarily for testing but now that storageS3New() is not called directly in the tests the value needs to be modified after configuring storage anyway.  
  
Since there is no need to make this parameter user-configurable just remove it.

M src/storage/helper.c
M src/storage/s3/storage.c
M src/storage/s3/storage.h

Simplify S3 configuration tests and add security token tests.

commit   : 8b34f854f3b88fb8c361d5b9d559f8b00295b894    
  
author   : David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 15:52:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 15:52:33 -0400

Click here for diff

Rather than calling storageS3New() directly, create the storage by loading a configuration and calling repoStorageGet(). This is a better end-to-end test and cuts down on a lot of redundant tests.  
  
Add tests that include security tokens in error messages to ensure they are redacted.

M test/define.yaml
M test/src/module/storage/s3Test.c

Add debug log function for HttpClient.

commit   : 38f66d4bd390ee0994559951b09298d700d02533    
  
author   : David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 14:32:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 14:32:21 -0400

Click here for diff

M src/common/io/http/client.c
M src/common/io/http/client.h

Redact x-amz-security-token header in errors.

commit   : 426e6933daec2ee71619bb14e7082983049de9d5    
  
author   : David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 14:18:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 14:18:03 -0400

Click here for diff

This header should not be displayed to the user in error output, even if it is useless by itself.

M src/storage/s3/storage.c

Move file descriptor read/write ready into IoRead/IoWrite.

commit   : 4d22d6eeca23aef038b85dc1c4c3a69fb79f1942    
  
author   : David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 11:23:37 -0400    
  
committer: GitHub <[email protected]>    
date     : Sat, 8 Aug 2020 11:23:37 -0400

Click here for diff

Move sckSessionReadyRead()/Write() into the IoRead/IoWrite interfaces. This is a more logical place for them and the alternative would be to add them to the IoSession interface, which does not seem like a good idea.
  

  
This is mostly a refactor, but a big change is the select() logic in fdRead.c has been replaced by ioReadReady(). This was duplicated code that was being used by our protocol but not TLS. Since we have not had any problems with requiring poll() in the field this seems like a good time to remove our dependence on select().
  

  
Also, IoFdWrite now requires a timeout so update where required, mostly in the tests.

M doc/xml/release.xml
M src/Makefile.in
M src/command/local/local.c
M src/command/remote/remote.c
M src/command/repo/get.c
M src/command/repo/ls.c
M src/common/exec.c
A src/common/io/fd.c
A src/common/io/fd.h
M src/common/io/fdRead.c
M src/common/io/fdWrite.c
M src/common/io/fdWrite.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/socket/common.c
M src/common/io/socket/common.h
M src/common/io/socket/session.c
M src/common/io/socket/session.h
M src/common/io/tls/session.c
M src/common/io/write.c
M src/common/io/write.h
M src/common/io/write.intern.h
M test/define.yaml
M test/src/module/command/archivePushTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/common/execTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/s3Test.c

Add IoClient and IoSession interfaces.

commit   : 111d33c12324a27e68cc3a1349f0566a319eef42    
  
author   : David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 10:39:39 -0400    
  
committer: GitHub <[email protected]>    
date     : Sat, 8 Aug 2020 10:39:39 -0400

Click here for diff

These interfaces allow the HttpClient and HttpSession objects to work with protocols other than TLS, .e.g. plain sockets. This is necessary to allow standard HTTP -- right now only HTTPS is allowed, i.e. HTTP over TLS.
  

  
For now only TlsClient and TlsSession have been converted to the new interfaces. SocketClient and SocketSession will also need to be converted but first sckSessionReadyRead() and sckSessionReadyWrite() need to be moved into the IoRead and IoWrite interfaces, since they are not a good fit for IoSession.

M doc/xml/release.xml
M src/Makefile.in
A src/common/io/client.c
A src/common/io/client.h
A src/common/io/client.intern.h
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/response.c
M src/common/io/http/session.c
M src/common/io/http/session.h
A src/common/io/session.c
A src/common/io/session.h
A src/common/io/session.intern.h
M src/common/io/tls/client.c
M src/common/io/tls/client.h
M src/common/io/tls/session.c
M src/common/io/tls/session.h
D src/common/io/tls/session.intern.h
M src/storage/azure/storage.c
M src/storage/s3/storage.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/common/harnessTls.c
M test/src/module/command/commandTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c

Improve error message in HARNESS_FORK_END().

commit   : 9b7fd1a894ad9fb9cb443458e2296337750b963f    
  
author   : David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 10:02:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 8 Aug 2020 10:02:04 -0400

Click here for diff

Also include expected exit status to make debugging easier.

M test/src/common/harnessFork.h

Clear the callback before calling storageReadPosixFreeResource().

commit   : 0b349c0f2f54d1e45beaa604dba40c2cf9efc15e    
  
author   : David Steele <[email protected]>    
date     : Fri, 7 Aug 2020 13:44:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 7 Aug 2020 13:44:01 -0400

Click here for diff

If storageReadPosixFreeResource() errors then it will be called again when the object is freed, which is not ideal since it might error in a different way and lose the original error.

M src/storage/posix/read.c

Rename handle to fd.

commit   : cde2c756ea8ae32663e0a817098fc9f5d4f2a1f2    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 18:25:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 18:25:07 -0400

Click here for diff

Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd.  
  
There were also plenty of places fd was used so now all uses are consistent.  
  
The Perl code was not updated but might be in a future commit.

M src/Makefile.in
M src/command/control/stop.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/local/local.c
M src/command/local/local.h
M src/command/remote/remote.c
M src/command/remote/remote.h
M src/command/repo/get.c
M src/command/repo/ls.c
M src/command/repo/put.c
M src/command/restore/file.c
M src/common/exec.c
M src/common/fork.c
R073 src/common/io/handleRead.c src/common/io/fdRead.c
R075 src/common/io/handleRead.h src/common/io/fdRead.h
R064 src/common/io/handleWrite.c src/common/io/fdWrite.c
R074 src/common/io/handleWrite.h src/common/io/fdWrite.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/read.intern.h
M src/common/io/write.c
M src/common/io/write.h
M src/common/io/write.intern.h
M src/common/lock.c
M src/common/log.c
M src/protocol/parallel.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/common/harnessFork.h
M test/src/common/harnessLog.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/common/execTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/logTest.c
M test/src/module/config/protocolTest.c
M test/src/module/db/dbTest.c
M test/src/module/performance/storageTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Fix macro spacing.

commit   : a1d7d50223510eda06d68234a03e3c19f972dc47    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 17:30:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 17:30:38 -0400

Click here for diff

M test/src/common/harnessFork.h

Remove unused port struct member.

commit   : d5edc3fe371e16d5612e841861ae9c10ae69cca7    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 15:24:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 15:24:13 -0400

Click here for diff

This does not appear to have been used in quite some time and the tests are equally useless because they don't prove the correct port was passed to httpClientNew().

M src/storage/s3/storage.c
M test/src/module/storage/s3Test.c

Add memContextFreeing().

commit   : d74c8f4b5813f3b0ce5df246c36469257a549e4d    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 14:07:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 14:07:04 -0400

Click here for diff

This can be used to determine if functions are running in a callback from memContextFree(). In this case, data in child mem contexts is not valid.

M src/common/memContext.c
M src/common/memContext.h
M test/src/module/common/memContextTest.c

Disable s390x builds on Travis-CI.

commit   : 6fc25c7a69a09f105f9fdd41ddb33c700d983a2e    
  
author   : David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 13:38:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 5 Aug 2020 13:38:44 -0400

Click here for diff

s390x has not been building for about a day which is clogging the integration branch.

M .travis.yml

Improve TLS error reporting.

commit   : 847e61ce21710cc0cb7baa9af9b3740d9250b4fa    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Aug 2020 15:15:24 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 4 Aug 2020 15:15:24 -0400

Click here for diff

Before 9f2d647 TLS errors included additional details in at least some cases. After 9f2d647 a connection to an HTTP server threw `TLS error [1]` instead of `unable to negotiate TLS connection: [336031996] unknown protocol`.
  

  
Bring back the detailed messages to make debugging TLS errors easier. Since the error routine is now generic the `unable to negotiate TLS connection context` is not available so the error looks like `TLS error [1:336031996] unknown protocol`.

M doc/xml/release.xml
M src/common/io/tls/session.c
M test/src/module/common/ioTlsTest.c

Proactively close file descriptors after forking async process.

commit   : 94d3a01f7368a547dfecaafce67995e24738be78    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Aug 2020 13:20:01 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 4 Aug 2020 13:20:01 -0400

Click here for diff

PostgreSQL may be using most of the available file descriptors when it executes the the archive-get/archive-push commands (especially archive-get). This can lead to problems depending on how many file descriptors are needed for parallelism in the async process.
  

  
Proactively free file descriptors between 3 and 1023 to help ensure there are enough available for reasonable values of process-max, i.e. <= 300.

M doc/xml/release.xml
M src/command/archive/common.c

Improve memory usage of unlogged relation detection in manifest build.

commit   : e81533bbabb2e4f809596c0a88671b2b6616da01    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Aug 2020 10:16:51 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 4 Aug 2020 10:16:51 -0400

Click here for diff

This loop was using a lot of memory without freeing it at intervals.
  

  
Rewrite to use char arrays when possible to reduce memory that needs to be allocated and freed.

M doc/xml/release.xml
M src/info/manifest.c
M test/lib/pgBackRestTest/Common/JobTest.pm

Use forkSafe() in common/exec module.

commit   : e044eaf55faf102d84389b04ff79a7a00fef0965    
  
author   : David Steele <[email protected]>    
date     : Tue, 4 Aug 2020 09:23:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 4 Aug 2020 09:23:59 -0400

Click here for diff

forkSafe() will throw an exception if the fork fails. This is a pretty unlikely scenario but the function is already there so may as well use it.

M src/common/exec.c

Add functions for Zigzag encoding/decoding.

commit   : 5a9856c2f9aeb6efccada86c229713665100eba0    
  
author   : David Steele <[email protected]>    
date     : Sat, 1 Aug 2020 09:42:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 1 Aug 2020 09:42:03 -0400

Click here for diff

Zigzag encoding places the sign bit in the least significant bit so that -1 is encoded as 1, 1 as 2, etc. This moves as many bits as possible into the low order bits which is good for other types of encoding, e.g. base-128.  
  
See https://en.wikipedia.org/wiki/Variable-length_quantity#Zigzag_encoding.

M src/common/type/convert.h
M test/define.yaml
M test/src/module/common/typeConvertTest.c

Improve comment locations.

commit   : f6d22ccb6fab7d7308d26c11c770362e312e600f    
  
author   : David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 17:33:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 17:33:15 -0400

Click here for diff

M src/storage/remote/protocol.c

Accept NULL filenames in hrnStorageInfoListCallback().

commit   : caef2ba482a887113d08fd926a582ff68786a39f    
  
author   : David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 16:35:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 16:35:58 -0400

Click here for diff

This allows it to be used with StorageInfo objects returned from storageInfo(), which do not contain a filename.

M test/src/common/harnessStorage.c

Add lstGetLast() and lstRemoveLast().

commit   : a8e47c38c6d70c202070fa585d4096592a9961cf    
  
author   : David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 16:27:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 16:27:57 -0400

Click here for diff

When a list is being treated as a stack it is useful to get/remove the last (top) item.

M src/common/type/list.c
M src/common/type/list.h
M test/src/module/common/typeListTest.c

Add file name to make performance/storage test more realistic.

commit   : bfb489a82d7451830abec45e3d8a79f4d8580827    
  
author   : David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 16:18:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 16:18:56 -0400

Click here for diff

Also add timing information.

M test/src/module/performance/storageTest.c

Remove special protocol encoding for StorageType.

commit   : 9f1e7d88da845083a1e45d99c123b1718af9808d    
  
author   : David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 16:13:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 16:13:05 -0400

Click here for diff

It seems like overkill to encode this when other enums (e.g. StorageInfoLevel) are passed as integers.  
  
Instead note that StorageType values should not be changed and remove the special encoding.

M src/storage/info.h
M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M test/src/module/storage/remoteTest.c

Fix missing space.

commit   : afe1568bb1713c3520618bac87ba15e4374274bb    
  
author   : David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 15:55:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 31 Jul 2020 15:55:17 -0400

Click here for diff

M src/common/type/buffer.h

Preserve Variant parsed from JSON in iniLoad().

commit   : ac72e1f193d8d4a9de7d8877528e78e317c135e2    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Jul 2020 10:59:50 -0400    
  
committer: GitHub <[email protected]>    
date     : Thu, 30 Jul 2020 10:59:50 -0400

Click here for diff

The fix for = characters in info files (039d314) added JSON validation but discarded the resulting Variant which means the JSON is being parsed twice. This nearly doubles the time to load a manifest since a lot of complex JSON is involved.
  

  
Time to load a million file manifest:
  
Before 039d314: 7.8s
  
039d314: 15.5s
  
This patch: 7.5s
  

  
To fix this regression return the Variant in the callback so the caller does not have to parse it again. The new code appears slightly more efficient overall, probably because there are fewer operations against Strings.

M doc/xml/release.xml
M src/common/ini.c
M src/common/ini.h
M src/info/info.c
M src/info/info.h
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M test/src/common/harnessInfo.c
M test/src/common/harnessInfo.h
M test/src/module/common/iniTest.c
M test/src/module/performance/typeTest.c

Rename strPtr()/strPtrNull() to strZ()/strZNull().

commit   : 3e9dce0d7615f0f56f3eff754095c57c6d56fd6b    
  
author   : David Steele <[email protected]>    
date     : Thu, 30 Jul 2020 07:49:06 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 30 Jul 2020 07:49:06 -0400

Click here for diff

We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern.  
  
As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.

M CODING.md
M CONTRIBUTING.md
M doc/xml/coding.xml
M doc/xml/contributing.xml
M src/command/archive/common.c
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/archive/get/protocol.c
M src/command/archive/push/file.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/backup/file.c
M src/command/check/check.c
M src/command/check/common.c
M src/command/command.c
M src/command/control/common.c
M src/command/control/start.c
M src/command/control/stop.c
M src/command/expire/expire.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/repo/get.c
M src/command/repo/ls.c
M src/command/restore/file.c
M src/command/restore/protocol.c
M src/command/restore/restore.c
M src/command/stanza/create.c
M src/command/stanza/delete.c
M src/command/stanza/upgrade.c
M src/common/compress/helper.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/common.c
M src/common/crypto/hash.c
M src/common/exec.c
M src/common/ini.c
M src/common/io/filter/filter.c
M src/common/io/handleRead.c
M src/common/io/handleWrite.c
M src/common/io/http/common.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/http/request.c
M src/common/io/http/response.c
M src/common/io/socket/client.c
M src/common/io/socket/common.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/lock.c
M src/common/regExp.c
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/json.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/common/type/variant.c
M src/common/type/variant.h
M src/common/type/xml.c
M src/common/user.c
M src/config/config.c
M src/config/exec.c
M src/config/load.c
M src/config/parse.c
M src/config/protocol.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M src/postgres/client.c
M src/postgres/interface.c
M src/protocol/client.c
M src/protocol/command.c
M src/protocol/helper.c
M src/protocol/parallelJob.c
M src/protocol/server.c
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/helper.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/read.c
M src/storage/remote/protocol.c
M src/storage/remote/storage.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/storage.c
M src/storage/write.c
M test/src/common/harnessInfo.c
M test/src/common/harnessLog.c
M test/src/common/harnessPq.c
M test/src/common/harnessPq.h
M test/src/common/harnessStorage.c
M test/src/common/harnessTest.h
M test/src/common/harnessTls.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/checkTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/expireTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/infoTest.c
M test/src/module/command/repoTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/common/compressTest.c
M test/src/module/common/execTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/logTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/config/configTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/typeTest.c
M test/src/module/postgres/clientTest.c
M test/src/module/storage/azureTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Fix issue with = character in file or database names.

commit   : 039d314438f7881e7c7062d645f7800ee2dd8872    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Jul 2020 14:00:23 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 28 Jul 2020 14:00:23 -0400

Click here for diff

The manifest uses the = character as the key/value separator so = characters in the key cause parsing errors and lead to an error or segfault.
  

  
Since the value must be valid JSON we can keep checking the value on the right side of the = and stop building the key when the value is valid. It's a bit hackish but it does seem to do the job without breaking the manifest format.
  

  
Unsurprisingly this makes parsing about 50% slower but it's still more than fast enough. Parsing 10 million key/values takes about 6.5s for the old code and 10s for the new code. Since the value is used as JSON downstream we can reclaim most of this time by just passing the JSON value rather than making the callback reparse it. We'll save that for another commit, though.

M doc/xml/release.xml
M src/common/ini.c
M test/src/module/common/iniTest.c
M test/src/module/info/manifestTest.c

Suppress errors when closing local/remote processes.

commit   : 63a93db6fd97c6e8e22478ee69b05487fa3db69d    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Jul 2020 12:15:33 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 28 Jul 2020 12:15:33 -0400

Click here for diff

Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened.
  

  
Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.

M doc/xml/release.xml
M src/common/exit.c
M src/protocol/helper.c
M src/protocol/helper.h
M src/protocol/parallel.c
M test/define.yaml
M test/src/module/protocol/protocolTest.c

Add quarterly press release instructions to doc/RELEASE.md.

commit   : d9309f13a4c7b962b401832881a593b92f29c438    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Jul 2020 11:20:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Jul 2020 11:20:07 -0400

Click here for diff

M doc/RELEASE.md

Add zstd to CentOS/RHEL 6 test container.

commit   : a260d4a53b68e40f8ebbdc9129143ac61d0ac88b    
  
author   : David Steele <[email protected]>    
date     : Tue, 28 Jul 2020 08:09:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 28 Jul 2020 08:09:10 -0400

Click here for diff

Zstd is now required by the upstream yum package.

M test/container.yaml
M test/lib/pgBackRestTest/Common/VmTest.pm

Add strBaseZ().

commit   : aa37a21084f75c7af8d8ca047be0376057a86bb6    
  
author   : David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 13:37:39 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 13:37:39 -0400

Click here for diff

Since the base name is at the end of the filename it is possible to return the char pointer without making a copy, which is preferred in some cases.

M src/common/type/string.c
M src/common/type/string.h
M src/config/parse.c

Build new manifest inside a temporary context.

commit   : 8e140ad7e06f7e4abdd2c6fa32018ff463d63a6f    
  
author   : David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 11:39:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 11:39:24 -0400

Click here for diff

There is a fair amount of temporary data that is created during the build and it should not end up as a permanent part of the manifest.

M src/info/manifest.c

Add manifestNewBuild() to performance/type test.

commit   : 8e776cc537117292504bc17a1691481fce9bae3d    
  
author   : David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 11:26:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 11:26:14 -0400

Click here for diff

Use a test storage driver to allow manifestNewBuild() to be run against a test cluster at any scale without having to write files to disk.  
  
Simplify the test by using the output of manifestNewBuild() to feed manifestSave() and manifestNewLoad().  
  
Also add manifest size to the output.

M test/src/module/performance/typeTest.c

Add memContextSize().

commit   : f9d923ca3b2c7b2d3eacb3a3cd4fba347cef701e    
  
author   : David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 11:06:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 11:06:25 -0400

Click here for diff

Calculates the memory used by the context and all child contexts.  
  
This is primarily useful for debugging but it is not conditional on DEBUG because it is useful for profile/performance tests.

M src/common/memContext.c
M src/common/memContext.h
M test/src/module/common/memContextTest.c

Move dummy storage driver to test harness.

commit   : 216a61d93626683acdcd0fbf2e145c8751aa7064    
  
author   : David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 08:44:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 25 Jul 2020 08:44:41 -0400

Click here for diff

The dummy driver is the basis for creating test storage drivers so it makes sense to locate it in the harness where all tests can access it.

M test/src/common/harnessStorage.c
M test/src/common/harnessStorage.h
M test/src/module/performance/storageTest.c

Fix non-JSON ini values in tests.

commit   : cbf5dd19c2e273ce8ec65eab256bde9591fb0760    
  
author   : David Steele <[email protected]>    
date     : Fri, 24 Jul 2020 11:05:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 24 Jul 2020 11:05:09 -0400

Click here for diff

A number of tests used invalid JSON values where an error was expected or the value would be ignored.  
  
Update these tests to use valid JSON values so all values in the file can be validated even if they are not used.

M test/src/module/common/iniTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/typeTest.c

Error in jsonToVar() when input not entirely consumed.

commit   : 9d8f6d9fc2a91f20dfc1d3c531459f37fa642a49    
  
author   : David Steele <[email protected]>    
date     : Fri, 24 Jul 2020 10:47:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 24 Jul 2020 10:47:51 -0400

Click here for diff

Something like 3="string" would return an Int64 variant and ignore the invalid portion after the integer. Other JSON interface functions have this check but it was forgotten here.  
  
There are no current issues because of this but we want to be able to validate arbitrary JSON strings and this function was not working correctly for that usage.

M src/common/type/json.c
M test/src/module/common/typeJsonTest.c

Add storage parameter to pgWalFromFile().

commit   : 78ef442a18d62368f4bd05d946660bdcfb7ea40b    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 21 Jul 2020 16:28:05 -0400    
  
committer: GitHub <[email protected]>    
date     : Tue, 21 Jul 2020 16:28:05 -0400

Click here for diff

M src/command/archive/push/file.c
M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

Clarify that expire must be run regularly when expire-auto is disabled.

commit   : ed8829386103b27053c4a1d9e29f1d767d0238ce    
  
author   : David Steele <[email protected]>    
date     : Tue, 21 Jul 2020 10:57:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Jul 2020 10:57:47 -0400

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml
M src/config/define.auto.c

Rearrange Travis CI jobs based on current runtimes.

commit   : d1ac2f7637866c2044f0d40068dcae2de373aa3b    
  
author   : David Steele <[email protected]>    
date     : Tue, 21 Jul 2020 07:34:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 21 Jul 2020 07:34:02 -0400

Click here for diff

In general it is better to have slower jobs run earlier.

M .travis.yml

Remove unused constructor strLstNewParam().

commit   : 90ffde8881f50ba8b91d9d9e881bf59c6488052e    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 18:09:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 18:09:32 -0400

Click here for diff

This function is not used in the core code so remove it and update the test where it was used.  
  
There may eventually be a need for a strLstNewP() function but it doesn't seem worth the code churn until there is an actual requirement.

M src/common/type/stringList.c
M src/common/type/stringList.h
M test/src/module/command/expireTest.c

Remove lstNew() constructor in favor of lstNewP().

commit   : 1783e0490abe266233e65ba1904670a7adde48bb    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 15:22:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 15:22:33 -0400

Click here for diff

The old constructor was left around to reduce code churn during the migration but it just makes the code harder to read and search.  
  
Remove the old constructor and rename all remaining instances to lstNewP(), which by default has the same semantics.

M src/command/backup/backup.c
M src/command/expire/expire.c
M src/command/restore/restore.c
M src/common/io/filter/group.c
M src/common/io/http/client.c
M src/common/type/keyValue.c
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/mcv.c
M src/common/type/variantList.c
M src/common/type/xml.c
M src/db/protocol.c
M src/info/infoPg.c
M src/info/manifest.c
M src/protocol/parallel.c
M src/protocol/server.c
M test/src/common/harnessLog.c
M test/src/module/common/typeListTest.c
M test/src/module/protocol/protocolTest.c

Small wording change in website index / Github readme.

commit   : a92b1f9776acea52627d59fc734707ec90d78b4e    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 13:44:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 13:44:11 -0400

Click here for diff

M doc/xml/index.xml

Update config.guess and config.sub to latest versions.

commit   : 738900a3a9e9fb7d63fbc5e4e4642977b8b65e2e    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 11:08:18 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 11:08:18 -0400

Click here for diff

Also update the policy in doc/RELEASE.md to get the latest versions at the beginning of the release cycle. The older policy was created when we were getting new versions right before the release.

M doc/RELEASE.md
M src/build/config.guess
M src/build/config.sub

Add support for testing on 64-bit big-endian architectures.

commit   : 3f0b41eb9c1137b0b19138d22d2c26009d477602    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 09:59:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 09:59:16 -0400

Click here for diff

In particular add support for s390x but we hope this will work for other 64-bit big-endian architectures.  
  
Run basic unit tests on Travis CI for 390x.

M .travis.yml
M test/src/common/harnessTest.h
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c
M test/src/module/postgres/interfaceTest.c

Reduce reliance on static checksums in unit tests.

commit   : 55277357b8457e3a215bc00e8e97de523949eba6    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 09:47:43 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 09:47:43 -0400

Click here for diff

Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures.  
  
Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.

M test/src/common/harnessTest.h
M test/src/module/command/archivePushTest.c
M test/src/module/command/backupTest.c

Fix ill-advised cast in CipherBlock.

commit   : f8c1957127c148e36843d77ef0bd26a8c8c78fd1    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 09:22:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 09:22:53 -0400

Click here for diff

Little-endian architectures store the low-order bytes in the lowest memory location so this worked even in the case that size_t and int had different byte representations. Since buffer sizes are constrained there was no chance of the integer becoming negative and causing a problem that way.  
  
On big-endian architectures this cast caused the low-order bytes to get loaded into the high-order bytes resulting in a huge buffer size that immediately triggered an assertion (and without the assertion would have certainly segfaulted).  
  
Instead use a temporary int variable and cast that to size_t after the function call. This is the correct way to do it regardless of architecture.  
  
This issue was detected while testing on the s390x architecture.

M src/common/crypto/cipherBlock.c

Begin v2.29 development.

commit   : 615c41e52564a8629e57b9892fd3361c92ae22f6    
  
author   : David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 09:08:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 20 Jul 2020 09:08:59 -0400

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h