pgBackRest 2.29 (upcoming) commit log

Write HTTP request as a buffer to hide secrets.

commit   : eaa05fdc49e3ee5166655c28105eea58138d4ffa    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 8 Jul 2020 15:07:29 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 8 Jul 2020 15:07:29 -0400    

Click here for diff

The prior method of writing headers as strings could expose secrets in trace level logs.  
  
Instead write the entire request as a buffer to prevent secrets from being logged and also reduce the amount of logging.  

M src/common/io/http/request.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringz.h

Add pgLsnFromWalSegment().

commit   : dd9e14b628db37fe7e87e3ab3a6e70cd105a9637    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 8 Jul 2020 12:25:39 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 8 Jul 2020 12:25:39 -0400    

Click here for diff

Provides the reverse operation for pgLsnToWalSegment().  

M doc/xml/release.xml
M src/postgres/interface.c
M src/postgres/interface.h
M test/src/module/postgres/interfaceTest.c

Remove dead test code that should have been removed in 3f4371d7.

commit   : a27ff7c33578c5e262a20ad126229cb23aed0a09    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 7 Jul 2020 08:24:08 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 7 Jul 2020 08:24:08 -0400    

Click here for diff

M test/src/module/storage/azureTest.c

S3 storage driver cleanup inspired by Azure review.

commit   : 57ddd38c51dfa202eeec720a230f08de47685f5e    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 6 Jul 2020 16:57:12 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 6 Jul 2020 16:57:12 -0400    

Click here for diff

These improvements were suggested during the review of 3f4371d7 and it seemed a good idea to apply them to the S3 driver as well.  

M src/storage/s3/storage.c

Fix restore --force acting like --force --delta.

commit   : 682ac656f552a35dba946c303610a69c70a4de42    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Mon, 6 Jul 2020 15:03:24 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Mon, 6 Jul 2020 15:03:24 -0400    

Click here for diff

This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue.

M doc/xml/release.xml
M src/command/restore/restore.c
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/restoreTest.c

Fix typo.

commit   : 8409ab7b2ba798ceaf738e1e38805bc2998feb7b    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Mon, 6 Jul 2020 12:04:35 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 6 Jul 2020 12:04:35 -0400    

Click here for diff

M src/common/io/http/common.c

Add httpUriDecode(), httpQueryNewStr(), and httpQueryMerge().

commit   : cf284fbe8ad8dd9c32ca9b5b39869e32fe04f9ac    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 6 Jul 2020 07:48:12 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 6 Jul 2020 07:48:12 -0400    

Click here for diff

httpUriDecode() reverses the encoding in httpUriEncode().  
  
httpQueryNewStr() creates a new HttpQuery by parsing a query string.  
  
httpQueryMerge() merges the contents of one query into another query.  

M src/common/io/http/common.c
M src/common/io/http/common.h
M src/common/io/http/query.c
M src/common/io/http/query.h
M src/common/type/stringz.h
M test/src/module/common/ioHttpTest.c

Azure support for repository storage.

commit   : 3f4371d7a2407b0ef12404cc2ff2661229bdf283    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 2 Jul 2020 16:24:34 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 2 Jul 2020 16:24:34 -0400    

Click here for diff

Azure and Azure-compatible object stores can now be used for repository storage.  
  
Currently only shared key authentication is supported but SAS will be added soon.  

M README.md
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/resource/fake-cert/README.md
A doc/resource/fake-cert/azure-server.crt
A doc/resource/fake-cert/azure-server.key
A doc/resource/fake-cert/azure.cnf
M doc/xml/index.xml
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/Makefile.in
M src/command/repo/create.c
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
A src/storage/azure/read.c
A src/storage/azure/read.h
A src/storage/azure/storage.c
A src/storage/azure/storage.h
A src/storage/azure/storage.intern.h
A src/storage/azure/write.c
A src/storage/azure/write.h
M src/storage/helper.c
M test/define.yaml
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
A test/lib/pgBackRestTest/Env/Host/HostAzureTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/helpTest.c
A test/src/module/storage/azureTest.c

Fix typo.

commit   : 3e2c8874f7249cc5a70ec8738370f654e280e582    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 1 Jul 2020 07:39:29 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 1 Jul 2020 07:39:29 -0400    

Click here for diff

M src/storage/s3/storage.c

Remove internal commands from HTML and man command references.

commit   : be16bf69a8e66f3eb42bb0789ff210e6d36ddee3    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 29 Jun 2020 15:07:17 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 29 Jun 2020 15:07:17 -0400    

Click here for diff

Some of these commands will be made public in the future but for now their interfaces are not stable so they remain internal.  

M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/xml/release.xml

Remove redundant storage type constants.

commit   : c2dea180fbac39782ed0051205affbb1266642a2    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 16:50:29 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 16:50:29 -0400    

Click here for diff

These constants predate the C storage drivers which now provide their own constants.  

M src/storage/helper.c
M src/storage/helper.h
M test/src/module/storage/cifsTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Add missing end braces in debug log functions.

commit   : 1416dc32251e66992bbf1639079f7137af3f2d99    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 16:44:10 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 16:44:10 -0400    

Click here for diff

These had no effect on functionality but made debug messages harder to read.  

M src/common/io/http/request.c
M src/common/io/http/response.c

PostgreSQL 13 beta2 support.

commit   : 96adf8e513f62d4fbcc2905286127b3558b5aa3d    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 07:44:56 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 07:44:56 -0400    

Click here for diff

There don't appear to be any behavioral changes since PostgreSQL 12 and all the tests pass.  
  
Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.  

M doc/xml/release.xml
M test/container.yaml

Add/update comments in storage/s3 module.

commit   : e53de3ce46497edee69dc4fb65d54185ccbcfd66    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 06:50:20 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 06:50:20 -0400    

Click here for diff

M src/storage/s3/storage.intern.h
M src/storage/s3/write.c
M src/storage/s3/write.h

Minor improvements to storage/s3 unit test.

commit   : 974cc10b90b11d67ba32ba765e1c445cf3a3b3c7    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 06:46:25 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 26 Jun 2020 06:46:25 -0400    

Click here for diff

M test/src/module/storage/s3Test.c

Add review for ea04ec7b.

commit   : e46eeefadae218c2552b17a0303b15b6af90718d    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Fri, 26 Jun 2020 06:34:21 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Fri, 26 Jun 2020 06:34:21 -0400    

Click here for diff

M doc/xml/release.xml

Disable query parallelism in PostgreSQL sessions used for backup control.

commit   : ea04ec7b3f4c6cf25c1b827c25c6a3c5896159a8    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 25 Jun 2020 08:02:48 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 25 Jun 2020 08:02:48 -0400    

Click here for diff

There is no need to have parallelism enabled in a backup control session. In particular, 9.6 marks pg_stop_backup() as parallel-safe but an error will be thrown if pg_stop_backup() is run in a worker.  

M doc/xml/release.xml
M src/db/db.c
M src/postgres/version.h
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/src/common/harnessPq.h
M test/src/module/command/backupTest.c
M test/src/module/db/dbTest.c

Add comment about setting application_name.

commit   : c5a507b9a64b2d775f3269690e925698ce56e9e1    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 24 Jun 2020 19:33:56 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 24 Jun 2020 19:33:56 -0400    

Click here for diff

M src/db/db.c

Replace HRNPQ_MACRO_OPEN_92() test macro with HRNPQ_MACRO_OPEN_GE_92().

commit   : ce98e326e1a9477d2cb1253e00f917e350f96527    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 24 Jun 2020 18:40:19 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 24 Jun 2020 18:40:19 -0400    

Click here for diff

M test/src/common/harnessPq.h
M test/src/module/command/checkTest.c
M test/src/module/command/stanzaTest.c
M test/src/module/db/dbTest.c

Fix versions passed to HRNPQ_MACRO_OPEN_GE_92() test macro.

commit   : f55cb386d4d1ab87404643619f054e1b27d35944    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 24 Jun 2020 18:33:20 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 24 Jun 2020 18:33:20 -0400    

Click here for diff

These were not noticed because currently 9.3 and 9.6 behave the same on open.  

M test/src/module/command/backupTest.c

Asynchronous S3 multipart upload.

commit   : c5892d129169b2d6e62390e7aaa597bc412f528b    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Wed, 24 Jun 2020 13:44:00 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Wed, 24 Jun 2020 13:44:00 -0400    

Click here for diff

When uploading large files the upload is split into multiple parts which are assembled at the end to create the final file. Previously we waited until each part was acknowledged before starting on the processing (i.e. compression, etc.) of the next part.
  

  
Now, the request for each part is sent while processing continues and the response is read just before sending the request for the next part. This asynchronous method allows us to continue processing while the S3 server formulates a response.
  

  
Testing from outside AWS in a high-bandwidth, low-latency environment showed a 35% improvement in the upload time of 1GB files. The time spent waiting for multipart notifications was reduced by ~300% (this measurement included the final part which is not uploaded asynchronously).
  

  
There are still some possible improvements: 1) the creation of the multipart id could be made asynchronous when it looks like the upload will need to be multipart (this may incur cost if the upload turns out not to be multipart). 2) allow more than one async request (this will use more memory).
  

  
A fair amount of refactoring was required to make the HTTP responses asynchronous. This may seem like overkill but having well-defined request, response, and session objects will also be advantageous for the upcoming HTTP server functionality.
  

  
Another advantage is that the lifecycle of an HttpSession is better defined. We only want to reuse sessions that complete the request/response cycle successfully, otherwise we consider the session to be in a bad state and would prefer to start clean with a new one. Previously, this required complex notifications to mark a session as "successfully done". Now, ownership of the session is passed to the request and then the response and only returned to the client after a successful response. If an error occurs anywhere along the way the session will be automatically closed by the object destructor when the request/response object is freed (depending on which one currently owns the session).

M doc/xml/release.xml
M src/Makefile.in
M src/command/repo/create.c
D src/common/io/http/cache.c
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/query.c
M src/common/io/http/query.h
A src/common/io/http/request.c
A src/common/io/http/request.h
A src/common/io/http/response.c
A src/common/io/http/response.h
A src/common/io/http/session.c
R054 src/common/io/http/cache.h src/common/io/http/session.h
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/storage.intern.h
M src/storage/s3/write.c
M test/define.yaml
M test/expect/mock-stanza-002.log
M test/src/common/harnessTest.h
M test/src/module/common/ioHttpTest.c
M test/src/module/storage/s3Test.c

Add strCatZ().

commit   : 45d9b0313632dce934d842e8daf7245a70bc026c    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Wed, 24 Jun 2020 12:09:24 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Wed, 24 Jun 2020 12:09:24 -0400    

Click here for diff

strCat() did not follow our convention of appending Z to functions that accept zero-terminated strings rather than String objects.
  

  
Add strCatZ() to accept zero-terminated strings and update strCat() to accept String objects.
  

  
Use LF_STR where appropriate but don't use other String constants because they do not improve readability.

M src/command/backup/backup.c
M src/command/backup/common.c
M src/command/command.c
M src/command/help/help.c
M src/command/info/info.c
M src/command/restore/restore.c
M src/common/compress/helper.c
M src/common/exit.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/type/buffer.c
M src/common/type/json.c
M src/common/type/string.c
M src/common/type/string.h
M src/common/type/stringList.c
M src/config/load.c
M src/config/parse.c
M src/db/db.c
M src/info/manifest.c
M src/protocol/client.c
M src/storage/s3/storage.c
M test/src/common/harnessLog.c
M test/src/common/harnessStorage.c
M test/src/module/command/backupTest.c
M test/src/module/command/expireTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/config/parseTest.c
M test/src/module/performance/typeTest.c
M test/src/module/storage/s3Test.c

Remove expect logs obsoleted in a3e5e66f.

commit   : dab00e2010dca8acdd559ebfc0161b69a936842d    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 24 Jun 2020 07:45:00 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 24 Jun 2020 07:45:00 -0400    

Click here for diff

These expect logs are no longer used but are not automatically removed by test.pl.  

D test/expect/real-all-002.log
D test/expect/real-all-003.log
D test/expect/real-all-004.log
D test/expect/real-all-005.log
D test/expect/real-all-006.log

Simplify test matrix for real/all tests.

commit   : a3e5e66f05362cc3a0ea41db0098c2fca56c4cd7    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Jun 2020 13:44:29 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Jun 2020 13:44:29 -0400    

Click here for diff

Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added.  
  
Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro.  
  
Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it.  
  
Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6.  
  
The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them.  
  
The following is an example of the new test log (note longer runtime of each test):  
  
module=real, test=all, run=1, pg-version=10 (106.91s)  
module=real, test=all, run=1, pg-version=9.5 (151.09s)  
module=real, test=all, run=1, pg-version=9.2 (123.11s)  
module=real, test=all, run=1, pg-version=9.1 (129s)  
  
vs. the old test log (sub-second tests were skipped entirely):  
  
module=real, test=all, run=2, pg-version=10 (0.31s)  
module=real, test=all, run=3, pg-version=10 (0.26s)  
module=real, test=all, run=4, pg-version=10 (60.39s)  
module=real, test=all, run=1, pg-version=10 (69.12s)  
module=real, test=all, run=6, pg-version=10 (34s)  
module=real, test=all, run=5, pg-version=10 (42.75s)  
module=real, test=all, run=2, pg-version=9.5 (0.21s)  
module=real, test=all, run=3, pg-version=9.5 (0.21s)  
module=real, test=all, run=4, pg-version=9.5 (0.21s)  
module=real, test=all, run=5, pg-version=9.5 (0.26s)  
module=real, test=all, run=6, pg-version=9.5 (0.21s)  
module=real, test=all, run=1, pg-version=9.2 (72.78s)  
module=real, test=all, run=2, pg-version=9.2 (0.26s)  
module=real, test=all, run=3, pg-version=9.2 (0.31s)  
module=real, test=all, run=4, pg-version=9.2 (0.21s)  
module=real, test=all, run=5, pg-version=9.2 (0.21s)  
module=real, test=all, run=6, pg-version=9.2 (0.21s)  
module=real, test=all, run=1, pg-version=9.5 (88.41s)  
module=real, test=all, run=2, pg-version=9.1 (0.21s)  
module=real, test=all, run=3, pg-version=9.1 (0.26s)  
module=real, test=all, run=4, pg-version=9.1 (0.21s)  
module=real, test=all, run=5, pg-version=9.1 (0.31s)  
module=real, test=all, run=6, pg-version=9.1 (0.26s)  
module=real, test=all, run=1, pg-version=9.1 (72.4s)  

M test/define.yaml
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Ignore "unsupported frontend protocol" error on Centos/RHEL 6.

commit   : d560c1bf19b2ef3699971790e1f887dac4168964    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Jun 2020 12:42:46 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Jun 2020 12:42:46 -0400    

Click here for diff

The unsupported version error is showing up on older versions of PostgreSQL (e.g. 9.1, 9.2) on RHEL6 when setting up a standby with streaming replication. The error occurs when a client does not properly send a version number and it's not clear why it is happening here, but it does not appear to have anything to do with pgBackRest and only affects RHEL6, i.e. 9.1 and 9.2 do not show this error on other distros.  
  
For now ignore the error since RHEL6 is nearly EOL.  

M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm

Increase log level of checkManifest() to debug.

commit   : 04b2e4a831409d1a2edff669bedb6108a9ca20ed    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 23 Jun 2020 09:24:18 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 23 Jun 2020 09:24:18 -0400    

Click here for diff

This function is only called once and is very likely throw errors so debug level is more appropriate.  

M src/command/check/check.c

Rename http/Http to HTTP in comments and messages.

commit   : 1aedc75b03c5d17acde853918150c2dc043dae35    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 21 Jun 2020 11:47:41 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 21 Jun 2020 11:47:41 -0400    

Click here for diff

HTTP is an acronym so it should be capitalized. Coding conventions dictate otherwise for function and type names but that should not have been propagated to comments and messages.  

M src/common/io/http/cache.c
M src/common/io/http/cache.h
M src/common/io/http/client.c
M src/common/io/http/client.h
M src/common/io/http/common.c
M src/common/io/http/common.h
M src/common/io/http/header.c
M src/common/io/http/header.h
M src/common/io/http/query.c
M src/common/io/http/query.h
M src/storage/s3/read.c
M src/storage/s3/storage.c
M test/src/module/common/ioHttpTest.c

Add httpDateFromTime().

commit   : 911384d9b9b7cfd042a18eeb79b5a6f0c79544e7    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 21 Jun 2020 11:07:18 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 21 Jun 2020 11:07:18 -0400    

Click here for diff

Also rename httpLastModifiedToTime() to httpDateToTime() since the RFC-2822 date format used by HTTP is used in all Date headers.  

M src/common/io/http/common.c
M src/common/io/http/common.h
M src/storage/s3/storage.c
M test/src/module/common/ioHttpTest.c

Inline strPtr() to increase profiling accuracy.

commit   : fbff29957cd165e6262359629a672e240d45c3bb    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Thu, 18 Jun 2020 13:13:55 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Thu, 18 Jun 2020 13:13:55 -0400    

Click here for diff

strPtr() is called more than any other function and during profiling (with or without optimization) it can end up using a disproportionate amount of the total runtime. Even though it is fast, the profiler has a minimum resolution for each function call so strPtr() will often end up towards the top of the list even though the real runtime is quite small.
  

  
Instead, inline strPtr() and indicate to gcc that it should be inlined even for non-optimized builds, since that's how profiles are usually generated.
  

  
To make strPtr() smaller require "this" to be non-NULL and add another function, strPtrNull(), to deal with the few cases where we need NULL handling.
  

  
As a bonus this makes the executable about 1% smaller even when compared to a prior optimized build which would inline some percentage of strPtr() calls.

M doc/xml/release.xml
M src/common/io/tls/client.c
M src/common/type/string.c
M src/common/type/string.h
M test/src/common/harnessTest.h
M test/src/module/common/typeStringTest.c

Use PostgreSQL instead of postmaster where appropriate.

commit   : 3d74ec1190c14f103c88852e3419d8b1b49cca50    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Wed, 17 Jun 2020 15:14:59 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Wed, 17 Jun 2020 15:14:59 -0400    

Click here for diff

Using postmaster in messages was not very helpful since users rarely interact directly with the postmaster. Using PostgreSQL instead seems clearer.

M build/error.yaml
M doc/lib/pgBackRestDoc/Common/Exception.pm
M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/restore/restore.c
M src/command/stanza/delete.c
M src/common/error.auto.c
M src/common/error.auto.h
M src/info/manifest.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/command/stanzaTest.c

Add --no-coverage-report to test.pl to disable report generation.

commit   : 417818dccae7b6e1b559d3adb423a26999040882    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 17 Jun 2020 15:07:30 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 17 Jun 2020 15:07:30 -0400    

Click here for diff

There is no sense in generating detailed coverage reports in CI environments where they will never be seen. It takes time and format differences in some older versions can cause problems in the report generation code.  
  
Note that missing coverage will still be reported on stdout and the test will fail.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/test.pl
M test/travis.pl

Update TEST_RESULT_PTR() to TEST_RESULT_STR() where appropriate.

commit   : ea984c4d3e4749afa8b17bfd991023e590cb76b8    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 17 Jun 2020 09:46:09 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 17 Jun 2020 09:46:09 -0400    

Click here for diff

These were missed in d41eea68 when the functionality of TEST_RESULT_STR() was changed. Using TEST_RESULT_STR() instead of TEST_RESULT_PTR() is more type-safe and clearer.  
  
Add a comment to make it clear that TEST_RESULT_PTR() should be used only when a better alternative is not available.  

M test/src/common/harnessTest.h
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/typeXmlTest.c
M test/src/module/config/configTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/infoTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Fix incorrect param log types.

commit   : c4fe09dabe48c9aa2ccff2b1cd94d0614c3808e9    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 16 Jun 2020 19:25:16 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 16 Jun 2020 19:25:16 -0400    

Click here for diff

M src/command/info/info.c

Make sure functions/types needed for profiling are defined.

commit   : 6a851994f3c2822115e1819f9739368344d9c08a    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 16 Jun 2020 15:40:32 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 16 Jun 2020 15:40:32 -0400    

Click here for diff

Profiling runs with debugging disabled but the tests still need TestError and stackTraceTestFileLineSet() to be valid even if they are noops.  

M src/common/error.h
M src/common/stackTrace.h

Rename most instances of master to primary in tests.

commit   : 0680cfc8dca832d4733302cce5bc67bfe1fedee2    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 16 Jun 2020 14:06:38 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 16 Jun 2020 14:06:38 -0400    

Click here for diff

This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702).  
  
Usage in the backup.manifest tests has not been updated since it might break the file format.  

M doc/xml/release.xml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/stanzaTest.c

Add hint when checksum delta is enabled after a timeline switch.

commit   : 11c192f30ee859e011f1db026fcff76e6560eed0    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Tue, 16 Jun 2020 13:20:01 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Tue, 16 Jun 2020 13:20:01 -0400    

Click here for diff

This warning is normal when restoring a backup or promoting a standby so add a hint to make that clear.

M doc/xml/release.xml
M src/info/manifest.c
M test/src/module/command/backupTest.c
M test/src/module/info/manifestTest.c

Update the PITR FAQ to clarify the default behavior.

commit   : 1094a2d802a9c3e0af9f222e8e0e74dc81128086    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Fri, 12 Jun 2020 11:27:18 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Fri, 12 Jun 2020 11:27:18 -0400    

Click here for diff

M doc/xml/faq.xml
M doc/xml/release.xml

Update FAQ page for expiring a specific backup set.

commit   : a60d4c939accb0e3d380968ed2bbcbe55ac8249c    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Thu, 11 Jun 2020 14:06:36 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Thu, 11 Jun 2020 14:06:36 -0400    

Click here for diff

The FAQ should have been updated with the addition of ad hoc expire in 1c1a7104.

M doc/xml/faq.xml
M doc/xml/release.xml

Improve behavior of the repo-ls command.

commit   : 6fe60a2428f5c28a2da66b84cb2d7910155df60c    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Thu, 11 Jun 2020 13:17:35 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Thu, 11 Jun 2020 13:17:35 -0400    

Click here for diff

* Exclude linefeed when there is no output to avoid a blank line.
  
* Honor filter when adding . path or listing a single file.

M doc/xml/release.xml
M src/command/repo/ls.c
M test/src/module/command/repoTest.c

Fix expression when recursion enabled in storageInfoListP().

commit   : 237ba54d206b2a598ba0113bbaa5bedf39b6f583    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Thu, 11 Jun 2020 11:48:42 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Thu, 11 Jun 2020 11:48:42 -0400    

Click here for diff

Expressions only worked at the first level of recursion because the expression was also being applied to paths so the path had to match the filter in order to recurse.
  

  
This is not considered a bug since it does not affect any existing code paths, but it is required for the general-purpose repo-ls command.

M doc/xml/release.xml
M src/storage/storage.c
M test/src/module/storage/posixTest.c

Improve error when pg1-path option missing for archive-get command.

commit   : da4f15663b517ce7922012efc05105e1f9d0e6e6    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 10 Jun 2020 11:41:08 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 10 Jun 2020 11:41:08 -0400    

Click here for diff

The assert thrown was not as descriptive as a proper option missing error.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/config/define.auto.c
M test/src/module/command/archiveCommonTest.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/controlTest.c
M test/src/module/command/localTest.c
M test/src/module/command/remoteTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/config/protocolTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/cifsTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/s3Test.c

Better error when closing an already closed session in TLS test harness.

commit   : d0e08a537d36bb1b321ab3c8a305e4988666364d    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 5 Jun 2020 15:08:39 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 5 Jun 2020 15:08:39 -0400    

Click here for diff

M test/src/common/harnessTls.c

Add missing ioReadClose().

commit   : 38bf3d51546486a5085e9dda3cfc7c5bdad24b46    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 3 Jun 2020 09:02:53 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 3 Jun 2020 09:02:53 -0400    

Click here for diff

This probably wouldn't be a big issue since the close would happen when the IoRead object was destroyed, but better to be safe.  

M src/common/ini.c

Fix incorrect example for repo-retention-full-type option.

commit   : 9efbafc84c4a51f48aaf64616c10de1f395fda10    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 1 Jun 2020 13:19:47 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 1 Jun 2020 13:19:47 -0400    

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml

Update RHEL package to add logrotate script.

commit   : ae15aced9983daebe5b935670e55095e8b03e4f6    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 1 Jun 2020 12:24:19 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 1 Jun 2020 12:24:19 -0400    

Click here for diff

M test/test.pl

Remove exclamations from test data.

commit   : fe829af4ece88afabc026e876250fc04851bb51c    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 28 May 2020 10:27:45 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 28 May 2020 10:27:45 -0400    

Click here for diff

Three exclamations are commonly used to mark areas of the code that need attention before commit so having them in a test is distracting.  

M test/src/module/command/backupTest.c

Add missing parameter logging.

commit   : 28676e1707a36e273b7f236d495b3e611a06328d    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 28 May 2020 09:19:12 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 28 May 2020 09:19:12 -0400    

Click here for diff

M src/common/io/handleRead.c

Improve handling of invalid HTTP response status.

commit   : 3b5f76b434a2c5831104fc582327ef1f584f5fe0    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Wed, 27 May 2020 15:13:55 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Wed, 27 May 2020 15:13:55 -0400    

Click here for diff

A truncated HTTP response status could lead to an an unfriendly error message, which would be retried, but could be confusing if the error was persistent and required debugging.
  

  
Improve the error handling overall to catch more error cases explicitly and respond better to edge cases.
  

  
Also update the terminology in comments to align with the RFC. Variable and function names were not changed because a refactor is intended for HTTP response and it doesn't seem worth the additional code churn.

M doc/xml/release.xml
M src/common/io/http/client.c
M test/src/module/common/ioHttpTest.c

Remove Debian package patch now that it has been merged upstream.

commit   : d05090ab7bacdd95cc5c2d4b45fd25b7bc71a602    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 27 May 2020 09:22:30 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 27 May 2020 09:22:30 -0400    

Click here for diff

D test/patch/debian-package.patch

Refactor TLS server test harness for ease of use.

commit   : b27f9e886b4f21ac199ededde3a189355eac9f88    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 26 May 2020 09:16:57 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 26 May 2020 09:16:57 -0400    

Click here for diff

The prior harness required a separate function to contain the server behavior but this made keeping the client/server code in sync very difficult and in general meant test writing took longer.  
  
Now, commands to define server behavior are inline with the client code, which should greatly simplify test writing.  

M test/src/common/harnessTls.c
M test/src/common/harnessTls.h
M test/src/module/common/ioHttpTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/storage/s3Test.c

Update install-sh to v1.16.2.

commit   : d92d0513c09c3e28a4c5961c4fc5a62aaac263d8    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 26 May 2020 08:40:40 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 26 May 2020 08:40:40 -0400    

Click here for diff

No version bump for some reason.  
  
https://git.savannah.gnu.org/gitweb/?p=automake.git;a=commit;h=6774c9b219cd78445964cd694588cfc4df9b8316  

M doc/RELEASE.md
M src/build/install-sh

Begin v2.28 development.

commit   : 943b80e1a712b9040967ff8a4aeb4a0f0e8f39a9    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 26 May 2020 08:30:27 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 26 May 2020 08:30:27 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h