pgBackRest 2.25 (upcoming) commit log

Skip pg_internal.init temp file during backup.

commit   : cc743f2e04db05cc3277e43023ebd8e2007ef4ed    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 21 Feb 2020 11:51:39 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 21 Feb 2020 11:51:39 -0500    

Click here for diff

If PostgreSQL crashes it can leave behind a pg_internal.init temp file with the pid as the extension, as discussed in https://www.postgresql.org/message-id/flat/20200131045352.GB2631%40paquier.xyz#7700b9481ef5b0dd5f09cc410b4750f6.  On restart this file is not cleaned up so it can persist for the lifetime of the cluster or until another process with the same id happens to write pg_internal.init.  
  
This is arguably a bug in PostgreSQL, but in any case it makes sense not to backup this file.  

M doc/xml/release.xml
M src/info/manifest.c
M test/src/module/info/manifestTest.c

Remove dead LibC macros.

commit   : 48d0f77fe3e6edf7347b49d9e8c67052db00355b    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 21 Feb 2020 11:31:31 -0500    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 21 Feb 2020 11:31:31 -0500    

Click here for diff

These macros were made obsolete when code was removed from LibC after the C migration was completed.  

M libc/LibC.h

Fix typo.

commit   : dfc5f6723379526fd3c5fe5de61a37793e2cced8    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 17 Feb 2020 17:12:22 -0600    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 17 Feb 2020 17:12:22 -0600    

Click here for diff

M src/storage/remote/write.c

Reclassify release note to documentation improvement.

commit   : ea0af890d82b2b335cc4d9a9dcc29d440a583706    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 17:27:44 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 17:27:44 -0700    

Click here for diff

M doc/xml/release.xml

Add reviewer.

commit   : c6b89d74ec5a21fc644f8f5106ee42def1135c98    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 17:20:21 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 17:20:21 -0700    

Click here for diff

M doc/xml/release.xml

Error when archive-get/archive-push/restore are not run on a PostgreSQL host.

commit   : 6353e9428df1d241b97d02c843f1a737e7c36c85    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 17:18:48 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 17:18:48 -0700    

Click here for diff

This error was lost during the migration to C.  The error that occurred instead (generally an SSH auth error) was hard to debug.  
  
Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands.  reset-pg1-host can be used to suppress the error when required.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/archivePushTest.c

Add pgIsLocalVerify().

commit   : dac8119bf11e74b7d393b601bcfbe447a80e85e4    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 15:47:07 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 15:47:07 -0700    

Click here for diff

This functionality is required in commands other than restore, so centralize it.  

M src/Makefile.in
M src/command/restore/restore.c
M src/protocol/helper.c
M src/protocol/helper.h
M test/src/module/protocol/protocolTest.c

Prevent defunct processes in asynchronous archive commands.

commit   : e2c304d4738144f1c5c72d43d70ec5c9e67d052b    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 12:17:23 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 12 Feb 2020 12:17:23 -0700    

Click here for diff

The main improvement is a double-fork to prevent zombie processes if the parent process exits after the (child) async process. This is a real possibility since the parent process sticks around to monitor the results of the async process.  
  
In the first fork, ignore SIGCHLD in the very unlikely case that the async process exits before the first fork. This is probably only possible if the async process exits immediately, perhaps due to a chdir() failure. Set SIGCHLD back to default in the async process so waitpid() will work as expected.  
  
Also update the comment on chdir() to more accurately reflect what is happening.  
  
Finally, add a test in certain debug builds to ensure the first fork exits very quickly. This only works when valgrind is not in use because valgrind makes forking so slow that it is hard to tell if the async process performed work or not (in the case that the second fork goes missing and the async process is a direct child).  

M doc/xml/release.xml
M src/command/archive/common.c
M src/common/fork.c
M test/lib/pgBackRestTest/Common/JobTest.pm

Prevent lock-bot from marking locked issues as "resolved".

commit   : 1fa3ae2fcd5b90a6f2a4e5dc83b3580297e83c1b    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 11 Feb 2020 19:54:17 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 11 Feb 2020 19:54:17 -0700    

Click here for diff

This is certainly not true in all cases, e.g. an issue may be closed if it is added to the backlog.  

M .github/lock.yml

Prevent lock-bot from adding comments to locked issues.

commit   : 1be9e6854e8ab4bc21d693371a09290a6c3d5db6    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 11 Feb 2020 19:52:23 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 11 Feb 2020 19:52:23 -0700    

Click here for diff

This will hopefully prevent users from getting notifications when an issue is locked.  

M .github/lock.yml
M doc/xml/release.xml

Fix resume when the resumable backup was created by Perl.

commit   : 43936c58a894187b41501ba5acaf16e92842c261    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 11 Feb 2020 19:44:06 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 11 Feb 2020 19:44:06 -0700    

Click here for diff

In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully.  

M doc/xml/release.xml
M src/info/manifest.c
M test/src/module/info/manifestTest.c

Consolidate archive async exec code.

commit   : 44adf21c834ca8624b47dc6dbc2794bc429463ad    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 10 Feb 2020 21:30:43 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 10 Feb 2020 21:30:43 -0700    

Click here for diff

Move duplicated code to the common module.  This will reduce copy and paste between the get and push modules when changes are made.  

M src/Makefile.in
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M test/define.yaml

Improve async archive error file removal.

commit   : 0eaedc9a6ae98eaf4e733a3f63cb462a5ebcfa05    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 10 Feb 2020 19:17:11 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 10 Feb 2020 19:17:11 -0700    

Click here for diff

2a06df93 removed the error file so an old error would not be reported before the async process had a chance to try again.  However, if the async process was already running this might lead to a timeout error before reporting the correct error.  
  
Instead, remove the error files once we know that the async process will start, i.e. after the archive lock has been acquired.  
  
This effectively reverts 2a06df93.  

M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M test/src/module/command/archiveCommonTest.c

Fix incorrect error code.

commit   : 8cfbc294fcc066fce594c09649865f3d88bb2364    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 10 Feb 2020 18:48:47 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 10 Feb 2020 18:48:47 -0700    

Click here for diff

M src/common/fork.c

Add missing linefeed.

commit   : 1ce71b1e9b5e0613182d64673e9f730efd0e8176    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 10 Feb 2020 17:44:39 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 10 Feb 2020 17:44:39 -0700    

Click here for diff

M src/storage/helper.c

Rename confessOnError to throwOnError.

commit   : 71b4cc56cbcf20ce6a39e2155e0d9da11c83aaf8    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 6 Feb 2020 21:11:15 -0800    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 6 Feb 2020 21:11:15 -0800    

Click here for diff

Confess is awfully Perl-ish and was likely copied verbatim during the migration.  Rename to what we do now, i.e. throw.  

M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/archive/push/push.c

Remove async archive error file when not throwing an error.

commit   : 2a06df93f379bdc28a6c5084d204cb1e5391dbb3    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 6 Feb 2020 20:59:04 -0800    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 6 Feb 2020 20:59:04 -0800    

Click here for diff

This ensures that the error will not be thrown before the async process has a chance to retry.  

M src/command/archive/common.c
M test/src/module/command/archiveCommonTest.c

Clarify why some recovery options are not commented out for PG >= 12.

commit   : 3721e57a0ec709326c28eb6d31f9b4fe9e3fd7b5    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 6 Feb 2020 18:28:54 -0800    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 6 Feb 2020 18:28:54 -0800    

Click here for diff

M src/command/restore/restore.c

Fix release note typo.

commit   : efff54490f456b653e51abb05507b0917792376e    
  
author   : Mike Palmiotto <mike.palmiotto@crunchydata.com>    
date     : Tue, 4 Feb 2020 21:19:21 -0800    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 4 Feb 2020 21:19:21 -0800    

Click here for diff

M doc/xml/release.xml

Update contributor name.

commit   : 296aec03be7c924f2689e091e667e80daf06e6f7    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 31 Jan 2020 07:50:03 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 31 Jan 2020 07:50:03 -0700    

Click here for diff

M doc/xml/release.xml

Read HTTP content to eof when size/encoding not specified.

commit   : 0f8ec3e478cc17243bc3f043a13f5da73f98de66    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 30 Jan 2020 14:51:26 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 30 Jan 2020 14:51:26 -0700    

Click here for diff

Generally, the content-size or content-encoding headers will be used to specify how much content should be expected.  
  
There is a special case where the server sends 'Connection:close' without the content headers and the content may be read up until eof.  
  
This appears to be an atypical usage but it is required by the specification.  

M doc/xml/release.xml
M src/common/io/http/client.c
M test/src/module/common/ioHttpTest.c

Auto-select backup set on restore when time target is specified.

commit   : 856980ae99efe088e0cf10588c2a18cd78ff32e6    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Thu, 30 Jan 2020 14:38:05 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 30 Jan 2020 14:38:05 -0700    

Click here for diff

Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used.  
  
Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.  

M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/restore/restore.c
M test/define.yaml
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/command/restoreTest.c

Add timezone calculations to time module.

commit   : f46d1fa74c4c0cef7a0ac00f3e43ebeecdc1315e    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Thu, 30 Jan 2020 11:28:30 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 30 Jan 2020 11:28:30 -0700    

Click here for diff

Add tzPartsValid() and tzOffsetSecond() to calculate timezone offsets from user provided values.  
  
Update epochFromParts() to accept a timezone offset in seconds.  

M src/common/io/http/common.c
M src/common/time.c
M src/common/time.h
M src/storage/s3/storage.c
M test/src/module/common/timeTest.c

Add linefeeds to function declarations.

commit   : dbaa5e3473ebfb4651e879edaab6817fdd79092a    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Wed, 29 Jan 2020 08:21:36 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 29 Jan 2020 08:21:36 -0700    

Click here for diff

M src/common/time.c

Free TLS connection in common/io-http test.

commit   : 80687cbe740ad786a15db5e76777f24fd7cb8c69    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 28 Jan 2020 10:19:58 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 28 Jan 2020 10:19:58 -0700    

Click here for diff

The test that checks for no output from the server was leaving a connection open which valgrind was complaining about.  
  
Wait on the server long enough to cause the error on the client then close the connection to free the memory.  

M test/src/module/common/ioHttpTest.c

```Revert ‘Add lib path for libpq in case it is in a non-standard location.````

commit   : 846efaa40f2a04273a61057a3684973098deb690    
  
author   : David Steele <david@pgmasters.net>    
date     : Tue, 28 Jan 2020 07:36:20 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Tue, 28 Jan 2020 07:36:20 -0700    

Click here for diff

Putting this before AC_CHECK_LIB breaks on many systems because the location of pg_config is not yet known.  

M src/configure
M src/configure.ac

Designated initializer cleanup.

commit   : 324f7cebe04e97a9ec34e325f6fbd927f77edb2c    
  
author   : Cynthia Shang <cynthia.shang@crunchydata.com>    
date     : Mon, 27 Jan 2020 17:50:07 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 27 Jan 2020 17:50:07 -0700    

Click here for diff

Cleanup designated initializers created in b134175f by moving struct members in or out for clarity.  

M src/command/restore/restore.c
M src/db/db.c
M src/protocol/parallelJob.c

Fix incomplete comment.

commit   : 24d2494c8209079d0288784de38e9045d5c6cb95    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 27 Jan 2020 11:25:24 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 27 Jan 2020 11:25:24 -0700    

Click here for diff

M src/info/manifest.c

Fix typo.

commit   : 0a845214a1d306d16dce5618f2f998f8f6e6778d    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 26 Jan 2020 23:10:29 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 26 Jan 2020 23:10:29 -0700    

Click here for diff

M doc/resource/git-history.cache

Add more validations to the manifest on backup.

commit   : 697150eaf875742a4522e2ed8c00dfaf0489c4ac    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 26 Jan 2020 23:07:07 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 26 Jan 2020 23:07:07 -0700    

Click here for diff

Validate that checksums exist for zero size files.  This means that the checksums for zero size files are explicitly set by backup even though they'll always be the same.  Also validate that zero length files have the correct checksum.  
  
Validate that repo size is > 0 if size is > 0.  No matter what compression type is used a non-zero amount of data cannot be stored in zero bytes.  

M doc/xml/release.xml
M src/command/backup/backup.c
M src/command/restore/restore.c
M src/info/manifest.c
M src/info/manifest.h
M test/src/module/info/manifestTest.c

Begin v2.24 development.

commit   : bb45a80d46b738a9b6672fc77158585395b04473    
  
author   : David Steele <david@pgmasters.net>    
date     : Sun, 26 Jan 2020 22:47:53 -0700    
  
committer: David Steele <david@pgmasters.net>    
date     : Sun, 26 Jan 2020 22:47:53 -0700    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/version.h