pgBackRest 2.4 commit log

by Yogesh Sharma
pgBackRest
CommitLog

v2.04: Critical Bug Fix for Backup Resume

commit   : a8143ec1253f3d9187c053def0b8c515d4f13fd5    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Jul 2018 20:16:41 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Jul 2018 20:16:41 -0400

Click here for diff

IMPORTANT NOTE: This release fixes a critical bug in the backup resume feature. All resumed backups prior to this release should be considered inconsistent. A backup will be resumed after a prior backup fails, unless resume=n has been specified. A resumed backup can be identified by checking the backup log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". If the message exists, do not use this backup or any backup in the same set for a restore and check the restore logs to see if a resumed backup was restored. If so, there may be inconsistent data in the cluster.  
  
Bug Fixes:  
  
* Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. See note above for more details. (Reported by David Youatt, Yogesh Sharma, Stephen Frost.)  
* Fix error in selective restore when only one user database exists in the cluster. (Fixed by Cynthia Shang. Reported by Nj Baliyan.)  
* Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz.)  
  
Features:  
  
* PostgreSQL 11 Beta 2 support.  
  
Improvements:  
  
* Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)  
* Set search_path = 'pg_catalog' on PostgreSQL connections. (Suggested by Stephen Frost.)

M README.md
M doc/resource/exe.cache
M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h

Remove call to lscpu which can vary widely by build host.

commit   : 6f6dad6dd5fb0f2ba5824b1812e71abf047704a4    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Jul 2018 16:49:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Jul 2018 16:49:46 -0400

Click here for diff

M doc/xml/release.xml
M doc/xml/user-guide.xml

Add zero-length file to mock/all test.

commit   : 22adb395db804e03646fc53d84b6e7935c38c46e    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Jul 2018 15:40:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Jul 2018 15:40:50 -0400

Click here for diff

M doc/xml/release.xml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Fix critical bug in resume that resulted in inconsistent backups.

commit   : db17973cd0a8a5692e408621703fd83fbce40166    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 Jul 2018 14:01:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 Jul 2018 14:01:57 -0400

Click here for diff

A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed".  
  
Reported by David Youatt, Yogesh Sharma, Stephen Frost.

M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Backup/File.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Make ls ordering deterministic in mock/all test.

commit   : 3d07e01fe920bef0d4848ddd55560dc13e38ef85    
  
author   : David Steele <[email protected]>    
date     : Sun, 1 Jul 2018 20:02:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 1 Jul 2018 20:02:37 -0400

Click here for diff

M doc/xml/release.xml
M test/expect/mock-all-001.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Fix non-compliant ISO-8601 timestamp format in S3 authorization headers.

commit   : 1bd98b61df7aeaec25b6d6b5baa71f8a458784ee    
  
author   : Andrew Schwartz <[email protected]>    
date     : Sun, 1 Jul 2018 08:17:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 1 Jul 2018 08:17:27 -0400

Click here for diff

AWS and some gateways were tolerant of space rather than zero-padded hours while others were not.  
  
Fixed by Andrew Schwartz.

M doc/xml/release.xml
M lib/pgBackRest/Storage/S3/Auth.pm
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Storage/StorageS3AuthPerlTest.pm

PostgreSQL 11 Beta 2 support.

commit   : 7e65ddad34db74a14e54aab125f0e62596268f6e    
  
author   : David Steele <[email protected]>    
date     : Sat, 30 Jun 2018 14:55:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 30 Jun 2018 14:55:25 -0400

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Db.pm
M src/perl/embed.auto.c
M src/postgres/info.c
M test/src/module/postgres/infoTest.c

Improve the HTTP client to set content-length to 0 when not specified by the server.

commit   : 7b0e65d48823e94a7a17ae377e54a898bb07d7fb    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jun 2018 17:27:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jun 2018 17:27:22 -0400

Click here for diff

S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either.  
  
Suggested by Adam K. Sumner.

M doc/xml/release.xml
M lib/pgBackRest/Common/Http/Client.pm
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Common/CommonHttpClientPerlTest.pm

Update parameters for VBoxService start.

commit   : e8c68b9c6657d7fabfd70cbd8ad3778e123bcb41    
  
author   : David Steele <[email protected]>    
date     : Tue, 26 Jun 2018 16:56:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 26 Jun 2018 16:56:05 -0400

Click here for diff

M doc/xml/release.xml
M test/Vagrantfile

Add sample S3 policy to restrict bucket privileges.

commit   : ead70e16bbead338b7ba14b4a03014187bb80c23    
  
author   : David Steele <[email protected]>    
date     : Sat, 23 Jun 2018 13:09:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 23 Jun 2018 13:09:56 -0400

Click here for diff

Suggested by Douglas J Hunley, Jason O'Donnell.

M doc/xml/release.xml
M doc/xml/user-guide.xml

Set search_path = 'pg_catalog' on PostgreSQL connections.

commit   : aa41e00c9cdbb966a6cda088a21fb16c443163d4    
  
author   : David Steele <[email protected]>    
date     : Thu, 21 Jun 2018 11:39:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 21 Jun 2018 11:39:37 -0400

Click here for diff

Suggested by Stephen Frost.

M doc/xml/release.xml
M lib/pgBackRest/Db.pm
M src/perl/embed.auto.c

Remove Debian package patch since it has been committed upstream.

commit   : 7fee739710b052f41ff881b513ae376acb1e5c2a    
  
author   : David Steele <[email protected]>    
date     : Wed, 20 Jun 2018 18:26:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 20 Jun 2018 18:26:53 -0400

Click here for diff

M doc/xml/release.xml
D test/patch/debian-package.patch

Fix default location of pgbackrest.conf in option reference.

commit   : 6084e9be04dd958f8ab8906c1af8e3ed1fca4994    
  
author   : David Steele <[email protected]>    
date     : Sat, 16 Jun 2018 13:07:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 16 Jun 2018 13:07:01 -0400

Click here for diff

M doc/xml/reference.xml
M doc/xml/release.xml

Preliminary documentation for PostgreSQL 11 unprivileged user backup.

commit   : 1a0d56860059e7f9998e1faf5d384017ca1511ee    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Jun 2018 19:05:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Jun 2018 19:05:35 -0400

Click here for diff

M doc/xml/release.xml
M doc/xml/user-guide.xml

Convert the not very portable uint type to unsigned int.

commit   : 3793ae1e4f0a4deeb156fa3d0ace4ddab0b85804    
  
author   : David Steele <[email protected]>    
date     : Thu, 14 Jun 2018 17:59:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Jun 2018 17:59:10 -0400

Click here for diff

Suggested by Devrim Gündüz.

M doc/xml/release.xml
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/get.c
M src/command/help/help.c
M src/postgres/info.c
M src/postgres/info.h
M test/src/module/common/typeConvertTest.c

Fix error in selective restore when only one user database exists in the cluster.

commit   : 4e7692b74d229538fdde123a6020258a9ff5f8f3    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 14 Jun 2018 17:44:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 14 Jun 2018 17:44:44 -0400

Click here for diff

Fixed by Cynthia Shang.  
Reported by Nj Baliyan.

M doc/xml/release.xml
M lib/pgBackRest/Restore.pm
M src/perl/embed.auto.c

Create a new section to describe building pgBackRest and build on a separate host.

commit   : fbd1da05230bb4790f143443dd515453fada0bba    
  
author   : David Steele <[email protected]>    
date     : Wed, 13 Jun 2018 13:43:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 13 Jun 2018 13:43:17 -0400

Click here for diff

M doc/xml/release.xml
M doc/xml/user-guide.xml

Use a prebuilt s3 server container for documentation and tests.

commit   : d55e609959547750116be2954f956e7e918013b5    
  
author   : David Steele <[email protected]>    
date     : Tue, 12 Jun 2018 13:43:15 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 12 Jun 2018 13:43:15 -0400

Click here for diff

M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/lib/BackRestDoc/Latex/DocLatexSection.pm
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml
M doc/xml/user-guide.xml
A test/docker/s3-server.docker
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Move cryptographic hash functions to C using OpenSSL.

commit   : 350b30fa4977e23b205e05b787d707eaceb5a67f    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Jun 2018 14:52:26 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 11 Jun 2018 14:52:26 -0400

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Common/Ini.pm
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Manifest.pm
M lib/pgBackRest/Storage/Filter/CipherBlock.pm
M lib/pgBackRest/Storage/Filter/Sha.pm
M lib/pgBackRest/Storage/S3/Auth.pm
M lib/pgBackRest/Storage/S3/Request.pm
M libc/LibC.xs
M libc/Makefile.PL
M libc/build/lib/pgBackRestLibC/Build.pm
M libc/typemap
A libc/xs/crypto/hash.xs
A libc/xs/crypto/hash.xsh
M src/Makefile
A src/crypto/hash.c
A src/crypto/hash.h
M src/perl/embed.auto.c
M src/perl/libc.auto.c
M test/define.yaml
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Archive/ArchiveGetPerlTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageFilterCipherBlockPerlTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageFilterGzipPerlTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageFilterShaPerlTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageLocalPerlTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageS3PerlTest.pm
A test/src/module/crypto/hashTest.c
M test/test.pl

Rename cipher module to the more general crypto.

commit   : 064ec757e9da187ff4a6dad481dc32775454dee3    
  
author   : David Steele <[email protected]>    
date     : Mon, 11 Jun 2018 10:53:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 11 Jun 2018 10:53:16 -0400

Click here for diff

M doc/xml/release.xml
M libc/LibC.xs
M libc/Makefile.PL
R100 libc/xs/cipher/block.xs libc/xs/crypto/cipherBlock.xs
R095 libc/xs/cipher/block.xsh libc/xs/crypto/cipherBlock.xsh
R100 libc/xs/cipher/random.xs libc/xs/crypto/random.xs
M src/Makefile
R064 src/cipher/cipher.h src/crypto/cipher.h
R098 src/cipher/block.c src/crypto/cipherBlock.c
R095 src/cipher/block.h src/crypto/cipherBlock.h
R087 src/cipher/cipher.c src/crypto/crypto.c
A src/crypto/crypto.h
R097 src/cipher/random.c src/crypto/random.c
R092 src/cipher/random.h src/crypto/random.h
M src/perl/libc.auto.c
M test/define.yaml
R099 test/src/module/cipher/blockTest.c test/src/module/crypto/cipherBlockTest.c
A test/src/module/crypto/cryptoTest.c
R100 test/src/module/cipher/randomTest.c test/src/module/crypto/randomTest.c

Document generator improvements.

commit   : 835396751f220be3f8e46f344d4eadfd7b7305b3    
  
author   : David Steele <[email protected]>    
date     : Sun, 10 Jun 2018 14:13:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 10 Jun 2018 14:13:56 -0400

Click here for diff

* Build containers from scratch for more accurate testing.  
* Allow environment load to be skipped.  
* Allow bash wrapping to be skipped.  
* Allow forcing a command to run as a user without sudo.

M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml
M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Common/HostGroupTest.pm
M test/lib/pgBackRestTest/Common/HostTest.pm
M test/travis.pl

Update Debian package to add debug symbols to pgBackRest executable.

commit   : 5c0099781d170409a0121355fb8b4917ca06ac0a    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Jun 2018 22:19:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 Jun 2018 22:19:01 -0400

Click here for diff

M doc/xml/release.xml
M test/patch/debian-package.patch

Update primary test environment (Vagrant and Docker) to Ubuntu 18.04.

commit   : a385cb520b02c8e216bde78de51e57c216534778    
  
author   : David Steele <[email protected]>    
date     : Wed, 6 Jun 2018 15:52:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 6 Jun 2018 15:52:28 -0400

Click here for diff

M .travis.yml
M doc/xml/release.xml
M lib/pgBackRest/Common/Ini.pm
M src/common/type/buffer.c
M src/config/config.c
M src/config/parse.c
M src/perl/embed.auto.c
M src/perl/libc.auto.c
M test/.gitignore
M test/Vagrantfile
M test/define.yaml
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
A test/package/u14-libdevel-cover-perl_1.29-2_amd64.deb
A test/package/u18-libdevel-cover-perl_1.29-2_amd64.deb
M test/test.pl
M test/travis.pl

PostgreSQL 11 Beta 1 support.

commit   : d309a85b51d732cb983ea781b69a9a1d2d4a1831    
  
author   : David Steele <[email protected]>    
date     : Tue, 5 Jun 2018 08:59:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 5 Jun 2018 08:59:17 -0400

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Archive/Common.pm
M lib/pgBackRest/Db.pm
M lib/pgBackRest/DbVersion.pm
M src/perl/embed.auto.c
M src/postgres/info.c
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/src/module/postgres/infoTest.c

Document generator improvements.

commit   : e57840d739e12aec86fb4772044e0d722f1d9dd6    
  
author   : David Steele <[email protected]>    
date     : Fri, 25 May 2018 13:42:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 25 May 2018 13:42:09 -0400

Click here for diff

* Allow parameters to be passed when a container is created.  
* Allow /etc/hosts update to be skipped (for containers without bash).

M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/HostGroupTest.pm
M test/lib/pgBackRestTest/Common/HostTest.pm

Improve efficiency of C library builds now that they are used only for testing.

commit   : 40093f160cf891c9bb522f4da1f0414aeacbc3e5    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 May 2018 14:01:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 May 2018 14:01:24 -0400

Click here for diff

M doc/xml/release.xml
M libc/Makefile.PL
D libc/t/sanity.t
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/test.pl

Remove RHEL package patch since it has been committed upstream.

commit   : 915b09635a5300f1cb45506c5237c3b883b5a85a    
  
author   : David Steele <[email protected]>    
date     : Thu, 24 May 2018 11:00:47 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 24 May 2018 11:00:47 -0400

Click here for diff

M doc/xml/release.xml
D test/patch/rhel-package.patch

Auto-generate Makefile with dependencies.

commit   : 8eb9921c7ca3d3aaacf1894756be02cbe7d17e18    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 May 2018 14:57:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 23 May 2018 14:57:08 -0400

Click here for diff

M doc/xml/release.xml
M src/Makefile
M src/common/time.c
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/test.pl

Split log levels into separate header file.

commit   : 2f6017511fda3c6aeb67c3daafcc6c1dc35b2dd4    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 May 2018 09:45:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 23 May 2018 09:45:08 -0400

Click here for diff

Many modules that use debug.h do not need to do logging so this reduces dependencies for those modules.

M doc/xml/release.xml
M src/cipher/block.c
M src/cipher/cipher.c
M src/cipher/random.c
M src/common/fork.c
M src/common/io/handle.c
M src/common/lock.c
M src/common/log.h
A src/common/logLevel.h
M src/common/stackTrace.c
M src/common/stackTrace.h
M src/common/wait.c
M src/postgres/info.c
M src/postgres/pageChecksum.c
M src/storage/driver/posix/driver.c
M src/storage/driver/posix/driverRead.c
M src/storage/driver/posix/driverWrite.c
M src/storage/fileRead.c
M src/storage/fileWrite.c
M src/storage/storage.c
M test/src/module/common/debugOnTest.c

Begin v2.04 development.

commit   : dfa92b359deddb37b10db5e1bac47750d8d1011c    
  
author   : David Steele <[email protected]>    
date     : Wed, 23 May 2018 09:10:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 23 May 2018 09:10:14 -0400

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h