pgBackRest 2.7 commit log

v2.07: Automatic Backup Checksum Delta

commit   : 04d9e4d5a88512707c350e3885e359d53c3ca36b    
  
author   : David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 09:50:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 09:50:50 -0500    

Click here for diff

Bug Fixes:  
  
* Fix issue with archive-push-queue-max not being honored on connection error. (Reported by Lardière Sébastien.)  
* Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.  
* Fix error after log file open failure when processing should continue. (Reported by vthriller.)  
  
Features:  
  
* Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. (Contributed by Cynthia Shang.)  
  
Improvements:  
  
* Retry all S3 5xx errors rather than just 500 internal errors. (Suggested by Craig A. James.)  

M README.md
M doc/resource/exe.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml

Fix incorrect config constant introduced in 5e3b7cbe.

commit   : 332a68ea8d713f24d1db08e20d5b95bcf32cc74e    
  
author   : David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 08:48:02 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 16 Nov 2018 08:48:02 -0500    

Click here for diff

This commit introduced PGBACKREST_CONFIG_ORIG_PATH_FILE_STR as a String constant for PGBACKREST_CONFIG_ORIG_PATH_FILE but failed to get the value correct.  
  
Also, no test was added for PGBACKREST_CONFIG_ORIG_PATH_FILE_STR to prevent regressions as there is for  PGBACKREST_CONFIG_ORIG_PATH_FILE.  

M src/config/parse.c
M test/src/module/config/parseTest.c

Add \n, \r, and 0 String constants.

commit   : 75f6e45de26cf4ab087dc791f2aa177553584472    
  
author   : David Steele <[email protected]>    
date     : Thu, 15 Nov 2018 16:31:46 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 15 Nov 2018 16:31:46 -0500    

Click here for diff

M src/common/type/string.c
M src/common/type/string.h

Add destructors to IoRead and IoWrite objects.

commit   : b6f7cbc31531b6e88a89ecfe9c8d3ebc1e682d2e    
  
author   : David Steele <[email protected]>    
date     : Thu, 15 Nov 2018 16:25:46 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 15 Nov 2018 16:25:46 -0500    

Click here for diff

These interfaces previously used the memory context of the object they were associated with and did not have their own destructors.  
  
There are times when it is useful to free the interface without also freeing the underlying object so give IoRead and IoWrite their own memory contexts and destructors.  
  
In passing fix a comment type in bufferRead.c.  

M doc/xml/release.xml
M src/common/io/bufferRead.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/write.c
M src/common/io/write.h
M test/src/module/common/ioTest.c

Add ioWriteFlush() to flush pending output.

commit   : 480e1da7986a278a3b5c8aca965e539d383da9cc    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:53:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:53:42 -0500    

Click here for diff

By default the IoWrite object does not write until the output buffer is full but this is a problem for protocol messages that must be sent in order to get a response.  
  
ioWriteFlush() is not called internally by IoWrite but can be used at any time to immediately write all bytes from the output buffer without closing the IoWrite object.  

M doc/xml/release.xml
M src/common/io/write.c
M src/common/io/write.h
M test/src/module/common/ioTest.c

Modify value of PERL_EMBED_ERROR macro.

commit   : 7918629f5931b2eeca316d8b37191718db4650f4    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:16:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:16:53 -0500    

Click here for diff

Remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.  

M doc/xml/release.xml
M src/perl/exec.h

Add comment regarding vars being required in blocks.

commit   : 504e7d77faeecd213df99ec320246e10a526f9ad    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:13:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:13:01 -0500    

Click here for diff

Documentation block syntax requires that at least one var be specified.  
  
This limitation should be removed but for now add a comment to describe why a bogus var is defined.  

M doc/xml/user-guide.xml

Reword misleading message in stack trace when parameter buffer is full.

commit   : 14d2f8b86c4fd3755af6cd9f5210cf1e34e8d090    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:11:11 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:11:11 -0500    

Click here for diff

The prior message stated that there had been a buffer overrun which is not true since the code prevents that.  
  
In fact, this message means the parameter buffer filled while building the parameter list. Rather than display a partial list we output this message instead.  
  
Also remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.  

M doc/xml/release.xml
M src/common/stackTrace.c
M test/src/module/common/stackTraceTest.c

Clarify comment about main.c being excluded from unit testing.

commit   : 8c7e97a369a145f2d71856f2ff2ed734452a9040    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:08:26 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 08:08:26 -0500    

Click here for diff

Also remove !!! which by convention we use as a marker for code that needs attention before it can be committed to master.  

M test/lib/pgBackRestTest/Common/JobTest.pm

Remove obsolete comments.

commit   : 73102034913ec2e3ca1b1cc485f8ce04820bba27    
  
author   : David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 07:42:28 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 14 Nov 2018 07:42:28 -0500    

Click here for diff

M test/src/module/command/commandTest.c
M test/src/module/perl/configTest.c

Add TEST_LOG() and TEST_LOG_FMT() macros.

commit   : a0b2af567a8b6893c6e9b7f3b9a73acd4496332e    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Nov 2018 20:58:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Nov 2018 20:58:32 -0500    

Click here for diff

These macros provide a convenient way to output debug information in tests.  
  
They are not intended to be left in test code when it is committed to master.  

M doc/xml/release.xml
M test/src/common/harnessTest.h

Add logging macros for TimeMSec type.

commit   : a582ad837f9ada7b924d6c0a23580f88c495ce2e    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Nov 2018 20:43:08 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Nov 2018 20:43:08 -0500    

Click here for diff

M doc/xml/release.xml
M src/common/time.h

Tighten limits on code coverage context selection.

commit   : acb579c4698f856e4c1ceedca751aaeadc9d2d3f    
  
author   : David Steele <[email protected]>    
date     : Tue, 13 Nov 2018 10:37:58 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 13 Nov 2018 10:37:58 -0500    

Click here for diff

If the last } of a function was marked as uncovered then the context selection would overrun into the next function.  
  
Start checking context on the current line to prevent this.  Make the same change for start context even though it doesn't seem to have an issue.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm

Make ioReadLine() read less aggressively.

commit   : 086bc35ddc71b008a5d20f8e5e3d0305508dfdf5    
  
author   : David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 21:18:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 21:18:53 -0500    

Click here for diff

ioReadLine() calls ioRead(), which aggressively tries to fill the output buffer, but this doesn't play well with blocking reads.  
  
Give ioReadLine() an option that tells it to read only what is available.  That doesn't mean the function will never block but at least it won't do so by reading too far.  

M doc/xml/release.xml
M src/common/io/read.c
M test/src/module/common/ioTest.c

Remove error suppression for pgBackRest make.

commit   : bc810e5a87176ee68491e79473325a8a411a34c9    
  
author   : David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 20:44:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 20:44:56 -0500    

Click here for diff

This looks like a copy-pasto because make works just fine without it.  

M doc/xml/release.xml
M doc/xml/user-guide.xml

Allow storage path and file mode to be 0.

commit   : a2672557f80da95d6103d49e307321568285e5d6    
  
author   : David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 20:00:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 20:00:57 -0500    

Click here for diff

Not all storage drivers support Posix-style permissions, e.g. S3.  

M doc/xml/release.xml
M src/storage/driver/posix/storage.c
M src/storage/storage.c
M test/src/module/storage/posixTest.c

Expand context shown in coverage and update colors.

commit   : 7107cc68d23f344e7b13206eaec506a039a00350    
  
author   : David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 18:11:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 18:11:16 -0500    

Click here for diff

Too few lines were shown for coverage context so show the entire function if it has any missing coverage.  
  
Update colors to work with light and dark browser modes.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm

Add EMPTY_STR to String object.

commit   : 0ba2292a29fcef873991098cc48b488580fbb713    
  
author   : David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 14:23:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 14:23:42 -0500    

Click here for diff

M src/common/type/string.c
M src/common/type/string.h

Add strEmpty() to String object.

commit   : aff9693e640c8a0b76e46f31d4fa569fd7d63289    
  
author   : David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 14:13:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Mon, 12 Nov 2018 14:13:55 -0500    

Click here for diff

strEmpty() is more concise and explanatory than strSize() == 0.  

M doc/xml/release.xml
M src/common/type/string.c
M src/common/type/string.h
M test/src/module/common/typeStringTest.c

Use THROW_ON_SYS_ERROR macro to improve fork code coverage.

commit   : 38c5f657704c4c6899a0bf86bbce5fd332d97971    
  
author   : David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 18:12:43 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 18:12:43 -0500    

Click here for diff

Lack of code coverage in this module is just noise since these functions cannot generally be made to fail.  
  
Any failures are fatal.  

M doc/xml/release.xml
M src/common/fork.c

Add KernelError to report miscellaneous kernel errors.

commit   : 72ea47bfb341ff78f2582607e27e7b7df2ac25f8    
  
author   : David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 18:07:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 18:07:56 -0500    

Click here for diff

M build/error.yaml
M doc/xml/release.xml
M lib/pgBackRest/Common/ExceptionAuto.pm
M src/common/error.auto.c
M src/common/error.auto.h
M src/perl/embed.auto.c

Add THROW*_ON_SYS_ERROR* macros to test and throw system errors.

commit   : 257df96b12f38cba45751449dcd64ca712dd96c2    
  
author   : David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 18:06:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 18:06:09 -0500    

Click here for diff

These macros check the error result internally and are appropriate for system calls that won't return errors and so break coverage.  

M doc/xml/release.xml
M src/common/error.h

New, concise coverage report for C.

commit   : 22ecbc153a447b4f635fad2ab71fdd92ed8294bc    
  
author   : David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 17:32:42 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 17:32:42 -0500    

Click here for diff

The report HTML generated by lcov is overly verbose and cumbersome to navigate. Since we maintain 100% coverage it's far more interesting to look at what is not covered than what is.  
  
The new report presents all missing coverage on a single page and excludes code that is covered for brevity.  

M doc/xml/release.xml
A test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/test.pl

Add new HTML tags and strExtra to DocHtmlElement.

commit   : 3cd43fd60cd46430f86084f37b6ed874124e2848    
  
author   : David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 16:34:15 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 11 Nov 2018 16:34:15 -0500    

Click here for diff

Add HTML tags for table elements.  
  
The strExtra parameter allows adhoc tags to be added to an element for features that can't be implemented with CSS, e.g. colspan.  

M doc/lib/BackRestDoc/Html/DocHtmlBuilder.pm
M doc/lib/BackRestDoc/Html/DocHtmlElement.pm
M doc/xml/release.xml

Apply String constant macros where appropriate.

commit   : 5e3b7cbe6ee7a409ad6e311668eb764fee82939c    
  
author   : David Steele <[email protected]>    
date     : Sat, 10 Nov 2018 18:32:49 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 10 Nov 2018 18:32:49 -0500    

Click here for diff

This may not be all the places where the macros can be useful but it's a good start.  

M doc/xml/release.xml
M src/Makefile
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/file.c
M src/command/archive/get/get.c
M src/command/help/help.c
M src/common/io/filter/buffer.c
M src/common/io/filter/size.c
M src/common/lock.c
M src/common/type/variant.c
M src/compress/gzipCompress.c
M src/compress/gzipDecompress.c
M src/config/define.c
M src/config/define.h
M src/config/parse.c
M src/crypto/hash.c
M src/crypto/hash.h
M src/info/info.c
M src/info/infoPg.c
M src/postgres/interface.c
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h
M src/storage/helper.c
M src/storage/helper.h

Add constant macros to String object.

commit   : 8f857a975e22d6a7c0e99a494169f24ed50b6c06    
  
author   : David Steele <[email protected]>    
date     : Sat, 10 Nov 2018 09:37:12 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 10 Nov 2018 09:37:12 -0500    

Click here for diff

There are many places (and the number is growing) where a zero-terminated string constant must be transformed into a String object to be usable.  This pattern wastes time and memory, especially since the created string is generally used in a read-only fashion.  
  
Define macros to create constant String objects that are initialized at compile time rather than at run time.  

M doc/xml/release.xml
M src/common/type/string.c
M src/common/type/string.h
M test/define.yaml
M test/src/module/common/typeStringTest.c

Add regExpPrefix() to aid in static prefix searches.

commit   : df200bee2acf614edf9d008cccde67f02d89f103    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Nov 2018 16:50:22 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Nov 2018 16:50:22 -0500    

Click here for diff

The storageList() command accepts a regular expression as a filter.  This works fine for local filesystems where it is relatively cheap to get a complete list of files and filter them in code.  However, for remote filesystems like S3 it can be expensive to fetch a complete list of files only to discard the bulk of them locally.  
  
S3 does not filter on regular expressions but it can accept a static prefix so this function extracts a prefix from a regular expression when possible.  
  
Even a few characters can drastically reduce the amount of data that must be fetched remotely so the function does not try to be too clever.  It requires a ^ anchor and stops scanning when the first special character is found.  

M doc/xml/release.xml
M src/common/regExp.c
M src/common/regExp.h
M test/define.yaml
M test/src/module/common/regExpTest.c

Improve alignment of expected vs. got error test results.

commit   : 8c504bd2f9e6c14f134b6115f469ec3fe064cb73    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Nov 2018 10:37:59 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Nov 2018 10:37:59 -0500    

Click here for diff

It is easier to compare the error messages when they start at the same column.  

M doc/xml/release.xml
M test/src/common/harnessTest.h

Add strCatChr() to String object.

commit   : b5a3c8c84b789558d11d67199fbcc4029721d939    
  
author   : David Steele <[email protected]>    
date     : Fri, 9 Nov 2018 09:54:55 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 9 Nov 2018 09:54:55 -0500    

Click here for diff

M doc/xml/release.xml
M src/common/type/string.c
M src/common/type/string.h
M test/src/module/common/typeStringTest.c

New test containers.

commit   : 3e695af9614e61f72e9f0d7c3dc729cc5a27f3eb    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 21:41:41 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 21:41:41 -0500    

Click here for diff

* Add libxml2 library needed for S3 development.  
* Minor version updates for PostgreSQL.  
* Remove PostgreSQL 11 beta/rc repository.  

M doc/xml/release.xml
M test/Vagrantfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Add time since the beginning of the run to each test statement.

commit   : 7cf1f42c5c65deed8ad0bdb86a33d273438fe9a0    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 19:50:31 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 19:50:31 -0500    

Click here for diff

Output the time in seconds of each test statement since the start of the test run.  
  
This helps find individual tests that are running slowly.  

M doc/xml/release.xml
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h

Storage interface methods no longer declare the driver as const.

commit   : 2150a26424cacc80ded57697b67af1e3c2a47211    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 18:03:18 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 18:03:18 -0500    

Click here for diff

This works for the Posix driver but is bad for more dynamic drivers (e.g. S3) that need write access to the driver object.  

M doc/xml/release.xml
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileRead.h
M src/storage/driver/posix/fileWrite.c
M src/storage/driver/posix/fileWrite.h
M src/storage/driver/posix/storage.c
M src/storage/driver/posix/storage.h
M src/storage/storage.c
M src/storage/storage.intern.h

Limit usable Buffer size without changing allocated size.

commit   : 12b3be1d8ed38beea67c4d4de582af9039c5bd4b    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 16:48:33 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 16:48:33 -0500    

Click here for diff

Allow buffers to report a lower size than their allocated size. This means a larger buffer can be used to do the work of a smaller buffer without having to create a new buffer and concatenate.  
  
This is useful for blocking I/O where the buffer may be too large for the amount of data that is available to read.  

M doc/xml/release.xml
M src/common/type/buffer.c
M src/common/type/buffer.h
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/src/module/common/typeBufferTest.c

Construct Wait object in milliseconds instead of fractional seconds.

commit   : edb2c6eb26ca77eb3e3e05c006f3036dee2865b7    
  
author   : David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 08:37:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 8 Nov 2018 08:37:57 -0500    

Click here for diff

The Wait object accepted a double in the constructor for wait time but used TimeMSec internally.  This was done for compatibility with the Perl code.  
  
Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code.  
  
Note that Perl still uses a double for its Wait object so translation is needed in some places.  There are no plans to update the Perl code as it will become obsolete.  

M doc/xml/release.xml
M libc/xs/common/lock.xs
M src/Makefile
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/common/debug.h
M src/common/lock.c
M src/common/lock.h
M src/common/wait.c
M src/common/wait.h
M src/perl/libc.auto.c
M src/storage/storage.c
M src/storage/storage.h
M test/src/module/archive/getTest.c
M test/src/module/archive/pushTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/waitTest.c
M test/src/module/storage/posixTest.c

Add memContextCallbackClear() to prevent double free() calls.

commit   : a9feaba9e521e39928ff5fdbc025eeff8fc17646    
  
author   : David Steele <[email protected]>    
date     : Wed, 7 Nov 2018 08:51:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 7 Nov 2018 08:51:32 -0500    

Click here for diff

If an object free() method was called manually when a callback was set then the callback would call free() again.  This meant that each free() method had to protect against a subsequent call.  
  
Instead, clear the callback (if present) before calling memContextFree().  This is faster (since there is no unecessary callback) and removes the need for semaphores to protect against a double free().  

M doc/xml/release.xml
M src/common/memContext.c
M src/common/memContext.h
M src/common/regExp.c
M src/compress/gzipCompress.c
M src/compress/gzipDecompress.c
M src/crypto/cipherBlock.c
M src/crypto/hash.c
M src/storage/driver/posix/fileRead.c
M src/storage/driver/posix/fileWrite.c
M test/src/module/common/memContextTest.c
M test/src/module/compress/gzipTest.c

Merge crypto/random module into crypto/crypto.

commit   : 48d2795f312224f03dc88e99434865540ca71c7e    
  
author   : David Steele <[email protected]>    
date     : Tue, 6 Nov 2018 20:04:16 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 6 Nov 2018 20:04:16 -0500    

Click here for diff

There wasn't enough code to justify a separate module/test and it seems to fit just fine in crypto/crypto.  

M doc/xml/release.xml
M lib/pgBackRest/Common/Cipher.pm
M lib/pgBackRest/LibCAuto.pm
M libc/LibC.xs
M libc/Makefile.PL
M libc/build/lib/pgBackRestLibC/Build.pm
M libc/xs/crypto/random.xs
M src/Makefile
M src/crypto/cipherBlock.c
M src/crypto/crypto.c
M src/crypto/crypto.h
D src/crypto/random.c
D src/crypto/random.h
M src/perl/embed.auto.c
M src/perl/libc.auto.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/module/crypto/cryptoTest.c
D test/src/module/crypto/randomTest.c

Rename CipherError to CryptoError.

commit   : 8efa5e6a6af848a2a9d3e4c3260084d259949ee5    
  
author   : David Steele <[email protected]>    
date     : Tue, 6 Nov 2018 19:38:38 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 6 Nov 2018 19:38:38 -0500    

Click here for diff

This aligns with the general renaming from cipher to crypto.  

M build/error.yaml
M doc/xml/release.xml
M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Backup/Info.pm
M lib/pgBackRest/Common/ExceptionAuto.pm
M lib/pgBackRest/Common/Ini.pm
M lib/pgBackRest/Info.pm
M lib/pgBackRest/Stanza.pm
M src/common/error.auto.c
M src/common/error.auto.h
M src/crypto/cipherBlock.c
M src/crypto/crypto.c
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Archive/ArchiveInfoUnitPerlTest.pm
M test/lib/pgBackRestTest/Module/Backup/BackupInfoUnitPerlTest.pm
M test/lib/pgBackRestTest/Module/Common/CommonIniPerlTest.pm
M test/lib/pgBackRestTest/Module/Info/InfoUnitPerlTest.pm
M test/lib/pgBackRestTest/Module/Stanza/StanzaAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Storage/StorageFilterCipherBlockPerlTest.pm
M test/src/module/crypto/cipherBlockTest.c
M test/src/module/crypto/cryptoTest.c

Add cryptoError() and update crypto code to use it.

commit   : 2cb312ef5aa9f57da607229f9fe22ce5d7aa0352    
  
author   : David Steele <[email protected]>    
date     : Tue, 6 Nov 2018 19:16:00 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 6 Nov 2018 19:16:00 -0500    

Click here for diff

This adds detail to error messages when available and improves code coverage.  

M doc/xml/release.xml
M src/Makefile
M src/crypto/cipherBlock.c
M src/crypto/crypto.c
M src/crypto/crypto.h
M src/crypto/hash.c
M test/define.yaml
M test/src/module/crypto/cryptoTest.c

Add base variants to all integer to string conversion functions.

commit   : bef31f1802f231ce7b6ab06371148b74eb9eb6ab    
  
author   : David Steele <[email protected]>    
date     : Tue, 6 Nov 2018 18:32:44 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 6 Nov 2018 18:32:44 -0500    

Click here for diff

Previously these functions were fixed at base 10 conversion. Add variants that can convert from any base.  

M doc/xml/release.xml
M src/common/type/convert.c
M src/common/type/convert.h
M test/src/module/archive/commonTest.c
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeVariantTest.c

Ignore deleted files in rsync to test/repo.

commit   : 7de9584435839f8e7b20e8f336820b84a70cafd8    
  
author   : David Steele <[email protected]>    
date     : Sun, 4 Nov 2018 08:39:48 -0500    
  
committer: David Steele <[email protected]>    
date     : Sun, 4 Nov 2018 08:39:48 -0500    

Click here for diff

Deleted files are showing up in git ls-files (added 57d78092) but they don't actually exist on disk.  
  
If there is someway to exclude deleted files from ls-files then I can't find it, so just tell rsync to ignore missing files.  

M test/test.pl

Improve efficiency of code generation.

commit   : 57d780929741e3d09f60b252a26ad12fb645254e    
  
author   : David Steele <[email protected]>    
date     : Sat, 3 Nov 2018 19:52:46 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 3 Nov 2018 19:52:46 -0400    

Click here for diff

Code generation saved files even when they had not changed, which often caused code generation cascades. So, don't save files unless they have changed.  
  
Use rsync to determine which files have changed since the last test run. The manifest of changed files is saved and not removed until all code generation and builds have completed. If an error occurs the work will be redone on the next run.  
  
The eventual goal is to do all the builds from the test/repo directory created by rsync but for now it is only used to track changes.  

M .gitignore
M build/lib/pgBackRestBuild/Build.pm
M doc/xml/release.xml
M libc/build/lib/pgBackRestLibC/Build.pm
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/test.pl

Improve single test run performance.

commit   : 1f8931f73274163f27ba38aea378ea50488ba557    
  
author   : David Steele <[email protected]>    
date     : Sat, 3 Nov 2018 16:34:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 3 Nov 2018 16:34:04 -0400    

Click here for diff

Improve on 7794ab50 by including the build flag files directly into the Makefile as dependencies (even though they are not includes).  This simplifies some of the rsync logic and allows make to do what it does best.  
  
Also split build flag files into test, harness, and build to reduce rebuilds.  Test flags are used to build test.c, harness flags are used to build the rest of the files in the test harness, and build flags are used for the files that are not directly involved in testing.  

M test/Vagrantfile
M test/define.yaml
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/src/module/common/debugOnTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/exitTest.c

Preserve contents of C unit test build directory between test.pl executions.

commit   : 7794ab50dc839088fc1e1137edaaf65354faa76a    
  
author   : David Steele <[email protected]>    
date     : Fri, 2 Nov 2018 11:56:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 2 Nov 2018 11:56:13 -0400    

Click here for diff

The contents were already preserved between tests in a single test.pl run but for a separate execution the entire project had to be built from scratch, which was getting slower as we added code.  
  
Save the important build flags in a file so the new execution knows whether the build contents can be reused.  

M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/test.pl

Mount tmpfs in Vagrantfile instead test.pl.

commit   : 1a98bd41b48b2619b318639faeeb688481dbc07e    
  
author   : David Steele <[email protected]>    
date     : Fri, 2 Nov 2018 08:37:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 2 Nov 2018 08:37:27 -0400    

Click here for diff

Mounting/unmounting tmpfs on /home/[user]/test takes time, forces at least 3GB of memory to be available for tests, and makes it harder to preserve data between tests.  
  
Instead, move mounting of tmpfs to the Vagrantfile and add it to fstab so it survives reboots.  

M doc/xml/release.xml
M test/Vagrantfile
M test/test.pl

Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected.

commit   : 34c63276cd26cd5310169343bb8e17e323feef95    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 1 Nov 2018 11:31:25 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 1 Nov 2018 11:31:25 -0400    

Click here for diff

There are a number of cases where a checksum delta is more appropriate than the default time-based delta:  
  
* Timeline has switched since the prior backup  
* File timestamp is older than recorded in the prior backup  
* File size changed but timestamp did not  
* File timestamp is in the future compared to the start of the backup  
* Online option has changed since the prior backup  
  
A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch.  In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention.  
  
Now, a checksum delta will be performed if the backup type is incr/diff.  This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/lib/pgBackRestTest/Module/Backup/BackupUnitPerlTest.pm
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Retry all S3 5xx errors rather than just 500 internal errors.

commit   : cca7a4ffd477871d5eab323f1a5f621fbe48a568    
  
author   : David Steele <[email protected]>    
date     : Tue, 30 Oct 2018 16:45:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 30 Oct 2018 16:45:42 -0400    

Click here for diff

We were already retrying 500 errors but 503 (rate-limiting) errors were not being retried and would cause an instant failure which aborted the command.  
  
There are only two 5xx errors currently implemented by S3 but instead of adding 503 simply retry all 5xx errors. This is consistent with the http definition of this error class, "the server failed to fulfill an apparently valid request."  
  
Suggested by Craig A. James.  

M doc/xml/release.xml
M lib/pgBackRest/Storage/S3/Request.pm
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Storage/StorageS3RequestPerlTest.pm

Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.

commit   : 286f7e501154a05d147d146d25e3dd0ea0d49d5c    
  
author   : David Steele <[email protected]>    
date     : Sat, 27 Oct 2018 20:00:00 +0100    
  
committer: David Steele <[email protected]>    
date     : Sat, 27 Oct 2018 20:00:00 +0100    

Click here for diff

This calculation was missed when the WAL segment size was made dynamic in preparation for PostgreSQL 11.  
  
Fix the calculation by checking the actual WAL file sizes instead of using an estimate based on WAL segment size.  This is more accurate because it takes into account .history and .backup files, which are smaller.  Since the calculation is done in the async process the additional processing time should not adversely affect performance.  
  
Remove the PG_WAL_SIZE constant and instead use local constants where the old value is still required.  This is only the case for some tests and PostgreSQL 8.3 which does not provide a way to get the WAL segment size from pg_control.  

M doc/xml/release.xml
M lib/pgBackRest/Archive/Push/Push.pm
M lib/pgBackRest/Db.pm
M lib/pgBackRest/DbVersion.pm
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Module/Archive/ArchivePushPerlTest.pm

Fix issue with archive-push-queue-max not being honored on connection error.

commit   : 41b00dc204a04a87b337aa5bebe7cfd4032d5eae    
  
author   : David Steele <[email protected]>    
date     : Sat, 27 Oct 2018 16:57:57 +0100    
  
committer: David Steele <[email protected]>    
date     : Sat, 27 Oct 2018 16:57:57 +0100    

Click here for diff

If an error occurred while acquiring a lock on a remote server the error would be reported correctly, but the queue max detection code was not reached.  The tests failed to detect this because they fixed the connection before queue max, allowing the ccde to be reached.  
  
Move the queue max code before the lock so it will run even when remote connections are not working.  This means that no attempt will be made to transfer WAL once queue max has been exceeded, but it makes it much more likely that the code will be reach without error.  
  
Update tests to continue errors up to the point where queue max is exceeded.  
  
Reported by Lardière Sébastien.  

M doc/xml/release.xml
M lib/pgBackRest/Archive/Push/Async.pm
M src/perl/embed.auto.c
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-archive-stop-003.log
M test/expect/mock-archive-stop-004.log
M test/expect/mock-archive-stop-005.log
M test/expect/mock-archive-stop-006.log
M test/expect/mock-archive-stop-007.log
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Module/Archive/ArchivePushPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm

Fix error after log file open failure when processing should continue.

commit   : 03b9db9aa2e84bf6b23af8331712d1480520ca19    
  
author   : David Steele <[email protected]>    
date     : Thu, 25 Oct 2018 14:58:25 +0100    
  
committer: David Steele <[email protected]>    
date     : Thu, 25 Oct 2018 14:58:25 +0100    

Click here for diff

The C code was warning on failure and continuing but the Perl logging code was never updated with the same feature.  
  
Rather than add the feature to Perl, just disable file logging if the log file cannot be opened.  Log files are always opened by C first, so this will eliminate the error in Perl.  
  
Reported by vthriller.  

M doc/xml/release.xml
M src/command/archive/get/get.c
M src/command/archive/push/push.c
M src/common/log.c
M src/common/log.h
M src/config/load.c
M src/config/load.h
M test/define.yaml
M test/src/module/common/logTest.c
M test/src/module/config/loadTest.c

Change infoArchiveCheckPg() to display the PostgreSQL version as a string in errors.

commit   : d301720c58145910189829e8b2ceacccd3543c1e    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 25 Oct 2018 10:48:37 +0100    
  
committer: David Steele <[email protected]>    
date     : Thu, 25 Oct 2018 10:48:37 +0100    

Click here for diff

Previously an integer representation (e.g. 90400) would be displayed which is correct but not as friendly to the user.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/Makefile
M src/info/infoArchive.c
M test/src/module/info/infoArchiveTest.c

Add tests for InfoPg history fixes introduced in 070455ce.

commit   : 5590bfa84c0a91d1e93ed03441e12c4dab1078c5    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 19 Oct 2018 18:31:05 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Oct 2018 18:31:05 +0200    

Click here for diff

The existing tests were not adequate to ensure the history was being added in the correct order when some entries were loaded from a file and others added with infoPgAdd().  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/info/info.c
M src/info/infoPg.c
M test/src/module/info/infoPgTest.c

Correct current history item in InfoPg to always be in position 0.

commit   : 070455ce44504c63a5bf2f70093c64ed278fa12e    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Oct 2018 12:31:56 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Oct 2018 12:31:56 +0200    

Click here for diff

The InfoPg object was partially modified in 960ad732 to place the current history item in position 0, but infoPgDataCurrent() didn't get updated correctly.  
  
Remove this->indexCurrent and make the current position always equal 0.  Use the new lstInsert() function when adding new history items via infoPgAdd(), but continue to use lstAdd() when loading from a file for efficiency.  
  
This does not appear to be a live bug because infoPgDataCurrent() and infoPgAdd() are not yet used in any production code.  The archive-get command is the only C code using InfoPG and it always looks at the entire list of items rather than just the current item.  
  
Suggested by Cynthia Shang.  

M doc/xml/release.xml
M src/info/infoPg.c
M src/info/infoPg.h
M test/expect/mock-archive-001.log
M test/expect/mock-stanza-001.log
M test/src/module/info/infoPgTest.c

Add lstInsert() to List object.

commit   : f345db3f7ca15480c5aa70d015150b399f3ce33a    
  
author   : David Steele <[email protected]>    
date     : Fri, 19 Oct 2018 11:52:17 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Oct 2018 11:52:17 +0200    

Click here for diff

Add general purpose insert function and make lstAdd() a special insert case.  

M doc/xml/release.xml
M src/common/type/list.c
M src/common/type/list.h
M test/src/module/common/typeListTest.c

Correct cfgDefDataFind() to use UINTP instead of VOIDPP.

commit   : 09a1ad2c10fa17e165b88731322f4dcf8de73ecc    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 19 Oct 2018 10:51:33 +0200    
  
committer: David Steele <[email protected]>    
date     : Fri, 19 Oct 2018 10:51:33 +0200    

Click here for diff

Also add the UINTP macro.  
  
Contributed by Cynthia Shang.  

M doc/xml/release.xml
M src/common/debug.h
M src/config/define.c

Begin v2.07 development.

commit   : 06d68eada00e6ec57c662aa9ad2bacdc86eb5861    
  
author   : David Steele <[email protected]>    
date     : Tue, 16 Oct 2018 17:21:01 +0100    
  
committer: David Steele <[email protected]>    
date     : Tue, 16 Oct 2018 17:21:01 +0100    

Click here for diff

M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h