pgBackRest 2.26 commit log

v2.25: LZ4 Compression Support

commit   : fd3dca036b1436564dec17b7c23b62b9d9fa1051    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 17:14:53 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 17:14:53 -0400    

Click here for diff

Features:  
  
* Add lz4 compression support. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)  
* Add --dry-run option to the expire command. Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything. (Contributed by Cynthia Shang, Luca Ferrari.)  
  
Improvements:  
  
* Improve performance of remote manifest build. (Suggested by Jens Wilke.)  
* Fix detection of keepalive options on Linux. (Contributed by Marc Cousin.)  
* Add configure host detection to set standards flags correctly. (Contributed by Marc Cousin.)  
* Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options. (Reviewed by Cynthia Shang.)  
* Limit backup file copy size to size reported at backup start. If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data. (Reviewed by Cynthia Shang.)  

M README.md
M doc/release.pl
M doc/resource/exe.cache
M doc/resource/git-history.cache
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h
M test/code-count/file-type.yaml

Fix race condition in real/all integration tests.

commit   : 8af0462c5d749ca798bc524e7a05d0e24f4ed0c7    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 15:30:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 15:30:59 -0400    

Click here for diff

If the tests are running quickly then the time target might end up the same as the end time of the prior full backup. That means restore auto-select will not pick it as a candidate and restore the last backup instead causing the restore compare to fail.  
  
So, sleep one second.  

M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Improve and centralize backup dependency calculation.

commit   : 86f71349efdc5970b9985c0155a65c2e3b193a04    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 26 Mar 2020 14:05:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 14:05:40 -0400    

Click here for diff

Add functions to select a current backup by label and to retrieve a backup dependency list for any given backup.  
  
Update the expire code to utilize the new functions and to expire backup sets from newest dependency to oldest.  

M src/command/expire/expire.c
M src/info/infoBackup.c
M src/info/infoBackup.h
M test/define.yaml
M test/src/module/command/expireTest.c
M test/src/module/info/infoBackupTest.c

Order commands alphabetically in definition of set option.

commit   : 048a36ea390334ddc019e958242c4d4ca0f9096f    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 26 Mar 2020 13:04:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 13:04:59 -0400    

Click here for diff

M build/lib/pgBackRestBuild/Config/Data.pm

Debug and optimize flag cleanup for unit tests.

commit   : 2c2bc8a32ba6e91e2a0241cc2d93d9809ab6c068    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 12:16:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 12:16:22 -0400    

Click here for diff

Decisions about when to optimize or enable debug code were spread out in too many places making it hard to keep them consistent.  
  
Centralize the logic as much as possible to make it easier to maintain.  

M .travis.yml

Debug and optimize flag cleanup for unit tests.

commit   : e63fdfbfd24bb1893ed5c0abb9aed72d2a230d56    
  
author   : David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 11:16:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 26 Mar 2020 11:16:35 -0400    

Click here for diff

Decisions about when to optimize or enable debug code were spread out in too many places making it hard to keep them consistent.  
  
Centralize the logic as much as possible to make it easier to maintain.  

M test/lib/pgBackRestTest/Common/JobTest.pm

Rough script to add tables to the user guide for performance tests.

commit   : d242114dc0feec9e87932e810ec3c1027752949b    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 19:00:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 19:00:09 -0400    

Click here for diff

This really needs to be worked on but for now at least preserve to code to be improved later.  

M doc/xml/user-guide.xml

Improve performance of jsonToStrInternal() and jsonFromStrInternal().

commit   : 10b4b9af8712c558ad1d3d1674dd494716fa1cb9    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 18:52:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 18:52:20 -0400    

Click here for diff

Most strings do not contain escape/d characters, so optimize the path where the string contains few or no escape/d characters.  
  
This means far fewer calls to strCatChr() in favor of strCatZN(), which is much more efficient.  

M src/common/type/json.c

Add srtCatZN().

commit   : 88d7ee621547b28113af9c00712267642729b7ec    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 18:37:35 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 18:37:35 -0400    

Click here for diff

Append N characters from a zero-terminated string.  
  
Note that the string does not actually need to be zero-terminated as long as N is <= the end of the string being concatenated.  

M src/common/type/string.c
M src/common/type/string.h
M test/src/module/common/typeStringTest.c

Use conditional in strPtr() to encourage inlining.

commit   : b10270eee87ebaa1c595d0ba486a61fefabd8d6d    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 16:03:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 16:03:33 -0400    

Click here for diff

This construct is logically equivalent but it seems to make the compiler more likely to inline the function, probably because the compiled code is slightly smaller.  

M src/common/type/string.c

Fix backup-prior for diff backups in mock/expire integration test.

commit   : eb7f7dd5caf9f48368b6c24b401f9094ad85a95e    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 25 Mar 2020 15:47:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 15:47:03 -0400    

Click here for diff

In the ExpireEnvTest.pm backupCreate() function, backup-prior was incorrectly set for diff backups to the previous backup regardless of what backup type the previous backup was. This did not cause any issues in the Mock Expire tests before because it was not being checked. However, in order to reduce churn in the expect logs for a new feature where the backup-prior is utilized, this is being fixed so that the full backup is always used as backup-prior.  

M test/expect/mock-expire-001.log
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm

Remove backup after timeline switch from user guide.

commit   : 4c31a922a3b258b075c095b92d637eb2d367d7f2    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 12:17:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 12:17:20 -0400    

Click here for diff

It is no longer necessary (since 2.07, 34c63276) to do a backup manually after a timeline switch.  

M doc/xml/user-guide.xml

More refactoring of command/expire unit test module.

commit   : 480a11066db83ecb57d2ca3f23924a93d8590b5e    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 25 Mar 2020 12:04:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 12:04:23 -0400    

Click here for diff

Buffer cleanup following on from e170c53e.  

M test/src/module/command/expireTest.c

Allocate a minimum amount of space when a string is likely to grow.

commit   : d20115d29e0b33062fce5dfc8285c0f884ef4246    
  
author   : David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 09:12:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 25 Mar 2020 09:12:51 -0400    

Click here for diff

This helps prevent excessive memory reallocation.  

M src/common/type/string.c
M src/common/type/string.h
M test/src/module/common/typeStringTest.c

Improve performance of memResize().

commit   : ad4187eb9f5ff3cfca20537dd6d998d1afa4a96b    
  
author   : David Steele <[email protected]>    
date     : Tue, 24 Mar 2020 19:08:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 24 Mar 2020 19:08:00 -0400    

Click here for diff

The major bottleneck was finding the memory allocation to be resized since it required a sequential search through a list.  
  
Instead, put the allocation header at the beginning of the allocation and return an offset to the user for their buffer. This allows us to use pointer arithmetic to get back to the allocation header quickly when resizing. A side effect is to make memFree() faster as well. The downside is we won't detect garbage pointers passed to memResize()/memFree(), which is also true for MemContext pointers.  
  
The performance benefits can be pretty large in certain cases, in particular when loading and saving manifests. The following are the before and after performance tests on a 900K file manifest.  
  
Before:  
  
run 003 - manifestNewLoad()/manifestSave()  
    000.000s l0125 - generate manifest  
    183.411s l0236 -     101.2MB manifest generated with 900000 files  
    183.411s l0239 - load manifest  
    403.816s l0243 -     completed in 220405ms  
    403.816s l0245 -        check file total  
    403.816s l0248 - save manifest  
    670.217s l0253 -     completed in 266401ms  
    670.217s l0256 - find all files  
    671.263s l0266 -     completed in 1046ms  
  
After:  
  
run 003 - manifestNewLoad()/manifestSave()  
    000.000s l0125 - generate manifest  
    007.730s l0236 -     101.2MB manifest generated with 900000 files  
    007.730s l0239 - load manifest  
    033.431s l0243 -     completed in 25701ms  
    033.431s l0245 -        check file total  
    033.431s l0248 - save manifest  
    057.755s l0253 -     completed in 24324ms  
    057.755s l0256 - find all files  
    058.689s l0266 -     completed in 934ms  

M src/common/memContext.c
M test/src/module/common/memContextTest.c

Free temp mem context periodically in storageRemoteInfoList().

commit   : 98e5affa8aa1b1f7076e2d389d3afdf71dd80644    
  
author   : David Steele <[email protected]>    
date     : Tue, 24 Mar 2020 18:44:13 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 24 Mar 2020 18:44:13 -0400    

Click here for diff

Other storage*InfoList() functions do this but it was missed here.  
  
memResize()/memFree() operations become more expensive as the mem context grows larger so freeing it periodically saves processing time.  

M doc/xml/release.xml
M src/storage/remote/storage.c

Refactor command/expire unit test module.

commit   : e170c53e7efb84dc65e3a0c0b8e4ed3c060030d8    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 23 Mar 2020 14:31:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Mar 2020 14:31:04 -0400    

Click here for diff

Add titles and use a Buffer to store backup.info instead of a String.  

M test/define.yaml
M test/src/module/command/expireTest.c

More improvements to custom coverage report.

commit   : f9c86b11a54e5c37ad98afd5a2de76cd96ae1061    
  
author   : David Steele <[email protected]>    
date     : Mon, 23 Mar 2020 12:17:34 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 23 Mar 2020 12:17:34 -0400    

Click here for diff

* Fix a few issues with file names being truncated introduced in 787d3fd6.  
  
* Use function line info from the lcov file to calculate which lines to show for uncovered functions.  This is more accurate than what we were doing before and function comment headers are now excluded which reduces clutter in the report.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/test.pl

Implement TEST_RESULT_*() macros with functions, mostly.

commit   : dbb1248bfbb1d469bae030084e6672772702f89d    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 20:44:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 20:44:51 -0400    

Click here for diff

The prior macros had grown over time to be pretty significant pieces of code that required a lot of compile time, though runtime was efficient.  
  
Move most of the macro code into functions to reduce compile time, perhaps at a slight expense to runtime.  The overall performance benefit is 10-15% so this seems like a good tradeoff.  
  
Add TEST_RESULT_UINT_INT() to safely compare uint to int with range checking.  

M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h
M test/src/module/common/cryptoTest.c
M test/src/test.c

Fix incorrect result types in unit tests.

commit   : d6ffa9ea6d45fcae46abd79a5f6d6123e79c9168    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 20:25:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 20:25:31 -0400    

Click here for diff

Upcoming changes to the TEST_RESULT_* macros are more type safe and identified that the wrong macros were being used to test results in many cases.  
  
Commit these changes separately to verify that they work with the current macro versions.  
  
Note that no core bugs were exposed by these changes.  

M test/src/common/harnessTest.h
M test/src/module/command/archiveGetTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/encodeTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/ioTlsTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/stackTraceTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeJsonTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/waitTest.c
M test/src/module/config/configTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/info/infoArchiveTest.c
M test/src/module/info/infoBackupTest.c
M test/src/module/info/infoPgTest.c
M test/src/module/info/manifestTest.c
M test/src/module/postgres/interfaceTest.c
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c
M test/src/module/storage/s3Test.c

Remove TRY...CATCH blocks from TEST_RESULT*() macros.

commit   : e1da6a45e0e7f5223ac5210b2485f7946421a723    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 16:14:33 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 16:14:33 -0400    

Click here for diff

TRY...CATCH blocks are fairly expensive and when all the TEST_RESULT*() macros succeed they are not needed.  
  
Instead just record info at the start of the result test so a detailed exception can be thrown in test.c in the rare case where an exception occurs.  

M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h
M test/src/module/common/regExpTest.c
M test/src/test.c

Add test function to set stack trace function line number.

commit   : 5a8758cc8a49b88f731bb33d81579115cb1ea3c3    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 16:04:24 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 16:04:24 -0400    

Click here for diff

This is helpful for test macros that know the line number.  
  
The line number can now be non-zero below the top of the stack without WITH_BACKTRACE so instead ignore the line number for output when it is zero.  

M src/common/stackTrace.c
M src/common/stackTrace.h
M test/src/module/common/stackTraceTest.c

Add TestError for testing.

commit   : bd461a18997dd38443004eb962e37d348bcef03a    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 14:40:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 14:40:36 -0400    

Click here for diff

This error type distinguishes test errors with detailed debug info from normal errors which may need, for example, a stack trace to be added for context.  

M src/common/error.c
M src/common/error.h

Fix missing parameter in common/stack-trace test module.

commit   : c2df383aec3960fcc8046f93089114de8043cb14    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 14:24:28 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 14:24:28 -0400    

Click here for diff

This was passing since we don't test WITH_BACKTRACE in CI because it is used only for test builds.  
  
Ideally we would test this but it doesn't seem worth the trouble at the moment.  

M test/src/module/common/stackTraceTest.c

Fix space lost in d70ca259.

commit   : 8c76ea2d93926cb3c714490884fb42966a859cf2    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 14:18:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 14:18:16 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/JobTest.pm

Use --clean-only for reproducible builds in contributing documentation.

commit   : 4c831d8e83f4ffda02023351c17cb3072b5b6e10    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 13:50:31 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 13:50:31 -0400    

Click here for diff

If the work or result directories already contain data then the docs might be generated slightly differently.  Doing a clean ensures they will always produce the same output (provided the code does not change).  

M doc/xml/contributing.xml

Add --clean and --clean-only options to test.pl.

commit   : 06a3f82e912b549add76b65f51075bc389f4e29c    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 13:46:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 13:46:30 -0400    

Click here for diff

M test/test.pl

Suppress timestamps in contributing documentation.

commit   : d70ca2592f7df7d9a0adc2b91fd03c8cafc757b0    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 10:17:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 10:17:03 -0400    

Click here for diff

Timestamps cause a lot of churn and make it hard to tell if important changes are working.  

M CONTRIBUTING.md
M doc/xml/contributing.xml

Allow suppression of times in testing for reproducibility.

commit   : 3ec630f5b2008af92d1baab36afda4d20b8329a8    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 10:12:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 10:12:29 -0400    

Click here for diff

Timestamps, timings, etc. cause a lot of log churn when included in documentation.  

M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/common/harnessTest.intern.h
M test/src/test.c
M test/test.pl

Do variable replacements for cmd-extra in documentation.

commit   : a1802510111297b479c6150631df92bc6b065613    
  
author   : David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 09:59:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 22 Mar 2020 09:59:38 -0400    

Click here for diff

M doc/lib/pgBackRestDoc/Common/DocExecute.pm

Remove custom lcov build from contributing documentation.

commit   : 6a9fbccf7603bc1ae9626ff387013933271a1c05    
  
author   : David Steele <[email protected]>    
date     : Sat, 21 Mar 2020 19:08:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 21 Mar 2020 19:08:27 -0400    

Click here for diff

Using the stock lcov no longer seems to be a problem, probably due to a better implementation in CoverageTest.pm.  

M CONTRIBUTING.md
M doc/xml/contributing.xml

Build contributing documentation on Travis CI.

commit   : 56fb39937368463474c96307454051a66c1742c4    
  
author   : David Steele <[email protected]>    
date     : Sat, 21 Mar 2020 18:45:58 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 21 Mar 2020 18:45:58 -0400    

Click here for diff

Building the contributing document has some special requirements because it runs Docker in Docker so the repo path must align on the host and all Docker containers.  Run `pgbackrest/doc/doc.pl` from within the home directory of the user that will do the doc build, e.g. `home/vagrant`.  If the repo is not located directly in the home directory, e.g. `/home/vagrant/pgbackrest`, then a symlink may be used, e.g. `ln -s /path/to/repo /home/vagrant/pgbackrest`.  
  
Mount the repo in the Vagrantfile at /home/vagrant/pgbackrest but provide a link from the old location at /backrest to make the transition less painful.  

M doc/manifest.xml
M doc/xml/contributing.xml
M test/Vagrantfile
M test/travis.pl

Add missing package to contributing documentation.

commit   : ee2e15bf5524cf8bac3454d409dee250792411d9    
  
author   : David Steele <[email protected]>    
date     : Sat, 21 Mar 2020 09:51:12 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 21 Mar 2020 09:51:12 -0400    

Click here for diff

M CONTRIBUTING.md
M doc/xml/contributing.xml

Don't list directories as changed from the last run.

commit   : f405c82dcc99da8e8da2fc19b8e13f2dd9b542d2    
  
author   : David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 15:00:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 15:00:20 -0400    

Click here for diff

It's not very useful information and just clutters the list.  

M test/test.pl

Enable coverage testing on Fedora 30.

commit   : cfab67a775e0e1eb80e15df2fb5b1acc4d911432    
  
author   : David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 13:49:23 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 13:49:23 -0400    

Click here for diff

Now that coverage testing works reliably with gcc9 it makes sense to enable it for CI.  

M test/lib/pgBackRestTest/Common/VmTest.pm

Remove old coverage data before starting new test.

commit   : 782c9f89f47fcb8bac75b21c241c1f34d744c13c    
  
author   : David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 13:43:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 13:43:08 -0400    

Click here for diff

The old coverage data has been recorded so it is no longer needed. In newer versions of gcc leaving this file around can lead to an error when writing profile data after forking off to a non-pgbackrest binary (which we do in some unit tests).  

M test/lib/pgBackRestTest/Common/JobTest.pm

Improve custom coverage report.

commit   : 787d3fd67b3549b9b04a8053e751cda9494d8df5    
  
author   : David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 12:54:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 12:54:29 -0400    

Click here for diff

* Show all uncovered branch parts even when there are more than two parts per branch. This is the way gcc9 reports coverage so it needs to work even if it doesn't make as much sense as the old way.  
  
* Show covered branches in functions where coverage is missing.  Showing just the uncovered branches can be confusing because it's not always clear how the coverage relates to the code.  By showing all branch coverage (+ or -) this correspondence is made easier.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm

Fix lcov report when test module missing coverage.

commit   : 8af802900657174e1b24a1bdf9992c9fa78909d9    
  
author   : David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 09:16:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 20 Mar 2020 09:16:38 -0400    

Click here for diff

We don't report branch coverage on test modules (e.g. test/src/module/common/errorTest.c) but the code that excluded branch coverage from the test module would also exclude it from all core modules if the test module was included in the lcov report due to lack of function/line coverage.  
  
Adjust the coverage code to only exclude branches during the extraction of test module coverage.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm

Remove obsolete -O2 option for Fedora 30 unit test builds.

commit   : f6e9bb081963932b4a35e6fd6c01ac2a523195f1    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 19:30:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 19:30:09 -0400    

Click here for diff

For some reason gcc9 would not do -O0 builds in combination with one of the options that libperl required.  Now that libperl is gone this exception is no longer required.  

M test/lib/pgBackRestTest/Common/JobTest.pm

Remove obsolete deletes from Vagrantfile.

commit   : 2241524c0bdc92e1a04ac18e91a188a2bc1bcbdf    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 18:34:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 18:34:10 -0400    

Click here for diff

pgBackRest no longer writes files into the .vagrant path so there's no longer anything to delete.  

M test/Vagrantfile

Limit backup file copy size to size reported at backup start.

commit   : dcddf3a58bc6c4a8fbc9ac2c000cd5ff7b669cb5    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 13:16:05 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 13:16:05 -0400    

Click here for diff

If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data.  
  
This also reduces the likelihood of seeing torn pages during the copy. Torn pages can still occur in the middle of the file, though, so they must be handled.  

M doc/xml/release.xml
M src/command/backup/file.c
M test/src/module/command/backupTest.c

Clarifications of what limit == NULL or UINT64_MAX means.

commit   : e5bcc0c47e9db98ff99366a1e5806c1dcd6ebadd    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 12:34:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 12:34:27 -0400    

Click here for diff

M doc/xml/release.xml
M src/storage/posix/read.c
M src/storage/read.h
M src/storage/read.intern.h
M src/storage/storage.h
M src/storage/storage.intern.h

Fix typo.

commit   : 73315268fd9d405eee7da019014e8117c6a20693    
  
author   : Cynthia Shang <[email protected]>    
date     : Thu, 19 Mar 2020 12:11:20 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 12:11:20 -0400    

Click here for diff

M test/src/module/command/backupTest.c

Move coverage code to CoverageTest module.

commit   : d677b0708126302e3b01e24664866b29913b1c1d    
  
author   : David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 12:07:51 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 19 Mar 2020 12:07:51 -0400    

Click here for diff

This code needs some work, which will be easier if it is all in one place.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/test.pl

Updates to contributing documentation.

commit   : f538da0571e9e6b224b208fa5f2b3bc3025f5f02    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 18 Mar 2020 17:30:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Mar 2020 17:30:36 -0400    

Click here for diff

M CONTRIBUTING.md
M doc/xml/contributing.xml

Improve testing of files that change size during the backup.

commit   : 26c89b2c8c9ea362b9bd075cc4d541dc515af28b    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Mar 2020 13:40:16 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Mar 2020 13:40:16 -0400    

Click here for diff

Files can change size during a backup so update and add tests to cover the various scenarios more thoroughly.  

M test/src/module/command/backupTest.c

Added redacted manifest to testBackupValidate().

commit   : 4ec04e5163a361f88e3f2e09bdc22465a0868982    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Mar 2020 10:10:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Mar 2020 10:10:10 -0400    

Click here for diff

The manifest is excellent for validation but including the entire manifest is too noisy and some values are architecture/algorithm dependent.  
  
Output a redacted version that contains the most important information which can be improved on over time.  

M test/define.yaml
M test/src/module/command/backupTest.c

Add TEST_RESULT_STR_Z_KEYRPL() test macro.

commit   : b8cd1b6790b85f3ac36ae3fcb024be3997ac6ad1    
  
author   : David Steele <[email protected]>    
date     : Wed, 18 Mar 2020 10:05:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 18 Mar 2020 10:05:08 -0400    

Click here for diff

This macro will automatically do key replacement before the comparison. This saves the indentation required for an embedded function call.  
  
Possibly TEST_RESULT_Z_KEYRPL() would also be useful but it will be added when needed.  

M test/src/common/harnessTest.h

Allow storage reads to be limited by bytes.

commit   : f2548f45ce4c1e444e9d3175b2349e0a97b2783a    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Mar 2020 18:16:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Mar 2020 18:16:17 -0400    

Click here for diff

The current use case is reading files from the PostgreSQL cluster during backup.  
  
A file may grow during backup but we only need to copy the number of bytes that were reported during the manifest build.  The rest will be rebuilt from the WAL during recovery so copying more is just a waste of space.  
  
Limiting the copy sizes in backup will be part of a future commit.  

M src/storage/posix/read.c
M src/storage/posix/read.h
M src/storage/posix/storage.c
M src/storage/read.c
M src/storage/read.h
M src/storage/read.intern.h
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/read.h
M src/storage/remote/storage.c
M src/storage/storage.c
M src/storage/storage.h
M src/storage/storage.intern.h
M test/src/module/storage/posixTest.c
M test/src/module/storage/remoteTest.c

Test that shrunk file is backed up correctly.

commit   : 307e741298cd068e68bcbe8994c871a989bdff01    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Mar 2020 16:01:17 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Mar 2020 16:01:17 -0400    

Click here for diff

It's possible, though rare, for a file to shrink during a backup.  
  
There was no issue with the code but having a test is always a good idea.  

M test/src/module/command/backupTest.c

Fix typos.

commit   : 7f1817c8adea02e58af3ae82044e8e91e6f2c4f4    
  
author   : David Steele <[email protected]>    
date     : Tue, 17 Mar 2020 11:43:38 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 17 Mar 2020 11:43:38 -0400    

Click here for diff

M src/common/type/variant.c
M src/common/type/variant.h
M src/storage/storage.h

commit   : 9a47b88da3effe24b835908c12ce5dd5abf2304d    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 20:02:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 20:02:36 -0400    

Click here for diff

When multiple files were missing coverage it could be hard to locate the coverage report for a specific file.  
  
Add links for uncovered files to make this easier.  
  
Also move table titles out of the table so they are valid html.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm

Make cfgLoadLogSetting() static.

commit   : d6cea422a01b6645d78aebe0e547d4fd1688001b    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 18:18:30 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 18:18:30 -0400    

Click here for diff

This function was once required by the test harness.  

M src/config/load.c
M src/config/load.h

Reduce variables extern'd by the common/log module in debug builds.

commit   : f7dac144a6393736e16f44d4103ec4d4ff6133dc    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 18:16:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 18:16:27 -0400    

Click here for diff

These days it is better to include the module in define.yaml when we need to poke at the internal implementation.  
  
This doesn't quite work for the log test harness, so for now some variables will need to remain extern'd in debug builds.  

M src/common/log.c
M test/define.yaml
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c

Forbid access to /tmp/pgbackrest in the Vagrantfile.

commit   : 3fbfcba811499a59b943dd893eef9ff52090f7b6    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 17:27:01 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 17:27:01 -0400    

Click here for diff

This matches the error that will be thrown in the vm=none test on Travis CI if a unit test writes to /tmp/pgbackrest.  

M test/Vagrantfile

Make storage and logging dry-run aware.

commit   : 46911c64c19dc8ff76d2a6ff4548fc01647043e0    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 17:24:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 17:24:21 -0400    

Click here for diff

Enhance dry-run support added in 2fa69af8 by forbidding writes in the storage layer and adding prefixes to log messages.  
  
The former will protect against mistakes in dry-run implementations and the latter will make it clear when a command was executed in dry-run mode.  
  
Update expire unit tests with the new log prefix.  

M src/common/log.c
M src/common/log.h
M src/config/load.c
M src/config/parse.c
M src/storage/helper.c
M src/storage/helper.h
M test/define.yaml
M test/src/common/harnessConfig.c
M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/module/command/expireTest.c
M test/src/module/common/logTest.c
M test/src/module/config/loadTest.c
M test/src/module/storage/posixTest.c

Add --dry-run option to the expire command.

commit   : 2fa69af8da4075e82359a8585ecde2f2030db546    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 16 Mar 2020 13:56:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 13:56:52 -0400    

Click here for diff

Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/command/expire/expire.c
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
A test/lib/pgBackRestTest/LibCAuto.pm
M test/src/module/command/expireTest.c

Fix indentation.

commit   : 4608ca473e6d8c92762eee08169b50072bec0821    
  
author   : Cynthia Shang <[email protected]>    
date     : Mon, 16 Mar 2020 12:50:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 12:50:37 -0400    

Click here for diff

M src/command/expire/expire.c

Move raw coverage results to test/result/raw path.

commit   : 4328bc1ac64726e810dfa76d0d1149ded6c93770    
  
author   : David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 08:41:32 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 16 Mar 2020 08:41:32 -0400    

Click here for diff

These results were stored in the vagrant path along with a full copy of src.  
  
Instead store the raw coverage data in test/result/raw and change source references to the files that already exist in [test-path]/repo.  

M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/test.pl

Build binaries in the test path rather than the vagrant path.

commit   : d7022495075682b35441a31e22606bdcffdb9b86    
  
author   : David Steele <[email protected]>    
date     : Sun, 15 Mar 2020 10:09:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 15 Mar 2020 10:09:27 -0400    

Click here for diff

It makes more sense to build in the test path since many developers won't have a vagrant path. Anyway, it's better not to modify the vagrant path since it belongs to vagrant.  
  
Instead of installing the binary just mount it into the container from where it was built. This saves a bit of time and space.  

M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/test.pl

Improve stability of command/check test module.

commit   : 19d975346b2688aa1a7b9e2dc68baecb9f6e7af2    
  
author   : David Steele <[email protected]>    
date     : Sun, 15 Mar 2020 09:59:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 15 Mar 2020 09:59:22 -0400    

Click here for diff

When pgbackrest was present this test behaved unexpectedly.  
  
While the binary is not currently required for this test is might be in the future so fix the test to prevent a regression.  

M test/src/module/command/checkTest.c

Add VPATH to Makefile.

commit   : da89d0ad6157487f820632cbc7109c8a91f56cc9    
  
author   : David Steele <[email protected]>    
date     : Sun, 15 Mar 2020 09:53:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Sun, 15 Mar 2020 09:53:42 -0400    

Click here for diff

This allows builds in a separate directory without copying the source.  

M src/Makefile.in

Update code classification and remove XS definition.

commit   : 959dce569b150cf221f2f38a3c4f438964ca324e    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 18:26:14 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 18:26:14 -0400    

Click here for diff

D test/code-count/code.def
M test/lib/pgBackRestTest/Common/CodeCountTest.pm

Move docker files to test/result.

commit   : 213cc6e8becd9974351cb28d08bef01184bf6ad3    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 15:40:37 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 15:40:37 -0400    

Click here for diff

M test/lib/pgBackRestTest/Common/ContainerTest.pm

Move coverage results to test/result.

commit   : 6827e248cd3ce9b92ef4bbddb79dfe0b32c1a6ef    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 15:29:42 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 15:29:42 -0400    

Click here for diff

M test/.gitignore
M test/test.pl

Move profile results to test/result.

commit   : 75ff25f17fda6e3198aa8e227bc535dc5d8fb969    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 14:50:36 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 14:50:36 -0400    

Click here for diff

M test/.gitignore
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/test.pl

Build packages on demand only and change build path.

commit   : 0f7fe55f728345426da2514e2c866678921114a6    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 14:35:09 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 14:35:09 -0400    

Click here for diff

Building packages is not a normal part of development so don't build packages by default. Instead build them in CI as needed.  
  
Do the builds in test/result instead of .vagrant to be friendlier with hosts that are not running vagrant. Anyway, it's probably not a good idea to be creating files in the .vagrant path.  

M .travis.yml
M test/.gitignore
M test/test.pl

Add comments to test/.gitignore.

commit   : 5645c91ed5cb48f0dbcb92439405dcefe7d1156a    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 14:18:22 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 14:18:22 -0400    

Click here for diff

M test/.gitignore

Auto-detect presence of clobbered warning.

commit   : b4c94e42b903059c044128710ef9e61ff5f1aa67    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 13:06:21 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 13:06:21 -0400    

Click here for diff

Not all compilers (e.g. clang) support this warning so only disable it (i.e. -Wno-clobbered) when supported.  

M src/build/aclocal.m4
M src/build/configure.ac
M src/configure

Generate src/build/aclocal.m4 automatically.

commit   : 4cd060b7fe6e2f9cc7ca8f8c128d1b5c1de9c137    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 12:48:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 12:48:08 -0400    

Click here for diff

This file is required when macros from the autoconf archive are used in configure.ac  

M CONTRIBUTING.md
M doc/xml/contributing.xml
A src/build/aclocal.m4
M test/Vagrantfile
M test/test.pl

Use a checksum to build configure.ac more efficiently.

commit   : 9e80c5710e8e8005dee553b4583dc062dfc3ca62    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 12:39:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 12:39:29 -0400    

Click here for diff

Building the configure.ac script can take multiple seconds depending on the state of the autoconf cache. Use a checksum to only rebuild when configure.ac has changed no matter how the timestamps have changed.  

M src/.gitignore
M src/configure
M test/test.pl

Remove obsolete ignore.

commit   : 748f9502eb06b1315551b5967eb5c6255fe9e072    
  
author   : David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 10:04:49 -0400    
  
committer: David Steele <[email protected]>    
date     : Sat, 14 Mar 2020 10:04:49 -0400    

Click here for diff

M test/.gitignore

Configure and make improvements.

commit   : 237a3da4d69bf068652c6f5cc805c313f725b129    
  
author   : David Steele <[email protected]>    
date     : Fri, 13 Mar 2020 09:07:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Fri, 13 Mar 2020 09:07:57 -0400    

Click here for diff

Configure:  
  
* Use standard make variables, e.g. CFLAGS, rather than our own, e.g. CINCLUDE  
* Add PG_CONFIG var for configuring custom pg_config location  
* Don't error if xml_config or pg_config is missing (but error if libs/headers not found)  
* Check for zlib.h header  
* Check for lz4frame.h header when liblz4 is present  
  
Make:  
  
* Use gcc-style auto dependencies  
* Put src list at the top since it is most frequently modified  
* Add clean-all target to also remove auto-generated config files  

M src/.gitignore
M src/Makefile.in
M src/build/configure.ac
M src/configure
M test/patch/rhel-package.patch
M test/test.pl

Remove obsolete ignore.

commit   : 810b2a5265888a7ba46c85cee52ca7b7b269608d    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 13:18:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 13:18:08 -0400    

Click here for diff

M src/.gitignore

Fix detection of keepalive options on Linux.

commit   : 4b24a74afb5c3ed1f176287070a3b1f58fa4b479    
  
author   : Marc Cousin <[email protected]>    
date     : Thu, 12 Mar 2020 11:38:50 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 11:38:50 -0400    

Click here for diff

This code stanza was not being included on Linux platforms because of a missing header file.  
  
Also update the order of operations and make the timeout calculations more sensible.  

M doc/xml/release.xml
M src/common/io/tls/client.c

Fix missing cast.

commit   : e9c1569d8e2b55762784b8c74c1092b4ad3cadf9    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 10:48:40 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 10:48:40 -0400    

Click here for diff

This was a problem in testing when -Wconversion was enabled and TCP_KEEPIDLE was defined.  

M src/common/io/tls/client.c

Add configure host detection to set standards flags correctly.

commit   : d6ef6c57c336d56561ce84e63a75880e293e0b8f    
  
author   : Marc Cousin <[email protected]>    
date     : Thu, 12 Mar 2020 10:37:19 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 10:37:19 -0400    

Click here for diff

Linux and MacOS were tolerant of having all standards flags set but BSD was not.  
  
Instead detect the host type and set standards flags as appropriate.  

M doc/xml/release.xml
M src/Makefile.in
M src/build/configure.ac
M src/configure

Add scripts required for host detection by src/configure.

commit   : fa1f63198d7af2ae7c3a7d26af972f81de15d215    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 10:03:57 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 10:03:57 -0400    

Click here for diff

Add to the src/build directory so they will be out of the way.  
  
Also add instructions for updating the scripts to RELEASE.md.  

M doc/RELEASE.md
A src/build/config.guess
A src/build/config.sub
M src/build/configure.ac
A src/build/install-sh
M src/configure

Add separators to make configure.ac easier to read.

commit   : c3b6bd2e831c4f29fdfb69d36fe6d21da1a322be    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 09:50:10 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 09:50:10 -0400    

Click here for diff

M src/build/configure.ac
M src/configure

Add repo path env var to make release commands easier to run.

commit   : ccdf4bf3ccb091a1371646f65583a77105c43e0d    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 09:43:29 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 09:43:29 -0400    

Click here for diff

M doc/RELEASE.md

Move configure.ac to src/build.

commit   : 838ef4eca1a1fa2c791fa4913a3cae803eb4c8a8    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 09:34:52 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 09:34:52 -0400    

Click here for diff

This file is used to generate src/configure and is not required to make pgbackrest since src/configure is updated before distribution.  
  
Move to src/build so it is out of the way.  

R100 src/configure.ac src/build/configure.ac
M test/test.pl

Remove extra spaces.

commit   : 79a6798f39124db044f0af5091027ecb255ba033    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Mar 2020 20:51:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 Mar 2020 20:51:11 -0400    

Click here for diff

M src/storage/read.intern.h

Fix misleading comment.

commit   : 2ac9c19d4a2a3e75373b97ab2360f11469998448    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Mar 2020 18:34:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 Mar 2020 18:34:59 -0400    

Click here for diff

M test/test.pl

Detect changes in reference.xml for code auto-generation.

commit   : 181fa1fc8bbc4af06e1ff491b165b3b0e78cdfd6    
  
author   : David Steele <[email protected]>    
date     : Wed, 11 Mar 2020 18:33:11 -0400    
  
committer: David Steele <[email protected]>    
date     : Wed, 11 Mar 2020 18:33:11 -0400    

Click here for diff

Changes to reference.xml can affect the command-line documentation built into the binary so changes must trigger an auto-generated code build during smart builds.  

M test/test.pl

Get package source files dynamically during package build.

commit   : 0ba8062f5f04e22edb5700be3d8eb26255ea68d4    
  
author   : David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 08:48:45 -0400    
  
committer: David Steele <[email protected]>    
date     : Thu, 12 Mar 2020 08:48:45 -0400    

Click here for diff

The prior method was to build a special container to hold these files which meant they would get stale on development systems.  On CI the container was always rebuilt so failures would be seen there even when dev seemed to be working.  
  
Instead get the package source when the package is built to ensure it is as up-to-date as possible.  
  
This change was prompted by failures on the Ubuntu 12.04 container while getting the package source, probably due to an ancient version of git.  Package builds are no longer supported on that platform with the addition of lz4 compression so it didn't seem worth fixing.  

M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/test.pl

Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo.

commit   : 4a5bd002c0071960b537e6d1dc35efe626146956    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 17:57:02 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 17:57:02 -0400    

Click here for diff

The primary source for project info is now src/version.h.  
  
The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.  

M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/RELEASE.md
M doc/doc.pl
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/lib/pgBackRestDoc/Common/Ini.pm
M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
M doc/lib/pgBackRestDoc/Html/DocHtmlSite.pm
M doc/lib/pgBackRestDoc/Latex/DocLatex.pm
M doc/lib/pgBackRestDoc/Markdown/DocMarkdown.pm
A doc/lib/pgBackRestDoc/ProjectInfo.pm
M doc/manifest.xml
M doc/release.pl
D lib/pgBackRest/Version.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/StorageRepo.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/test.pl

Rename BackRestDoc Perl module to pgBackRestDoc.

commit   : 731b862e6faf266c7ddbbd01a85496147d8bf7a6    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 15:41:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 15:41:56 -0400    

Click here for diff

This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.  
  
More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.  

M build/lib/pgBackRestBuild/Build.pm
M build/lib/pgBackRestBuild/Build/Common.pm
M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M build/lib/pgBackRestBuild/Error/Build.pm
M doc/doc.pl
R099 doc/lib/BackRestDoc/Common/Doc.pm doc/lib/pgBackRestDoc/Common/Doc.pm
R099 doc/lib/BackRestDoc/Common/DocConfig.pm doc/lib/pgBackRestDoc/Common/DocConfig.pm
R099 doc/lib/BackRestDoc/Common/DocExecute.pm doc/lib/pgBackRestDoc/Common/DocExecute.pm
R098 doc/lib/BackRestDoc/Common/DocManifest.pm doc/lib/pgBackRestDoc/Common/DocManifest.pm
R098 doc/lib/BackRestDoc/Common/DocRender.pm doc/lib/pgBackRestDoc/Common/DocRender.pm
R097 doc/lib/BackRestDoc/Common/Exception.pm doc/lib/pgBackRestDoc/Common/Exception.pm
R099 doc/lib/BackRestDoc/Common/Ini.pm doc/lib/pgBackRestDoc/Common/Ini.pm
R099 doc/lib/BackRestDoc/Common/Log.pm doc/lib/pgBackRestDoc/Common/Log.pm
R098 doc/lib/BackRestDoc/Common/String.pm doc/lib/pgBackRestDoc/Common/String.pm
R099 doc/lib/BackRestDoc/Custom/DocCustomRelease.pm doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
R097 doc/lib/BackRestDoc/Html/DocHtmlBuilder.pm doc/lib/pgBackRestDoc/Html/DocHtmlBuilder.pm
R095 doc/lib/BackRestDoc/Html/DocHtmlElement.pm doc/lib/pgBackRestDoc/Html/DocHtmlElement.pm
R096 doc/lib/BackRestDoc/Html/DocHtmlPage.pm doc/lib/pgBackRestDoc/Html/DocHtmlPage.pm
R093 doc/lib/BackRestDoc/Html/DocHtmlSite.pm doc/lib/pgBackRestDoc/Html/DocHtmlSite.pm
R092 doc/lib/BackRestDoc/Latex/DocLatex.pm doc/lib/pgBackRestDoc/Latex/DocLatex.pm
R098 doc/lib/BackRestDoc/Latex/DocLatexSection.pm doc/lib/pgBackRestDoc/Latex/DocLatexSection.pm
R088 doc/lib/BackRestDoc/Markdown/DocMarkdown.pm doc/lib/pgBackRestDoc/Markdown/DocMarkdown.pm
R097 doc/lib/BackRestDoc/Markdown/DocMarkdownRender.pm doc/lib/pgBackRestDoc/Markdown/DocMarkdownRender.pm
M doc/manifest.xml
M doc/release.pl
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/CoverageTest.pm
M test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/ExecuteTest.pm
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Common/HostGroupTest.pm
M test/lib/pgBackRestTest/Common/HostTest.pm
M test/lib/pgBackRestTest/Common/Io/Base.pm
M test/lib/pgBackRestTest/Common/Io/Buffered.pm
M test/lib/pgBackRestTest/Common/Io/Filter.pm
M test/lib/pgBackRestTest/Common/Io/Handle.pm
M test/lib/pgBackRestTest/Common/Io/Process.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/Storage.pm
M test/lib/pgBackRestTest/Common/StorageBase.pm
M test/lib/pgBackRestTest/Common/StoragePosix.pm
M test/lib/pgBackRestTest/Common/StoragePosixRead.pm
M test/lib/pgBackRestTest/Common/StoragePosixWrite.pm
M test/lib/pgBackRestTest/Common/StorageRepo.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Common/Wait.pm
M test/lib/pgBackRestTest/Env/ArchiveInfo.pm
M test/lib/pgBackRestTest/Env/BackupInfo.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Env/Manifest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Performance/PerformanceArchivePerlTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/test.pl
M test/travis.pl

Move Perl modules out of lib directory.

commit   : 36d4ab9bff668efcbbbf3393d80bd53f0be09a67    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 15:12:44 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 15:12:44 -0400    

Click here for diff

This directory was once the home of the production Perl code but since f0ef73db this is no longer true.  
  
Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).  
  
The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.  

M build/lib/pgBackRestBuild/Build.pm
M build/lib/pgBackRestBuild/Build/Common.pm
M build/lib/pgBackRestBuild/Config/Build.pm
M build/lib/pgBackRestBuild/Config/BuildDefine.pm
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M build/lib/pgBackRestBuild/Config/Data.pm
M build/lib/pgBackRestBuild/Error/Build.pm
M doc/doc.pl
M doc/lib/BackRestDoc/Common/Doc.pm
M doc/lib/BackRestDoc/Common/DocConfig.pm
M doc/lib/BackRestDoc/Common/DocExecute.pm
M doc/lib/BackRestDoc/Common/DocManifest.pm
M doc/lib/BackRestDoc/Common/DocRender.pm
R094 lib/pgBackRest/Common/Exception.pm doc/lib/BackRestDoc/Common/Exception.pm
R099 lib/pgBackRest/Common/Ini.pm doc/lib/BackRestDoc/Common/Ini.pm
R099 lib/pgBackRest/Common/Log.pm doc/lib/BackRestDoc/Common/Log.pm
A doc/lib/BackRestDoc/Common/String.pm
M doc/lib/BackRestDoc/Custom/DocCustomRelease.pm
M doc/lib/BackRestDoc/Html/DocHtmlBuilder.pm
M doc/lib/BackRestDoc/Html/DocHtmlElement.pm
M doc/lib/BackRestDoc/Html/DocHtmlPage.pm
M doc/lib/BackRestDoc/Html/DocHtmlSite.pm
M doc/lib/BackRestDoc/Latex/DocLatex.pm
M doc/lib/BackRestDoc/Latex/DocLatexSection.pm
M doc/lib/BackRestDoc/Markdown/DocMarkdown.pm
M doc/lib/BackRestDoc/Markdown/DocMarkdownRender.pm
M doc/release.pl
D lib/pgBackRest/Archive/Common.pm
D lib/pgBackRest/Backup/Common.pm
D lib/pgBackRest/Common/String.pm
D lib/pgBackRest/Storage/Helper.pm
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/CoverageTest.pm
R088 lib/pgBackRest/DbVersion.pm test/lib/pgBackRestTest/Common/DbVersion.pm
M test/lib/pgBackRestTest/Common/DefineTest.pm
M test/lib/pgBackRestTest/Common/ExecuteTest.pm
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Common/HostGroupTest.pm
M test/lib/pgBackRestTest/Common/HostTest.pm
R098 lib/pgBackRest/Common/Io/Base.pm test/lib/pgBackRestTest/Common/Io/Base.pm
R097 lib/pgBackRest/Common/Io/Buffered.pm test/lib/pgBackRestTest/Common/Io/Buffered.pm
R097 lib/pgBackRest/Common/Io/Filter.pm test/lib/pgBackRestTest/Common/Io/Filter.pm
R097 lib/pgBackRest/Common/Io/Handle.pm test/lib/pgBackRestTest/Common/Io/Handle.pm
R093 lib/pgBackRest/Common/Io/Process.pm test/lib/pgBackRestTest/Common/Io/Process.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/LogTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/Storage.pm
R097 lib/pgBackRest/Storage/Base.pm test/lib/pgBackRestTest/Common/StorageBase.pm
M test/lib/pgBackRestTest/Common/StoragePosix.pm
M test/lib/pgBackRestTest/Common/StoragePosixRead.pm
M test/lib/pgBackRestTest/Common/StoragePosixWrite.pm
R081 lib/pgBackRest/Storage/Storage.pm test/lib/pgBackRestTest/Common/StorageRepo.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
R098 lib/pgBackRest/Common/Wait.pm test/lib/pgBackRestTest/Common/Wait.pm
R092 lib/pgBackRest/Archive/Info.pm test/lib/pgBackRestTest/Env/ArchiveInfo.pm
R096 lib/pgBackRest/Backup/Info.pm test/lib/pgBackRestTest/Env/BackupInfo.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
R097 lib/pgBackRest/InfoCommon.pm test/lib/pgBackRestTest/Env/InfoCommon.pm
R099 lib/pgBackRest/Manifest.pm test/lib/pgBackRestTest/Env/Manifest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Performance/PerformanceArchivePerlTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/test.pl
M test/travis.pl

Add lz4 compression support.

commit   : c279a00279e3c5b956f6dc4496454a9c4ccaa487    
  
author   : David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 14:45:27 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 14:45:27 -0400    

Click here for diff

LZ4 compresses data faster than gzip but at a lower ratio.  This can be a good tradeoff in certain scenarios.  
  
Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.  

M .travis.yml
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/Makefile.in
M src/build.auto.h.in
M src/common/compress/helper.c
A src/common/compress/lz4/common.c
A src/common/compress/lz4/common.h
A src/common/compress/lz4/compress.c
A src/common/compress/lz4/compress.h
A src/common/compress/lz4/decompress.c
A src/common/compress/lz4/decompress.h
M src/config/define.auto.c
M src/configure
M src/configure.ac
M test/Vagrantfile
M test/container.yaml
M test/define.yaml
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-004.log
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
A test/patch/debian-package.patch
A test/patch/rhel-package.patch
M test/src/module/common/compressTest.c
M test/travis.pl

Rename flush->flushing to be consistent with usage in other modules.

commit   : cc9d7315dbb4e66cd281dc2156faf529e566ddbc    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 10 Mar 2020 14:05:00 -0400    
  
committer: David Steele <[email protected]>    
date     : Tue, 10 Mar 2020 14:05:00 -0400    

Click here for diff

M src/common/compress/gz/compress.c

Remove LibC.

commit   : 79cfd3aebf4848c325d6eb98f5fe88f7f03805b0    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 17:41:59 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 17:41:59 -0400    

Click here for diff

This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests.  This is expensive code to maintain just for testing.  
  
The main dependency was the interface to storage, no matter where it was located, e.g. S3.  Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line.  
  
The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary.  Replace these with hard-coded strings in most places and create new constants for commonly used values.  
  
Remove all auto-generated Perl code.  This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm.  This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired.  
  
Update test.pl and related code to remove LibC builds.  
  
Ding, dong, LibC is dead.  

M doc/xml/user-guide.xml
M lib/pgBackRest/Archive/Common.pm
M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Backup/Common.pm
M lib/pgBackRest/Backup/Info.pm
M lib/pgBackRest/Common/Exception.pm
D lib/pgBackRest/Common/ExceptionAuto.pm
M lib/pgBackRest/Common/Ini.pm
A lib/pgBackRest/Common/Io/Process.pm
D lib/pgBackRest/Config/Config.pm
D lib/pgBackRest/LibC.pm
D lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Manifest.pm
D lib/pgBackRest/Protocol/Storage/Helper.pm
M lib/pgBackRest/Storage/Base.pm
M lib/pgBackRest/Storage/Helper.pm
M lib/pgBackRest/Storage/Storage.pm
D lib/pgBackRest/Storage/StorageRead.pm
D lib/pgBackRest/Storage/StorageWrite.pm
D libc/LibC.h
D libc/LibC.xs
D libc/Makefile.PL
D libc/build/lib/pgBackRestLibC/Build.pm
D libc/typemap
D libc/xs/config/config.xs
D libc/xs/config/configTest.xs
D libc/xs/config/configTest.xsh
D libc/xs/config/define.xs
D libc/xs/storage/storage.xs
D libc/xs/storage/storage.xsh
D libc/xs/storage/storageRead.xs
D libc/xs/storage/storageRead.xsh
D libc/xs/storage/storageWrite.xs
D libc/xs/storage/storageWrite.xsh
M test/Vagrantfile
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/BuildTest.pm
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/lib/pgBackRestTest/Common/ListTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Common/Storage.pm
M test/lib/pgBackRestTest/Common/StoragePosix.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
D test/lib/pgBackRestTest/Env/ConfigEnvTest.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBaseTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbCommonTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbSyntheticTest.pm
M test/lib/pgBackRestTest/Env/Host/HostDbTest.pm
M test/lib/pgBackRestTest/Env/Host/HostS3Test.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Performance/PerformanceArchivePerlTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/test.pl

Add repo-create, repo-get, repo-put, and repo-rm commands.

commit   : d3c83453deffa6435ec5c4a932ad62a3d4d2cd95    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 17:15:03 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 17:15:03 -0400    

Click here for diff

These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage.  
  
All the commands are currently internal only and should not be used on production repositories.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/LibCAuto.pm
M src/Makefile.in
A src/command/repo/create.c
A src/command/repo/create.h
A src/command/repo/get.c
A src/command/repo/get.h
A src/command/repo/put.c
A src/command/repo/put.h
A src/command/repo/rm.c
A src/command/repo/rm.h
M src/common/io/io.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/main.c
M test/define.yaml
M test/src/module/command/repoTest.c

Update repo-ls command to work better with files.

commit   : 948835fb84010f515ec90395bd0c009e9884ad82    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 16:54:07 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 16:54:07 -0400    

Click here for diff

If the command was passed a file it would return no results since it was originally intended to list files when passed a path.  
  
However, as a general purpose command working directly with files makes sense.  

M src/command/repo/ls.c
M test/src/module/command/repoTest.c

Rename ls command to repo-ls.

commit   : 5e1291a29f65d0430966756c4031f6cc307ca208    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 16:41:04 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 16:41:04 -0400    

Click here for diff

This command only makes sense for the repository storage since other storage (e.g. pg and spool) must be located on a local Posix filesystem and can be listed using standard unix commands.  Since the repo storage can be located lots of places having a common way to list it makes sense.  
  
Prefix with repo- to make the scope of this command clear.  
  
Update documentation to reflect this change.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/auto/metric-coverage-report.auto.xml
M doc/xml/reference.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/LibCAuto.pm
M src/Makefile.in
R096 src/command/storage/list.c src/command/repo/ls.c
R091 src/command/storage/list.h src/command/repo/ls.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/main.c
M test/code-count/file-type.yaml
M test/define.yaml
R097 test/src/module/command/storageTest.c test/src/module/command/repoTest.c

Remove valgrind suppressions made obsolete by f0ef73db.

commit   : f581edfa509c0879164597eba5ecf9dc287ca08d    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 13:31:08 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 13:31:08 -0400    

Click here for diff

M test/src/valgrind.suppress

Remove Perl unit tests made obsolete in 434cd832.

commit   : 3c4f91b31929c56f43ebb8f9bd48f39295dae917    
  
author   : David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 13:26:56 -0400    
  
committer: David Steele <[email protected]>    
date     : Mon, 9 Mar 2020 13:26:56 -0400    

Click here for diff

These were replaced by C unit tests but not all the unit test setup code was removed in the Perl module.  

M test/lib/pgBackRestTest/Module/Mock/MockExpireTest.pm

Fix incorrect parameter in logging function.

commit   : e93f1dd898ea76e4068d3723f36282bf715cd9d8    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 6 Mar 2020 15:33:57 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 15:33:57 -0500    

Click here for diff

M src/common/compress/gz/compress.c

Cleanup pgPageChecksum() test in postgres/interface module.

commit   : 54bc3b454a18677610d54e37813c51ad36b6c7d8    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 15:01:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 15:01:50 -0500    

Click here for diff

Some of the comments were wrong or inconsistent.  
  
Update TEST_RESULT_U16_HEX() to the less-specific TEST_RESULT_UINT_HEX().  

M test/src/common/harnessTest.h
M test/src/module/postgres/interfaceTest.c

Add infrastructure for multiple compression type support.

commit   : 438b957f9cf11b1c23636ec781c67f59bf4a8af1    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 14:41:03 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 14:41:03 -0500    

Click here for diff

Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function.  
  
Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format).  
  
Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types.  
  
Additional compression types will be added in future commits.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Manifest.pm
M libc/Makefile.PL
M src/Makefile.in
M src/command/archive/common.c
M src/command/archive/common.h
M src/command/archive/get/file.c
M src/command/archive/push/file.c
M src/command/archive/push/file.h
M src/command/archive/push/protocol.c
M src/command/archive/push/push.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/file.h
M src/command/backup/protocol.c
M src/command/restore/file.c
M src/command/restore/file.h
M src/command/restore/protocol.c
M src/command/restore/restore.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/compress.h
A src/common/compress/helper.c
A src/common/compress/helper.h
A src/common/compress/helper.intern.h
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/info/infoBackup.c
M src/info/manifest.c
M src/info/manifest.h
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/write.c
M test/define.yaml
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveStopTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/common/harnessStorage.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/helpTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/compressTest.c
M test/src/module/config/loadTest.c
M test/src/module/info/manifestTest.c
M test/src/module/storage/remoteTest.c

Remove obsolete methods in pgBackRest::Storage::Storage module.

commit   : 02aa03d1a266827b7ff5cb739b9598e475777086    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 14:10:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 14:10:09 -0500    

Click here for diff

All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible.  
  
First remove storageDb() and use storageTest() instead.  Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC.  Now the only storage using the LibC interface is storageRepo().  
  
Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed.  Same for owner().  
  
Remove pathSync() because syncs are not required in the tests.  No test data is reused after a crash.  
  
Path create/exists functions should never be explicitly performed on a repo so remove those.  File exists can be implemented by calling info() instead.  
  
Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete.  
  
Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used.  That also means there are no filters returning results so remove all the result code.  
  
Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.  

M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Backup/Info.pm
M lib/pgBackRest/Common/Ini.pm
M lib/pgBackRest/Common/Io/Filter.pm
M lib/pgBackRest/Protocol/Storage/Helper.pm
M lib/pgBackRest/Storage/Base.pm
M lib/pgBackRest/Storage/Storage.pm
M lib/pgBackRest/Storage/StorageRead.pm
M lib/pgBackRest/Storage/StorageWrite.pm
M libc/xs/storage/storage.xs
M libc/xs/storage/storage.xsh
M libc/xs/storage/storageRead.xs
M libc/xs/storage/storageWrite.xs
M test/lib/pgBackRestTest/Common/FileTest.pm
M test/lib/pgBackRestTest/Common/RunTest.pm
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockStanzaTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Remove Perl Db module and LibC dependencies.

commit   : 00647c7109cab28110d7dabd3da039b391507d50    
  
author   : David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 07:21:17 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 6 Mar 2020 07:21:17 -0500    

Click here for diff

This was mostly dead code except the DB_BACKUP_ADVISORY_LOCK constant, moved to the real/all test module, and the function that pulls info from pg_control, moved to ExpireEnvTest.pm.  

D lib/pgBackRest/Db.pm
M libc/LibC.xs
M libc/Makefile.PL
M libc/typemap
D libc/xs/postgres/client.xs
D libc/xs/postgres/client.xsh
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

Remove dependency on LibC hash filter.

commit   : 2e0fe2565025d0408e3d0f740fe1e5916d3ebbc0    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 18:34:59 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 18:34:59 -0500    

Click here for diff

Perl provides Digest::SHA for hashing so there is no need to expose this via LibC anymore.  

M lib/pgBackRest/Common/Ini.pm
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Storage/Base.pm
M lib/pgBackRest/Storage/Storage.pm
M libc/LibC.xs
M libc/build/lib/pgBackRestLibC/Build.pm
D libc/xs/crypto/hash.xs
D libc/xs/crypto/hash.xsh
M libc/xs/storage/storage.xsh
M test/lib/pgBackRestTest/Env/HostEnvTest.pm

Move logic from postgres/pageChecksum to command/backup/pageChecksum().

commit   : e55443c890181ea63a350275447885331c8254e4    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 16:12:54 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 16:12:54 -0500    

Click here for diff

The postgres/pageChecksum module was designed as an interface to the C structs for the Perl code.  The new C code can do this directly so no need for an interface.  
  
Move the remaining test for pgPageChecksum() into the postgres/interface test module.  

M src/Makefile.in
M src/command/backup/pageChecksum.c
M src/postgres/interface/static.auto.h
M src/postgres/interface/version.auto.h
D src/postgres/pageChecksum.c
D src/postgres/pageChecksum.h
M test/define.yaml
M test/src/module/command/backupCommonTest.c
M test/src/module/postgres/interfaceTest.c
D test/src/module/postgres/pageChecksumTest.c

Use stock PostgreSQL page checksum implementation.

commit   : 3796b74dcac29d7e7e7f89b69d3fa92b9d105a17    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 14:23:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 14:23:01 -0500    

Click here for diff

We were using a customized version which worked fine but was hard to merge with upstream changes.  Now this code is maintained much like the types in static.auto.h that we copy and check with each release.  
  
The goal is to eventually build directly against PostgreSQL (either source or libcommon) and this brings us one step closer.  

M src/Makefile.in
M src/postgres/interface.h
A src/postgres/interface/page.c
A src/postgres/interface/pageChecksum.auto.c
M src/postgres/interface/static.auto.h
M src/postgres/pageChecksum.c
M src/postgres/pageChecksum.h
M test/test.pl

Remove invalid page checksum test.

commit   : 1b647a1a22ae36e2447dc04384ab794b011a74aa    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 14:06:36 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 14:06:36 -0500    

Click here for diff

All zero pages should not have checksums.  Not only is this test invalid but it will not work with the stock page checksum implementation in PostgreSQL, which checks for zero pages.  Since we will be using that code verbatim soon this test needs to go.  

M test/src/module/postgres/pageChecksumTest.c

Use static checksums in mock/all integration tests.

commit   : eb4347f20b85fa86208bb31b24b8b6f383432706    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 13:56:20 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 13:56:20 -0500    

Click here for diff

Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests.  
  
We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.  

M lib/pgBackRest/LibCAuto.pm
M libc/LibC.xs
M libc/Makefile.PL
M libc/build/lib/pgBackRestLibC/Build.pm
D libc/xs/postgres/pageChecksum.xs
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

Remove invalid const in pgPageChecksum() parameter.

commit   : 77853d3c1387cd6ba42396e7ca29fae0f18ab6ac    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 11:14:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 11:14:53 -0500    

Click here for diff

pgPageChecksum() must modify the page header in order to calculate the checksum.  The modification is temporary but make it clear that it happens by removing the const.  
  
Also make a note about our non-entirely-kosher usage of a const Buffer in the PageChecksum filter.  This is safe as currently coded but at the least we need to be aware of what is going on.  

M libc/xs/postgres/pageChecksum.xs
M src/command/backup/pageChecksum.c
M src/postgres/pageChecksum.c
M src/postgres/pageChecksum.h

Use PG_PAGE_SIZE_DEFAULT constant instead of pageSize variable.

commit   : 4ab8943ca886ec61934cea55182c1123cd4596a7    
  
author   : David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 09:14:27 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 5 Mar 2020 09:14:27 -0500    

Click here for diff

Page size is passed around a lot but in fact it can only have one value, PG_PAGE_SIZE_DEFAULT, which is checked when pg_control is loaded. There may be an argument for supporting multiple page sizes in the future but for now just use the constant to simplify the code.  
  
There is also a significant performance benefit.  Because pageSize was being used in pageChecksumBlock() the main loop was neither unrolled nor vectorized (-funroll-loops -ftree-vectorize) as it is now with a constant loop boundary.  

M libc/xs/postgres/pageChecksum.xs
M src/Makefile.in
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/backup/pageChecksum.c
M src/command/backup/pageChecksum.h
M src/postgres/pageChecksum.c
M src/postgres/pageChecksum.h
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/postgres/pageChecksumTest.c
M test/src/module/storage/remoteTest.c

Rename old page*() functions to conform to new conventions.

commit   : 91f321fb865fe4569f50d3a280d9ad1f502b7cbd    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 Mar 2020 14:24:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 4 Mar 2020 14:24:40 -0500    

Click here for diff

The general convention now is to prefix PostgreSQL functions with "pg".  

M lib/pgBackRest/LibCAuto.pm
M libc/build/lib/pgBackRestLibC/Build.pm
M libc/xs/postgres/pageChecksum.xs
M src/command/backup/pageChecksum.c
M src/postgres/pageChecksum.c
M src/postgres/pageChecksum.h
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
M test/src/module/postgres/pageChecksumTest.c

Remove obsolete function pageChecksumBufferTest().

commit   : a86253f1128ad7cd3921826df4baec2e899667f9    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 Mar 2020 14:12:02 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 4 Mar 2020 14:12:02 -0500    

Click here for diff

This function made validation faster in Perl because fewer calls (and buffer transformations) were required when all checksums were valid.  
  
In C calling pageChecksumTest() directly is just as efficient so there is no longer a need for pageChecksumBufferTest().  

M src/postgres/pageChecksum.c
M src/postgres/pageChecksum.h
M test/define.yaml
M test/src/module/postgres/pageChecksumTest.c

Centralize PostgreSQL page header data structures.

commit   : 9d488822682ec425e5ac4b0946a89eef240fc172    
  
author   : David Steele <[email protected]>    
date     : Wed, 4 Mar 2020 13:31:27 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 4 Mar 2020 13:31:27 -0500    

Click here for diff

These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them.  
  
To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support.  This allows us to have single, non-versioned set of headers and code for stable data structures like page headers.  
  
Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly.  
  
We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.  

M src/Makefile.in
A src/postgres/interface/static.auto.h
M src/postgres/interface/version.auto.h
M src/postgres/pageChecksum.c
M test/src/module/command/backupCommonTest.c
M test/src/module/command/backupTest.c
M test/src/module/postgres/pageChecksumTest.c

commit   : a88d709962f761a51b21dd60688c5915bbdccddb    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 Mar 2020 18:05:54 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 Mar 2020 18:05:54 -0500    

Click here for diff

M src/postgres/interface/version.auto.h

Remove incorrect comment.

commit   : e8daabb963e6220f8b14672a6a9e3b3a118b4655    
  
author   : David Steele <[email protected]>    
date     : Tue, 3 Mar 2020 18:01:56 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 3 Mar 2020 18:01:56 -0500    

Click here for diff

This was likely pasted from pageChecksum() and not removed.  

M src/postgres/pageChecksum.c

Minor adjustments to common prefix code updated in 8ec41efb.

commit   : 507a58460e9c33361baf3f1cbf2b95c9cb0dd5ae    
  
author   : David Steele <[email protected]>    
date     : Sat, 29 Feb 2020 07:52:54 -0500    
  
committer: David Steele <[email protected]>    
date     : Sat, 29 Feb 2020 07:52:54 -0500    

Click here for diff

M src/common/regExp.c

Improve poor man's regular expression common prefix generator.

commit   : 8ec41efb04ee45854d18ffaf765e4a4800dd3879    
  
author   : David Steele <[email protected]>    
date     : Fri, 28 Feb 2020 17:41:34 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 Feb 2020 17:41:34 -0500    

Click here for diff

The S3 driver depends on being able to generate a common prefix to limit the number of results from list commands, which saves on bandwidth.  
  
The prior implementation could be tricked by an expression like ^ABC|^DEF where there is more than one possible prefix.  To fix this disallow any prefix when another ^ anchor is found in the expression.  [^ and \^ are OK since they are not anchors.  
  
Note that this was not an active bug because there are currently no expressions with multiple ^ anchors.  

M src/common/regExp.c
M src/common/regExp.h
M test/src/module/common/regExpTest.c

Add missing asserts.

commit   : 3bbead548026155c8bb5b212bed80903a3ef6a97    
  
author   : David Steele <[email protected]>    
date     : Fri, 28 Feb 2020 15:03:01 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 Feb 2020 15:03:01 -0500    

Click here for diff

These interface functions did not exist when the Storage object was created but they are now required.  

M src/storage/storage.c

Add function comment.

commit   : 19ea699e90a4f797c7f168cc853faf4b8e807056    
  
author   : David Steele <[email protected]>    
date     : Fri, 28 Feb 2020 15:00:09 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 Feb 2020 15:00:09 -0500    

Click here for diff

M src/storage/posix/storage.c

Fix flapping test in real/all module.

commit   : ceb050e950c596835a9ab3b136b0de4983c69a54    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 28 Feb 2020 14:50:50 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 Feb 2020 14:50:50 -0500    

Click here for diff

The restore test function was passing strBackup to the restoreCompare function but when the restore is expected to pick a backup based on a timestamp, then strBackup may not be the one chosen.  
  
Modified the code so that strBackupExpected is set based on the parameters passed to the function and this is then passed to restoreCompare.  

M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm

Add sleep before/after retrieving timestamp in the user guide.

commit   : 089049ec563b021a69c46929f5fae13a1fdf0528    
  
author   : Cynthia Shang <[email protected]>    
date     : Fri, 28 Feb 2020 14:30:39 -0500    
  
committer: David Steele <[email protected]>    
date     : Fri, 28 Feb 2020 14:30:39 -0500    

Click here for diff

Adding a sleep before was necessary since only adding a sleep after did not always work. This helps to ensure the backup stop time for the previous backup does not equal time-recovery-timestamp. The sleep after allows enough time between the time retrieval and dropping important_table so PostgreSQL can consistently recover to before the table drop.  
  
Note that these issues were caused by picking a timestamp too close to the restore command or a database operation, not due to any problem in backup selection of the restore command.  

M doc/xml/user-guide.xml

Remove compress option from config tests.

commit   : 7d8c0d29fb5b2663c9c4b6be0b9f697a6c19c0fc    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 14:51:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 14:51:40 -0500    

Click here for diff

This option was used for boolean testing but it will soon be deprecated and the semantics changed.  To reduce churn it seems easiest to just use other options for testing.  This will also be helpful when the option is eventually removed.  

M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/parseTest.c

Remove compress/compress-level options from commands where unused.

commit   : dbf6255ab8ca9a141f9a7fbd1fda99006cce9e05    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 12:25:32 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 12:25:32 -0500    

Click here for diff

These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options.  

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/release.xml
M src/config/define.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/src/module/command/commandTest.c
M test/src/module/command/helpTest.c

Add missing static keywords.

commit   : 8f5337a136bcb8edf672a335164fc36768f22a73    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 12:21:53 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 12:21:53 -0500    

Click here for diff

Interface functions should be marked static since they can only be called through the IoFilter interface.  

M src/common/compress/gz/decompress.c

Remove raw option for gz compression.

commit   : 3f77a83e7367a05d7fee9612159e671910db25e0    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 12:19:40 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 12:19:40 -0500    

Click here for diff

This was a minor optimization used in protocol layer compression.  Even though it was slightly faster, it omitted the crc-32 that is generated during normal compression which could lead to corrupt data after a bad network transmission.  This would be caught on restore by our checksum but it seems better to catch an issue like this early.  
  
The raw option also made the function signature different than future compression formats which may not support raw, or require different code to support raw.  
  
In general, it doesn't seem worth the extra testing to support a format that has minimal benefit and is seldom used, since protocol compression is only enabled when the transmitted data is uncompressed.  

M libc/xs/storage/storage.xsh
M src/command/archive/get/file.c
M src/command/archive/push/file.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/restore/file.c
M src/common/compress/gz/common.c
M src/common/compress/gz/common.h
M src/common/compress/gz/compress.c
M src/common/compress/gz/compress.h
M src/common/compress/gz/decompress.c
M src/common/compress/gz/decompress.h
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/write.c
M test/define.yaml
M test/src/common/harnessStorage.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/compressTest.c
M test/src/module/storage/remoteTest.c

Rename "gzip" to "gz".

commit   : ee351682dae215ccbb2ae9d5c0932a41580635dd    
  
author   : David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 12:09:05 -0500    
  
committer: David Steele <[email protected]>    
date     : Thu, 27 Feb 2020 12:09:05 -0500    

Click here for diff

"gz" was used as the extension but "gzip" was generally used for function and type naming.  
  
With a new compression format on the way, it makes sense to standardize on a single abbreviation to represent a compression format in the code.  Since the extension is standard and we must use it, also use the extension for all naming.  

M doc/xml/auto/metric-coverage-report.auto.xml
M lib/pgBackRest/Archive/Info.pm
M lib/pgBackRest/Storage/Base.pm
M libc/Makefile.PL
M libc/xs/storage/storage.xsh
M src/Makefile.in
M src/command/archive/get/file.c
M src/command/archive/push/file.c
M src/command/backup/backup.c
M src/command/backup/file.c
M src/command/restore/file.c
R089 src/common/compress/gzip/common.c src/common/compress/gz/common.c
R079 src/common/compress/gzip/common.h src/common/compress/gz/common.h
R075 src/common/compress/gzip/compress.c src/common/compress/gz/compress.c
R070 src/common/compress/gzip/compress.h src/common/compress/gz/compress.h
R073 src/common/compress/gzip/decompress.c src/common/compress/gz/decompress.c
R070 src/common/compress/gzip/decompress.h src/common/compress/gz/decompress.h
M src/common/io/filter/filter.h
M src/configure
M src/configure.ac
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/write.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/common/harnessStorage.c
M test/src/module/command/archiveGetTest.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
R065 test/src/module/common/compressGzipTest.c test/src/module/common/compressTest.c
M test/src/module/storage/remoteTest.c

Improve performance of MEM_CONTEXT*() macros.

commit   : 5afd950ed98b7eea0bc0c52851fdbe4fb7637698    
  
author   : David Steele <[email protected]>    
date     : Wed, 26 Feb 2020 21:15:39 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Feb 2020 21:15:39 -0500    

Click here for diff

The prior code used TRY...CATCH blocks to cleanup mem contexts when an error occurred. This included freeing new mem contexts that were still being initialized when the error occurred and ensuring that the prior memory context was restored.  
  
This worked fine in production but it involved a lot of setjmp()/longjmp() calls that resulted in longer compilation times and sluggish performance under valgrind, profiling, and coverage testing.  
  
Instead maintain a stack of new contexts and context switches that can be used to do cleanup after an error. Normally, the stack is not used for this purpose and pushing/popping is a cheap operation. In the prior implementation most of the TRY...CATCH logic needed to be run even on success.  
  
One bonus is that the binary is about 8% smaller after this change.  Another benefit is that new contexts *must* be explicitly freed/discarded or an error will occur.  See info/manifest.c for an example of where this is useful outside the standard macros.  

M doc/xml/release.xml
M libc/LibC.h
M src/Makefile.in
M src/common/error.c
M src/common/memContext.c
M src/common/memContext.h
M src/info/manifest.c
M test/src/module/common/memContextTest.c

Fix incorrect lcov version in contributing guide.

commit   : d68771a4a5f0f349ba31679a27bc7b2ab8b0d736    
  
author   : Cynthia Shang <[email protected]>    
date     : Wed, 26 Feb 2020 20:40:24 -0500    
  
committer: David Steele <[email protected]>    
date     : Wed, 26 Feb 2020 20:40:24 -0500    

Click here for diff

M CONTRIBUTING.md
M doc/xml/contributing.xml

Update enum formatting and NULL test to project style.

commit   : 99b052a38ab41c2f10a68f234830508904341139    
  
author   : Cynthia Shang <[email protected]>    
date     : Tue, 25 Feb 2020 17:25:12 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 Feb 2020 17:25:12 -0500    

Click here for diff

M src/common/memContext.c

Begin v2.25 development.

commit   : 9e0dc83e878fbe407b8daf05710a8b8e73638c6e    
  
author   : David Steele <[email protected]>    
date     : Tue, 25 Feb 2020 17:18:25 -0500    
  
committer: David Steele <[email protected]>    
date     : Tue, 25 Feb 2020 17:18:25 -0500    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/configure
M src/configure.ac
M src/version.h