v2.05: Environment Variable Options and Exclude Temporary/Unlogged Relations
commit : bc7462d86d3529712cf7c196a04f7d6f8dd8ee34
author : David Steele <david@pgmasters.net>
date : Fri, 31 Aug 2018 13:19:43 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 31 Aug 2018 13:19:43 -0400
Bug Fixes:
* Fix issue where relative links in $PGDATA could be stored in the backup with the wrong path. This issue did not affect absolute links and relative tablespace links were caught by other checks. (Reported by Cynthia Shang.)
* Remove incompletely implemented online option from the check command. Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly. (Reported by Jason O'Donnell.)
* Fix issue where errors raised in C were not logged when called from Perl. pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging. (Reported by Douglas J Hunley.)
* Fix issue when a boolean option (e.g. delta) was specified more than once. (Reported by Yogesh Sharma.)
Features:
* Allow any option to be set in an environment variable. This includes options that previously could only be specified on the command line, e.g. stanza, and secret options that could not be specified on the command-line, e.g. repo1-s3-key-secret.
* Exclude temporary and unlogged relation (table/index) files from backup. Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced. (Contributed by Cynthia Shang.)
* Allow arbitrary directories and/or files to be excluded from a backup. Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using. (Reviewed by Cynthia Shang.)
* Add log-subprocess option to allow file logging for local and remote subprocesses.
* PostgreSQL 11 Beta 3 support.
Improvements:
* Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta. (Contributed by Cynthia Shang.)
* Improve asynchronous archive-get/archive-push performance by directly checking status files. (Contributed by Stephen Frost.)
* Improve error message when a command is missing the stanza option. (Suggested by Sarah Conway.)
M README.md
M doc/resource/exe.cache
M doc/resource/logo.png
M doc/xml/release.xml
M doc/xml/user-guide.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h
M test/code-count/file-type.yaml
Improve log file names for remote processes started by locals.
commit : d41570c37a2820380fa1708f31ed21c09e4065f8
author : David Steele <david@pgmasters.net>
date : Fri, 31 Aug 2018 11:31:13 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 31 Aug 2018 11:31:13 -0400
The log-subprocess feature added in 22765670 failed to take into account the naming for remote processes spawned by local processes. Not only was the local command used for the naming of log files but the process id was not pass through. This meant every remote log was named "[stanza]-local-remote-000" which is confusing and meant multiple processes were writing to the same log.
Instead, pass the real command and process id to the remote. This required a minor change in locking to ignore locks if process id is greater than 0 since remotes started by locals never lock.
M build/lib/pgBackRestBuild/Config/Data.pm
M lib/pgBackRest/Main.pm
M lib/pgBackRest/Protocol/Base/Minion.pm
M lib/pgBackRest/Protocol/Helper.pm
M src/config/define.auto.c
M src/config/load.c
M src/perl/embed.auto.c
M test/expect/mock-all-002.log
M test/expect/mock-archive-002.log
M test/expect/mock-stanza-002.log
M test/lib/pgBackRestTest/Module/Protocol/ProtocolCommonMinionPerlTest.pm
M test/lib/pgBackRestTest/Module/Protocol/ProtocolHelperPerlTest.pm
M test/src/module/config/loadTest.c
Allow secrets to be passed via environment variables.
commit : c2d0a21d63a3c76bf513ac79c622e694ca9dfe1d
author : David Steele <david@pgmasters.net>
date : Thu, 30 Aug 2018 18:44:40 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 30 Aug 2018 18:44:40 -0400
When environment variables were added in d0b9f986 they were classified as cfgSourceParam, but one of the restrictions on this type is that they can't pass secrets because they might be exposed in the process list.
The solution is to reclassify environment variables as cfgSourceConfig. This allows them to handle secrets because they will not pass values to subprocesses as parameters. Instead, each subprocess is expected to check the environment directly during configuration parsing.
In passing, move the error about secrets being passed on the command-line up to command-line parsing and make the error more generic with respect to the configuration file now that multiple configuration files are allowed.
M src/config/config.h
M src/config/parse.c
M test/src/module/config/parseTest.c
Fix issue where relative links in $PGDATA could be stored in the backup with the wrong path.
commit : 70514061fdca620ba99dd0e36bb99dccbca43e1d
author : David Steele <david@pgmasters.net>
date : Thu, 30 Aug 2018 16:27:36 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 30 Aug 2018 16:27:36 -0400
Relative link paths were being combined with the paths of previous links (relative or absolute) due to the $strPath variable being modified in the current iteration rather than simply being passed to the next level of recursion.
This issue did not affect absolute links and relative tablespace links were caught by other checks, though the error was confusing.
Reported by Cynthia Shang.
M doc/xml/release.xml
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
Embedded documentation update missed in c6384904.
commit : 1631f8b3cebbe761f5eb71c9b7203664f2ae5005
author : David Steele <david@pgmasters.net>
date : Wed, 29 Aug 2018 16:45:58 -0500
committer: David Steele <david@pgmasters.net>
date : Wed, 29 Aug 2018 16:45:58 -0500
M src/config/define.auto.c
Documentation updates for exclude feature based on review.
commit : c6384904512363baaa8c6dbec42799c0ddb1d4db
author : David Steele <david@pgmasters.net>
date : Tue, 28 Aug 2018 16:49:29 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 28 Aug 2018 16:49:29 -0400
Reviewed by Cynthia Shang.
M doc/xml/reference.xml
M doc/xml/release.xml
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
Limit manifest build recursion (i.e. links followed) to sixteen levels to detect link loops.
commit : 14cde54b37bec6ac831a2ddc870531bd61b034dd
author : David Steele <david@pgmasters.net>
date : Tue, 28 Aug 2018 16:27:10 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 28 Aug 2018 16:27:10 -0400
M doc/xml/release.xml
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
Prevent manifest from being built more than once.
commit : a6cecf7d5ef722bc962a1e6a3808dce3424ca710
author : David Steele <david@pgmasters.net>
date : Tue, 28 Aug 2018 16:22:30 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 28 Aug 2018 16:22:30 -0400
M doc/xml/release.xml
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
Improve asynchronous archive-get/archive-push performance by directly checking status files.
commit : 4498f726a59132421f8a1530783f8bb6cc8905c3
author : Stephen Frost <sfrost@snowman.net>
date : Tue, 28 Aug 2018 16:04:53 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 28 Aug 2018 16:04:53 -0400
Prior to this commit, an expression was used to search the spool directory for ok/error files for a specific WAL segment. This involved setting up a regular expression and using opendir/readdir.
Instead, directly probe for the status files, checking directly if a '.ok' or '.error' file exists, avoiding the regular expression and eliminating the directory scan.
Only the two files now probed for could have ever matched the regular expression which had been provided and it's unlikely that many more additional files will be added, so this is a good improvement, and optimization, with little downside.
Contributed by Stephen Frost.
M doc/xml/release.xml
M src/command/archive/common.c
Allow arbitrary directories and/or files to be excluded from a backup.
commit : bef58a79743d3273d14de221e26291a5d90409b3
author : David Steele <david@pgmasters.net>
date : Mon, 27 Aug 2018 15:51:05 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 27 Aug 2018 15:51:05 -0400
Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using.
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Manifest.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/parse.auto.c
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
Allow command/option constants to autonumber in both C and Perl to reduce churn when a new command/option is added.
commit : 77dca5b96817d0e0d49e34fb7d8e0944e60f71a8
author : David Steele <david@pgmasters.net>
date : Fri, 24 Aug 2018 19:31:45 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 24 Aug 2018 19:31:45 -0400
M build/lib/pgBackRestBuild/Config/Build.pm
M doc/xml/release.xml
M lib/pgBackRest/LibC.pm
M lib/pgBackRest/LibCAuto.pm
M libc/build/lib/pgBackRestLibC/Build.pm
M src/config/config.auto.h
M src/perl/embed.auto.c
Add cvtZToUInt() to convert string to unsigned int.
commit : 4c3cf435c036037ff32b7a76e53753e765358173
author : David Steele <david@pgmasters.net>
date : Fri, 24 Aug 2018 17:08:00 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 24 Aug 2018 17:08:00 -0400
M doc/xml/release.xml
M src/common/type/convert.c
M src/common/type/convert.h
M test/src/module/common/typeConvertTest.c
Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta.
commit : eb30d88b6a6048353b269d590c43fad9e6233a25
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Fri, 24 Aug 2018 16:50:33 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 24 Aug 2018 16:50:33 -0400
Contributed by Cynthia Shang.
M doc/xml/release.xml
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
Update Archive::Info->archiveIdList() to return a valid error code instead of unknown.
commit : 0ed37ab9e73fb59a5558b39bb292e0eb4f331ec9
author : David Steele <david@crunchydata.com>
date : Fri, 24 Aug 2018 12:13:10 -0400
committer: David Steele <david@crunchydata.com>
date : Fri, 24 Aug 2018 12:13:10 -0400
M doc/xml/release.xml
M lib/pgBackRest/Archive/Info.pm
M src/perl/embed.auto.c
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/lib/pgBackRestTest/Module/Archive/ArchiveGetPerlTest.pm
M test/lib/pgBackRestTest/Module/Archive/ArchiveInfoUnitPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockArchiveTest.pm
Add log-subprocess option to allow file logging for local and remote subprocesses.
commit : 22765670270067c5102a2ab9b3277d4cbc86c6af
author : David Steele <david@pgmasters.net>
date : Wed, 22 Aug 2018 20:05:49 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 22 Aug 2018 20:05:49 -0400
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/LibCAuto.pm
M lib/pgBackRest/Main.pm
M lib/pgBackRest/Protocol/Helper.pm
M lib/pgBackRest/Protocol/Local/Process.pm
M src/config/config.auto.c
M src/config/config.auto.h
M src/config/define.auto.c
M src/config/define.auto.h
M src/config/load.c
M src/config/parse.auto.c
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/expect/mock-all-003.log
M test/expect/mock-archive-001.log
M test/expect/mock-archive-002.log
M test/expect/mock-archive-003.log
M test/expect/mock-archive-stop-001.log
M test/expect/mock-archive-stop-002.log
M test/expect/mock-archive-stop-003.log
M test/expect/mock-archive-stop-004.log
M test/expect/mock-archive-stop-005.log
M test/expect/mock-archive-stop-006.log
M test/expect/mock-archive-stop-007.log
M test/expect/mock-expire-001.log
M test/expect/mock-expire-002.log
M test/expect/mock-stanza-001.log
M test/expect/mock-stanza-002.log
M test/expect/mock-stanza-003.log
M test/expect/real-all-001.log
M test/expect/real-all-002.log
M test/expect/real-all-003.log
M test/expect/real-all-004.log
M test/expect/real-all-005.log
M test/expect/real-all-006.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Protocol/ProtocolHelperPerlTest.pm
M test/src/module/config/configTest.c
M test/src/module/config/loadTest.c
M test/src/module/help/helpTest.c
Improve error reporting for TEST_ASSIGN() and TEST_RESULT_VOID() macros.
commit : 512da125ad7314d2d58931186316423f039c8c35
author : David Steele <david@pgmasters.net>
date : Wed, 22 Aug 2018 19:03:13 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 22 Aug 2018 19:03:13 -0400
M doc/xml/release.xml
M test/src/common/harnessTest.h
Fix issue when a boolean option (e.g. delta) was specified more than once.
commit : ad989068d283a99a8aaf20be2eb0337a47bb8f73
author : David Steele <david@pgmasters.net>
date : Wed, 22 Aug 2018 16:45:25 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 22 Aug 2018 16:45:25 -0400
Reported by Yogesh Sharma.
M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c
Enable -Wvla.
commit : 8a8738308c08ec9ac7a12b08f2ab7358dc4aa3af
author : David Steele <david@pgmasters.net>
date : Wed, 22 Aug 2018 14:48:37 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 22 Aug 2018 14:48:37 -0400
M doc/xml/release.xml
M src/Makefile
M test/lib/pgBackRestTest/Common/JobTest.pm
PostgreSQL 11 Beta 3 support.
commit : de5614db6b027900ef48f4ade456643cfecad79c
author : David Steele <david@pgmasters.net>
date : Wed, 15 Aug 2018 14:05:39 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 15 Aug 2018 14:05:39 -0400
M doc/xml/release.xml
M test/container.yaml
Allow any option to be set in an environment variable.
commit : d0b9f986a0eb30c550b2dc9031ad10dda941a26c
author : David Steele <david@pgmasters.net>
date : Wed, 15 Aug 2018 10:52:53 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 15 Aug 2018 10:52:53 -0400
This includes options that previously could only be specified on the command line, e.g. stanza.
M doc/xml/reference.xml
M doc/xml/release.xml
M doc/xml/user-guide.xml
M src/config/parse.c
M test/src/module/config/parseTest.c
Correct OptionInvalidError to OptionInvalidValueError in boolean option parsing.
commit : 4a822d3032389543d2bb53018cff9062155cd3f9
author : David Steele <david@pgmasters.net>
date : Wed, 15 Aug 2018 10:32:18 -0400
committer: David Steele <david@pgmasters.net>
date : Wed, 15 Aug 2018 10:32:18 -0400
M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c
Allow resolve array from cd5df357 to autosize to detect short arrays in testing.
commit : 238e5dae3c12ff37b3873e5db5a9c1cff3bed64f
author : David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 17:17:14 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 17:17:14 -0400
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M src/config/parse.auto.c
Add strReplaceChr() to String object.
commit : cb4b715533c1a566add50cb3e245cd2a65fba5fa
author : David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 16:49:38 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 16:49:38 -0400
M doc/xml/release.xml
M src/common/type/string.c
M src/common/type/string.h
M test/define.yaml
M test/src/module/common/typeStringTest.c
Add cvtCharToZ() and macro for debugging char params.
commit : 4a176681c3d7767c1f5cb98d1d5e46b7adb3f76b
author : David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 16:18:17 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 16:18:17 -0400
M doc/xml/release.xml
M src/common/debug.h
M src/common/type/convert.c
M src/common/type/convert.h
M test/define.yaml
M test/src/module/common/typeConvertTest.c
Define cipher magic size with sizeof() rather than using a constant.
commit : 9e3273fdf91e31d6f13ecd12658fa624dfbf4f3d
author : David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 16:08:58 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 16:08:58 -0400
M doc/xml/release.xml
M src/crypto/cipherBlock.c
M test/src/module/crypto/cipherBlockTest.c
Add gzip compression/decompression filters for C.
commit : 6643afe9a86bf617d2e90687130248c4a6ee6c14
author : David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 14:56:59 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 14:56:59 -0400
M doc/xml/release.xml
M libc/Makefile.PL
M src/Makefile
A src/compress/gzip.c
A src/compress/gzip.h
A src/compress/gzipCompress.c
A src/compress/gzipCompress.h
A src/compress/gzipDecompress.c
A src/compress/gzipDecompress.h
M test/Vagrantfile
M test/define.yaml
M test/lib/pgBackRestTest/Common/JobTest.pm
A test/src/module/compress/gzipTest.c
Filters can now produce output that differs from input.
commit : e3ff6b209d9cf50f88f51a70ac7247949b63ac55
author : David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 14:21:53 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 14:21:53 -0400
This allows filters such as compression, encryption, etc. to be implemented.
M doc/xml/release.xml
M libc/Makefile.PL
M src/Makefile
M src/common/io/bufferRead.c
M src/common/io/bufferRead.h
M src/common/io/bufferWrite.c
M src/common/io/bufferWrite.h
A src/common/io/filter/buffer.c
A src/common/io/filter/buffer.h
M src/common/io/filter/filter.c
M src/common/io/filter/filter.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
A src/common/io/filter/size.c
A src/common/io/filter/size.h
M src/common/io/handle.h
M src/common/io/io.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/write.c
M src/common/io/write.h
M src/crypto/hash.c
M src/crypto/hash.h
M test/define.yaml
M test/src/module/common/ioTest.c
M test/src/module/storage/fileTest.c
M test/src/module/storage/storageTest.c
Update to command-line docs missed in 40924e8f.
commit : a45d1b4a60545940a6a5eac37cf16473b97d2c41
author : David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 13:38:11 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 14 Aug 2018 13:38:11 -0400
M src/config/define.auto.c
Fix invalid log level in log-path option reference.
commit : 40924e8f5c84cfdac7eadde4b6cca092f3900430
author : David Steele <david@pgmasters.net>
date : Mon, 13 Aug 2018 07:47:26 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 13 Aug 2018 07:47:26 -0400
Reported by Camilo Aguilar.
M doc/xml/reference.xml
M doc/xml/release.xml
Remove incompletely implemented online option from the check command.
commit : 343424009795014bce2474f46762d76cc9ec4ef7
author : David Steele <david@pgmasters.net>
date : Sun, 12 Aug 2018 19:24:21 -0400
committer: David Steele <david@pgmasters.net>
date : Sun, 12 Aug 2018 19:24:21 -0400
Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly.
Reported by Jason O'Donnell.
M build/lib/pgBackRestBuild/Config/Data.pm
M doc/xml/reference.xml
M doc/xml/release.xml
M lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Stanza.pm
M src/config/define.auto.c
M src/perl/embed.auto.c
M test/lib/pgBackRestTest/Module/Stanza/StanzaAllPerlTest.pm
Stop trying to arrange contributors in release.xml by last/first name.
commit : cb6821e8533a9ff3c4e3727746bd1d259255d86a
author : David Steele <david@pgmasters.net>
date : Sun, 12 Aug 2018 18:19:34 -0400
committer: David Steele <david@pgmasters.net>
date : Sun, 12 Aug 2018 18:19:34 -0400
Contributor names have always been presented in the release notes exactly as given, but we tried to assign internal IDs based on last/first name which can be hard to determine and ultimately doesn't make sense.
Inspired by Christophe Pettus' PostgresOpen 2017 talk, "Human Beings Do Not Have a Primary Key".
M doc/xml/release.xml
Validate configuration options in a single pass.
commit : cd5df3570b0fcd5011e2339b48a3359a1f2c54da
author : David Steele <david@pgmasters.net>
date : Sat, 11 Aug 2018 12:55:33 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 11 Aug 2018 12:55:33 -0400
By pre-calculating and storing the option dependencies in parse.auto.c validation can be completed in a single pass, which is both simpler and faster.
M build/lib/pgBackRestBuild/Config/BuildParse.pm
M doc/xml/release.xml
M src/config/parse.auto.c
M src/config/parse.c
M test/src/module/config/parseTest.c
Improve error message when a command is missing the stanza option.
commit : f06bf9e8329fcb083271fab8d7aeccad1122a5b1
author : David Steele <david@pgmasters.net>
date : Sat, 11 Aug 2018 09:47:07 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 11 Aug 2018 09:47:07 -0400
Suggested by Sarah Conway.
M doc/xml/release.xml
M src/config/parse.c
M test/src/module/config/parseTest.c
Migrate minimum set of code for reading archive.info files from Perl to C.
commit : 8ab2e72960de60557331fe7087015519a516f318
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Thu, 9 Aug 2018 08:57:21 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 9 Aug 2018 08:57:21 -0400
Contributed by Cynthia Shang.
M doc/xml/release.xml
M src/Makefile
A src/info/info.c
A src/info/info.h
A src/info/infoArchive.c
A src/info/infoArchive.h
A src/info/infoPg.c
A src/info/infoPg.h
M src/version.h
M test/define.yaml
A test/src/module/info/infoArchiveTest.c
A test/src/module/info/infoPgTest.c
A test/src/module/info/infoTest.c
Add basic C JSON parser.
commit : 7993f1a96620ac6c00ba961036ba5ff6abc58f22
author : David Steele <david@pgmasters.net>
date : Thu, 9 Aug 2018 08:06:23 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 9 Aug 2018 08:06:23 -0400
M build/error.yaml
M doc/xml/release.xml
M lib/pgBackRest/Common/ExceptionAuto.pm
M src/Makefile
M src/common/error.auto.c
M src/common/error.auto.h
A src/common/type/json.c
A src/common/type/json.h
M src/perl/embed.auto.c
M test/define.yaml
A test/src/module/common/typeJsonTest.c
Enable -Wduplicated-branches, and -Wduplicated-cond.
commit : 31167d8f987cf171bd655c5ccb1bebebac9d7997
author : David Steele <david@pgmasters.net>
date : Sun, 5 Aug 2018 08:35:18 -0400
committer: David Steele <david@pgmasters.net>
date : Sun, 5 Aug 2018 08:35:18 -0400
M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm
Enable -Wpointer-arith.
commit : 14f21a8f6c0f1bec4c4af82af51235ddf8617beb
author : David Steele <david@pgmasters.net>
date : Sat, 4 Aug 2018 19:37:28 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 4 Aug 2018 19:37:28 -0400
M doc/xml/release.xml
M src/Makefile
M test/lib/pgBackRestTest/Common/JobTest.pm
Enable -Wstrict-prototypes and update all void functions to conform.
commit : 429a356e3323acf99f3d73a527baf7aaa55a3b5b
author : David Steele <david@pgmasters.net>
date : Fri, 3 Aug 2018 19:19:14 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 3 Aug 2018 19:19:14 -0400
M doc/xml/release.xml
M src/Makefile
M src/command/archive/get/get.c
M src/command/archive/get/get.h
M src/command/archive/push/push.c
M src/command/archive/push/push.h
M src/command/command.c
M src/command/command.h
M src/command/help/help.c
M src/command/help/help.h
M src/common/error.c
M src/common/error.h
M src/common/exit.c
M src/common/exit.h
M src/common/fork.c
M src/common/fork.h
M src/common/ini.c
M src/common/ini.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/io.c
M src/common/io/io.h
M src/common/memContext.c
M src/common/memContext.h
M src/common/stackTrace.c
M src/common/stackTrace.h
M src/common/time.c
M src/common/time.h
M src/common/type/keyValue.c
M src/common/type/keyValue.h
M src/common/type/stringList.c
M src/common/type/stringList.h
M src/common/type/variant.c
M src/common/type/variant.h
M src/common/type/variantList.c
M src/common/type/variantList.h
M src/config/config.c
M src/config/config.h
M src/config/define.c
M src/config/define.h
M src/config/load.c
M src/config/load.h
M src/crypto/crypto.c
M src/crypto/crypto.h
M src/perl/config.c
M src/perl/config.h
M src/perl/exec.c
M src/perl/exec.h
M src/storage/helper.c
M src/storage/helper.h
M src/storage/storage.c
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/common/harnessTest.c
M test/src/common/harnessTest.h
M test/src/module/archive/commonTest.c
M test/src/module/archive/getTest.c
M test/src/module/archive/pushTest.c
M test/src/module/command/commandTest.c
M test/src/module/common/assertOffTest.c
M test/src/module/common/assertOnTest.c
M test/src/module/common/debugOffTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/encodeTest.c
M test/src/module/common/errorTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/forkTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/lockTest.c
M test/src/module/common/logTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/regExpTest.c
M test/src/module/common/stackTraceTest.c
M test/src/module/common/timeTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeStringListTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/common/typeVariantListTest.c
M test/src/module/common/typeVariantTest.c
M test/src/module/common/typecTest.c
M test/src/module/common/waitTest.c
M test/src/module/config/configTest.c
M test/src/module/config/defineTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/crypto/cipherBlockTest.c
M test/src/module/crypto/cryptoTest.c
M test/src/module/crypto/hashTest.c
M test/src/module/crypto/randomTest.c
M test/src/module/help/helpTest.c
M test/src/module/perl/configTest.c
M test/src/module/perl/execTest.c
M test/src/module/postgres/infoTest.c
M test/src/module/postgres/pageChecksumTest.c
M test/src/module/storage/fileTest.c
M test/src/module/storage/helperTest.c
M test/src/module/storage/storageTest.c
Exclude temporary and unlogged relation (table/index) files from backup.
commit : bec4c176dc478d5eff855c18ba05d4a8301f0b22
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Mon, 30 Jul 2018 18:53:34 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 30 Jul 2018 18:53:34 -0400
Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced.
Contributed by Cynthia Shang.
M doc/xml/release.xml
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm
Add cvtBoolToConstZ() to simplify conversion of boolean to string.
commit : ac16cb7ff16b58c9de4ba9c27d5322b69639594e
author : David Steele <david@pgmasters.net>
date : Sat, 28 Jul 2018 09:39:52 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 28 Jul 2018 09:39:52 -0400
M doc/xml/release.xml
M src/common/type/convert.c
M src/common/type/convert.h
M src/storage/storage.c
Do nothing in memContextMove() when the context is already in the specified parent.
commit : 7a00bbebfafc94f17b7897b2074cfd079e6882b6
author : David Steele <david@pgmasters.net>
date : Sat, 28 Jul 2018 09:04:06 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 28 Jul 2018 09:04:06 -0400
M doc/xml/release.xml
M src/common/memContext.c
M test/src/module/common/memContextTest.c
Remove redundant lines from embedded Perl by combining blank lines.
commit : cb975a533095f863232843cac1b3831f24a8fe3c
author : David Steele <david@pgmasters.net>
date : Sat, 28 Jul 2018 08:43:58 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 28 Jul 2018 08:43:58 -0400
M build/lib/pgBackRestBuild/Embed/Build.pm
M doc/xml/release.xml
M src/perl/embed.auto.c
Update code count for new file types and exclusions.
commit : 666537cbdee986ebdc323f6c9d412edf2a5c7145
author : David Steele <david@pgmasters.net>
date : Sat, 28 Jul 2018 08:13:30 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 28 Jul 2018 08:13:30 -0400
M doc/xml/release.xml
M test/code-count/code.def
M test/code-count/file-type.yaml
M test/lib/pgBackRestTest/Common/CodeCountTest.pm
M test/src/module/config/parseTest.c
Implement filters that do not modify the buffer.
commit : 01aea0c067c2e667c47346620c563218f4c0f568
author : David Steele <david@pgmasters.net>
date : Tue, 24 Jul 2018 21:08:27 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 24 Jul 2018 21:08:27 -0400
Update cryptoHash to use the new interface.
M doc/xml/release.xml
M libc/Makefile.PL
M src/Makefile
A src/common/io/filter/filter.c
A src/common/io/filter/filter.h
A src/common/io/filter/group.c
A src/common/io/filter/group.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/write.c
M src/common/io/write.h
M src/crypto/hash.c
M src/crypto/hash.h
M src/storage/fileRead.c
M src/storage/fileRead.h
M src/storage/storage.c
M src/storage/storage.h
M test/define.yaml
M test/src/common/harnessTest.h
M test/src/module/common/ioTest.c
M test/src/module/crypto/hashTest.c
M test/src/module/storage/fileTest.c
M test/src/module/storage/storageTest.c
Move most host setup to containers defined in the documentation.
commit : 8e1017c1622bed5ebddc910545cbcaf288dd2f2f
author : David Steele <david@pgmasters.net>
date : Mon, 23 Jul 2018 07:59:24 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 23 Jul 2018 07:59:24 -0400
This includes PostgreSQL installation which had previously been included in the documentation. This way produces faster builds and there is no need for us to document PostgreSQL installation.
M doc/xml/release.xml
M doc/xml/user-guide.xml
Allow containers to be defined in a document.
commit : 27238310322096a52f1babf478cd4fffb344d949
author : David Steele <david@pgmasters.net>
date : Mon, 23 Jul 2018 07:49:15 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 23 Jul 2018 07:49:15 -0400
The defined containers are built before the document build begins which allows them to be reused.
M doc/doc.pl
M doc/lib/BackRestDoc/Common/DocManifest.pm
M doc/xml/dtd/doc.dtd
M doc/xml/release.xml
Fix typo in 18626306.
commit : ae72772e5b366f02f8a5026e3109cfd43aacb8f4
author : David Steele <david@pgmasters.net>
date : Mon, 23 Jul 2018 07:17:24 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 23 Jul 2018 07:17:24 -0400
M test/container.yaml
Use pre-built images from Docker Hub when the container definition has not changed.
commit : 1862630629bdf435a76575a31ed722cc351def11
author : David Steele <david@pgmasters.net>
date : Sat, 21 Jul 2018 17:02:42 -0400
committer: David Steele <david@pgmasters.net>
date : Sat, 21 Jul 2018 17:02:42 -0400
Downloading an image is quite a bit faster than building a new image from scratch and saves minutes per test run in CI.
M .travis.yml
M doc/xml/release.xml
M test/Vagrantfile
A test/container.yaml
M test/lib/pgBackRestTest/Common/CiTest.pm
M test/lib/pgBackRestTest/Common/ContainerTest.pm
M test/lib/pgBackRestTest/Common/VmTest.pm
Add --log-level-test option.
commit : 8568622a6f5941cbe9e0ef40ea9f605b62675838
author : David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 19:03:46 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 19:03:46 -0400
This allows setting the test log level independently from the general test harness setting, but current only works for the C tests. It is useful for seeing log output from functions on the console while a test is running.
M doc/xml/release.xml
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/common/harnessLog.c
M test/src/common/harnessLog.h
M test/src/module/command/commandTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/exitTest.c
M test/src/module/config/parseTest.c
M test/src/test.c
M test/test.pl
Refactor the common/log tests to not depend on common/harnessLog.
commit : 58e9f1e50cfbc2df22af17293fe8a298cac5513a
author : David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 18:51:42 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 18:51:42 -0400
common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way.
Add a few exceptions for config testing since the log levels are reset by default in config/parse.
M doc/xml/release.xml
M libc/xs/config/configTest.xs
M src/common/log.c
M src/common/log.h
M src/config/load.c
M src/config/parse.c
M src/config/parse.h
M src/perl/libc.auto.c
M test/code-count/file-type.yaml
M test/define.yaml
M test/src/common/harnessConfig.c
R077 test/src/common/logTest.c test/src/common/harnessLog.c
R052 test/src/common/logTest.h test/src/common/harnessLog.h
M test/src/common/harnessTest.c
M test/src/module/archive/commonTest.c
M test/src/module/archive/getTest.c
M test/src/module/archive/pushTest.c
M test/src/module/command/commandTest.c
M test/src/module/common/debugOnTest.c
M test/src/module/common/exitTest.c
M test/src/module/common/logTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/help/helpTest.c
M test/src/module/perl/execTest.c
M test/src/module/storage/storageTest.c
M test/src/test.c
Fix issue where errors raised in C were not logged when called from Perl.
commit : 1359e2908c777ef7b8cb128f7d7ede3b53bc2018
author : David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 08:11:34 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 08:11:34 -0400
pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging.
Reported by Douglas J Hunley.
M doc/xml/release.xml
M lib/pgBackRest/Common/Exception.pm
M lib/pgBackRest/Main.pm
M src/perl/embed.auto.c
M src/perl/exec.c
M test/src/module/perl/execTest.c
Rename error-handling variables in Main.pm to conform to standard.
commit : d3cfeebdf91a559a0af80bba5fc424f4d091b944
author : David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 08:03:44 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 08:03:44 -0400
M doc/xml/release.xml
M lib/pgBackRest/Main.pm
M src/perl/embed.auto.c
M src/perl/exec.c
M test/src/module/perl/execTest.c
Show exact log level required for stack trace param output instead of just "debug".
commit : fa53e2eddb95822553b6a860ef3e6d09fdb31f4a
author : David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 07:15:28 -0400
committer: David Steele <david@pgmasters.net>
date : Fri, 20 Jul 2018 07:15:28 -0400
M doc/xml/release.xml
M src/common/stackTrace.c
M test/src/module/common/stackTraceTest.c
Abstract IO layer out of the storage layer.
commit : 0ac176b722fe93301ec9f4d706698f0f01125666
author : David Steele <david@pgmasters.net>
date : Thu, 19 Jul 2018 16:04:20 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 19 Jul 2018 16:04:20 -0400
This allows the routines to be used for IO objects that do not have a storage representation.
Implement buffer read and write IO objects.
M doc/xml/release.xml
M libc/Makefile.PL
M src/Makefile
A src/common/io/bufferRead.c
A src/common/io/bufferRead.h
A src/common/io/bufferWrite.c
A src/common/io/bufferWrite.h
A src/common/io/io.c
A src/common/io/io.h
A src/common/io/read.c
A src/common/io/read.h
A src/common/io/write.c
A src/common/io/write.h
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/string.c
M src/config/load.c
M src/storage/driver/posix/driverRead.c
M src/storage/driver/posix/driverRead.h
M src/storage/fileRead.c
M src/storage/fileRead.h
M src/storage/fileWrite.c
M src/storage/fileWrite.h
M src/storage/helper.c
M src/storage/storage.c
M src/storage/storage.h
M test/define.yaml
M test/src/common/harnessTest.h
D test/src/module/common/ioHandleTest.c
A test/src/module/common/ioTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/common/typeStringTest.c
M test/src/module/config/loadTest.c
M test/src/module/storage/fileTest.c
M test/src/module/storage/storageTest.c
storageFileRead() accepts a buffer for output rather than creating one.
commit : 5dc8a2ec08ee7569063915c7a1b3399ac1f2d16c
author : David Steele <david@pgmasters.net>
date : Tue, 17 Jul 2018 19:01:54 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 17 Jul 2018 19:01:54 -0400
This is more efficient overall and allows the caller to specify how many bytes will be read on each call. Reads are appended if the buffer already contains data but the buffer size will never increase.
Allow Buffer object "used size" to be different than "allocated size". Add functions to manage used size and remaining size and update automatically when possible.
M doc/xml/release.xml
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/postgres/info.c
M src/storage/driver/posix/driverRead.c
M src/storage/driver/posix/driverRead.h
M src/storage/driver/posix/driverWrite.c
M src/storage/fileRead.c
M src/storage/fileRead.h
M src/storage/storage.c
M src/storage/storage.h
M test/lib/pgBackRestTest/Common/JobTest.pm
M test/src/module/archive/getTest.c
M test/src/module/common/typeBufferTest.c
M test/src/module/postgres/infoTest.c
M test/src/module/storage/fileTest.c
M test/src/module/storage/storageTest.c
Require PostgreSQL catalog version when instantiating a Manifest object (and not loading it from disk).
commit : 0acf70541601d99795f6da06ebca030eff69cd46
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Mon, 16 Jul 2018 17:25:15 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 16 Jul 2018 17:25:15 -0400
Contributed by Cynthia Shang.
M doc/xml/release.xml
M lib/pgBackRest/Backup/Backup.pm
M lib/pgBackRest/Check/Check.pm
M lib/pgBackRest/Manifest.pm
M src/perl/embed.auto.c
M test/expect/mock-all-001.log
M test/expect/mock-all-002.log
M test/lib/pgBackRestTest/Env/ExpireEnvTest.pm
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Env/HostEnvTest.pm
M test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/lib/pgBackRestTest/Module/Stanza/StanzaAllPerlTest.pm
Add iniSectionList() to Ini object and remove dead code.
commit : 4e38cbaea9fdfbc8618971c5148ffd0a7e7ef4ab
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Thu, 12 Jul 2018 15:28:46 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 12 Jul 2018 15:28:46 -0400
Contributed by Cynthia Shang.
M doc/xml/release.xml
M src/common/ini.c
M src/common/ini.h
M test/src/module/common/iniTest.c
Add uint64 variant type and supporting conversion functions.
commit : 0e6b927a172368a77473571d8c27265253894c0e
author : Cynthia Shang <cynthia.shang@crunchydata.com>
date : Thu, 12 Jul 2018 15:23:18 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 12 Jul 2018 15:23:18 -0400
Contributed by Cynthia Shang.
Reviewed by Stephen Frost.
M doc/xml/release.xml
M src/common/type/convert.c
M src/common/type/convert.h
M src/common/type/variant.c
M src/common/type/variant.h
M test/define.yaml
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeVariantTest.c
Improve performance of string to int conversion.
commit : 0e331b12ba0cf377e4292a89ae84de7bb74281d6
author : David Steele <david@pgmasters.net>
date : Thu, 12 Jul 2018 15:15:32 -0400
committer: David Steele <david@pgmasters.net>
date : Thu, 12 Jul 2018 15:15:32 -0400
Use strtoll() instead of sprintf() for conversion. Also use available integer min/max constants rather than hard-coded values.
Reviewed by Stephen Frost.
Suggested by Stephen Frost.
M doc/xml/release.xml
M src/common/type/convert.c
M src/common/type/variant.c
M test/src/module/common/typeConvertTest.c
M test/src/module/common/typeVariantTest.c
Error if LibC build is performed outside test environment.
commit : cf889790e60972adcacc6e21fa3853378676889a
author : David Steele <david@pgmasters.net>
date : Tue, 10 Jul 2018 15:39:03 -0400
committer: David Steele <david@pgmasters.net>
date : Tue, 10 Jul 2018 15:39:03 -0400
LibC is no longer required for production builds.
M doc/xml/release.xml
M libc/Makefile.PL
M test/test.pl
Begin v2.05 development.
commit : b1bc53657d9e683f30bdfcce190da286a383328f
author : David Steele <david@pgmasters.net>
date : Mon, 9 Jul 2018 08:15:16 -0400
committer: David Steele <david@pgmasters.net>
date : Mon, 9 Jul 2018 08:15:16 -0400
M doc/xml/release.xml
M lib/pgBackRest/Version.pm
M src/perl/embed.auto.c
M src/version.h