pgBackRest 2.37 (upcoming) commit log

Revert Azurite version for testing to 3.14.0.

commit   : c38d6926d6c9aa01b895a28e66fc0aa6965350a3    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 9 Sep 2021 08:48:45 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 9 Sep 2021 08:48:45 -0400    

Click here for diff

3.14.2 is causing breakage in the documentation. There is no obvious cause so for now just revert to the last working version.  

M doc/xml/user-guide.xml
M test/lib/pgBackRestTest/Env/Host/HostAzureTest.pm

Migrate command-line help generation to C.

commit   : f4e1babf6b4ce7087ace8221cac7cadb51488f0e    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 8 Sep 2021 18:16:06 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 8 Sep 2021 18:16:06 -0400    

Click here for diff

Command-line help is now generated at build time so it does not need to be committed. This reduces churn on commits that add configuration and/or update the help.  
  
Since churn is no longer an issue, help.auto.c is bzip2 compressed to save space in the binary.  
  
The Perl config parser (Data.pm) has been moved to doc/lib since the Perl build path is no longer required.  
  
Likewise doc/xml/reference.xml has been moved to src/build/help/help.xml since it is required at build time.  

M CONTRIBUTING.md
D build/lib/pgBackRestBuild/Build.pm
D build/lib/pgBackRestBuild/Build/Common.pm
D build/lib/pgBackRestBuild/Config/BuildHelp.pm
M doc/doc.pl
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/lib/pgBackRestDoc/Common/DocManifest.pm
M doc/lib/pgBackRestDoc/Common/DocRender.pm
R099 build/lib/pgBackRestBuild/Config/Data.pm doc/lib/pgBackRestDoc/Custom/DocConfigData.pm
M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
M doc/manifest.xml
M doc/xml/contributing.xml
M src/.gitignore
M src/Makefile.in
M src/build/config/parse.c
M src/build/config/parse.h
R100 doc/xml/reference.xml src/build/help/help.xml
A src/build/help/main.c
A src/build/help/parse.c
A src/build/help/parse.h
A src/build/help/render.c
A src/build/help/render.h
D src/command/help/help.auto.c
M src/command/help/help.c
M src/command/help/help.h
M src/main.c
M test/code-count/file-type.yaml
M test/define.yaml
A test/src/module/build/helpTest.c
M test/src/module/command/helpTest.c
M test/test.pl

Eliminate linefeed formatting from documentation.

commit   : def7d513cdd2d4579acf6e8c675a3d6f7da4f655    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 8 Sep 2021 17:35:45 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 8 Sep 2021 17:35:45 -0400    

Click here for diff

Linefeeds were originally used in the place of <p> tags to denote a paragraph. While much of the linefeed usage has been replaced over time, there were many places where it was still being used, especially in reference.xml. This made it difficult to get consistent formatting across different output types. In particular there were formatting issues in the command-line help because it is harder to audit than HTML or PDF.  
  
Replace linefeed formatting with proper <p> tags to make formatting more consistent.  
  
Remove double spaces in all text where <p> tags were added since it does not add churn.  
  
Update all <ul>/<ol>/<li> tags to the more general <list>/<list-item> tags.  
  
Add a few missing periods.  

M doc/lib/pgBackRestDoc/Common/Doc.pm
M doc/lib/pgBackRestDoc/Common/DocRender.pm
M doc/lib/pgBackRestDoc/Custom/DocCustomRelease.pm
M doc/lib/pgBackRestDoc/Html/DocHtmlBuilder.pm
M doc/xml/dtd/doc.dtd
M doc/xml/index.xml
M doc/xml/reference.xml
M doc/xml/release.xml
M src/command/help/help.auto.c

Add initializer braces required by older versions of gcc.

commit   : 245b0c048ae62eb174b534d156b7a00b1de19a53    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 8 Sep 2021 13:32:13 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 8 Sep 2021 13:32:13 -0400    

Click here for diff

M src/build/common/yaml.c

Remove invalid comment block.

commit   : 38eeac48d617ec5d5f14a1cb612c9a8b721326d8    
  
author   : David Steele <david@pgmasters.net>    
date     : Fri, 3 Sep 2021 10:39:30 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Fri, 3 Sep 2021 10:39:30 -0400    

Click here for diff

M src/common/io/tls/session.h

Build newer valgrind and move CA cert install in test containers.

commit   : 1afea449e2f54e69f9f8f0d772a294fcf09e0d83    
  
author   : David Steele <david@pgmasters.net>    
date     : Thu, 2 Sep 2021 13:26:21 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Thu, 2 Sep 2021 13:26:21 -0400    

Click here for diff

The newer version of valgrind helps with some arm64 issues that have been fixed since the architecture has become more popular. Also add the valgrind builds to the Vagrantfile and Dockerfile.  
  
Move the CA cert install from the base container to the test container. This means the CA cert can be changed without rebuilding all the base containers.  

M test/Dockerfile
M test/Vagrantfile
M test/container.yaml
M test/lib/pgBackRestTest/Common/ContainerTest.pm

Allow additional memory to be allocated with a mem context.

commit   : 475b57c89b58e9709a3eafd08fe9949b467a99cd    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 1 Sep 2021 11:10:35 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 1 Sep 2021 11:10:35 -0400    

Click here for diff

The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer.  
  
Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size.  
  
Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall.  
  
Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.  

M CONTRIBUTING.md
M doc/xml/contributing.xml
M src/build/common/yaml.c
M src/command/backup/pageChecksum.c
M src/common/compress/bz2/compress.c
M src/common/compress/bz2/decompress.c
M src/common/compress/gz/compress.c
M src/common/compress/gz/decompress.c
M src/common/compress/lz4/compress.c
M src/common/compress/lz4/decompress.c
M src/common/compress/zst/compress.c
M src/common/compress/zst/decompress.c
M src/common/crypto/cipherBlock.c
M src/common/crypto/hash.c
M src/common/exec.c
M src/common/exec.h
M src/common/ini.c
M src/common/io/bufferRead.c
M src/common/io/bufferWrite.c
M src/common/io/client.c
M src/common/io/client.h
M src/common/io/fdRead.c
M src/common/io/fdWrite.c
M src/common/io/filter/buffer.c
M src/common/io/filter/filter.h
M src/common/io/filter/filter.intern.h
M src/common/io/filter/group.c
M src/common/io/filter/group.h
M src/common/io/filter/sink.c
M src/common/io/filter/size.c
M src/common/io/http/client.c
M src/common/io/http/header.c
M src/common/io/http/query.c
M src/common/io/http/request.c
M src/common/io/http/request.h
M src/common/io/http/response.c
M src/common/io/http/response.h
M src/common/io/http/session.c
M src/common/io/http/url.c
M src/common/io/http/url.h
M src/common/io/read.c
M src/common/io/read.h
M src/common/io/session.c
M src/common/io/session.h
M src/common/io/socket/client.c
M src/common/io/socket/session.c
M src/common/io/tls/client.c
M src/common/io/tls/session.c
M src/common/io/write.c
M src/common/io/write.h
M src/common/memContext.c
M src/common/memContext.h
M src/common/regExp.c
M src/common/type/buffer.c
M src/common/type/buffer.h
M src/common/type/keyValue.c
M src/common/type/keyValue.h
M src/common/type/list.c
M src/common/type/list.h
M src/common/type/mcv.c
M src/common/type/object.c
M src/common/type/object.h
M src/common/type/pack.c
M src/common/type/xml.c
M src/common/type/xml.h
M src/common/wait.c
M src/common/wait.h
M src/config/parse.c
M src/db/db.c
M src/info/info.c
M src/info/infoArchive.c
M src/info/infoBackup.c
M src/info/infoPg.c
M src/info/manifest.c
M src/postgres/client.c
M src/protocol/client.c
M src/protocol/client.h
M src/protocol/command.c
M src/protocol/parallel.c
M src/protocol/parallelJob.c
M src/protocol/parallelJob.h
M src/protocol/server.c
M src/storage/azure/read.c
M src/storage/azure/storage.c
M src/storage/azure/write.c
M src/storage/cifs/storage.c
M src/storage/gcs/read.c
M src/storage/gcs/storage.c
M src/storage/gcs/write.c
M src/storage/posix/read.c
M src/storage/posix/storage.c
M src/storage/posix/write.c
M src/storage/read.h
M src/storage/remote/protocol.c
M src/storage/remote/read.c
M src/storage/remote/storage.c
M src/storage/remote/write.c
M src/storage/s3/read.c
M src/storage/s3/storage.c
M src/storage/s3/write.c
M src/storage/write.h
M test/src/common/harnessProtocol.c
M test/src/module/command/backupTest.c
M test/src/module/command/restoreTest.c
M test/src/module/common/cryptoTest.c
M test/src/module/common/execTest.c
M test/src/module/common/iniTest.c
M test/src/module/common/ioTest.c
M test/src/module/common/memContextTest.c
M test/src/module/common/typeKeyValueTest.c
M test/src/module/common/typeListTest.c
M test/src/module/common/typeObjectTest.c
M test/src/module/info/manifestTest.c
M test/src/module/performance/storageTest.c
M test/src/module/performance/typeTest.c
M test/src/module/protocol/protocolTest.c
M test/src/module/storage/remoteTest.c

Increase max index allowed for pg/repo options to 256.

commit   : 02b06aa4957cf3fe64ceb7a44cc1893bff12fe99    
  
author   : David Steele <dwsteele@users.noreply.github.com>    
date     : Tue, 31 Aug 2021 12:09:50 -0400    
  
committer: GitHub <noreply@github.com>    
date     : Tue, 31 Aug 2021 12:09:50 -0400    

Click here for diff

The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained.
  

  
Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly.
  

  
Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX.
  

  
Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior.
  

  
cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.

M build/lib/pgBackRestBuild/Config/Data.pm
M doc/lib/pgBackRestDoc/Common/DocConfig.pm
M doc/lib/pgBackRestDoc/Common/DocExecute.pm
M doc/xml/release.xml
M src/build/config/config.yaml
M src/build/config/parse.c
M src/build/config/parse.h
M src/build/config/render.c
M src/command/help/help.c
M src/config/parse.auto.c
M src/config/parse.c
M src/config/parse.h
M test/expect/real-all-001.log
M test/lib/pgBackRestTest/Env/Host/HostBackupTest.pm
M test/lib/pgBackRestTest/Module/Real/RealAllTest.pm
M test/src/module/build/configTest.c
M test/src/module/config/loadTest.c
M test/src/module/config/parseTest.c
M test/src/module/storage/remoteTest.c

Add required header for Illumos.

commit   : 8a4063c2b69919d1bba305d986568f529139eaa3    
  
author   : John W. Terrell <john.terrell@outlook.com>    
date     : Fri, 27 Aug 2021 03:41:56 -0700    
  
committer: GitHub <noreply@github.com>    
date     : Fri, 27 Aug 2021 03:41:56 -0700    

Click here for diff

M src/common/io/fd.c

Do not use certs in container for testing.

commit   : 5d857e196a6eeb2e1d137f2faf9001cccc066cfd    
  
author   : David Steele <david@pgmasters.net>    
date     : Wed, 25 Aug 2021 07:33:31 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Wed, 25 Aug 2021 07:33:31 -0400    

Click here for diff

The certs are available in test/certificate so it makes more sense to use them there. In addition the container does not need to be rebuilt unless the CA cert changes.  

M test/src/common/harnessServer.c

Add missing header.

commit   : 7b4d69211e64855f9ca0db0e4b2950a6492afa59    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 16:09:14 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 16:09:14 -0400    

Click here for diff

M src/common/io/tls/client.h

Add const to memContextFreeing()/memContextName() parameters.

commit   : beafdbfc1873902f4b17fbbc5f88e47d6344d113    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 08:37:57 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 08:37:57 -0400    

Click here for diff

These functions do not modify the mem context so they can use const pointers.  

M src/common/memContext.c
M src/common/memContext.h

Remove useless if in memContextFree().

commit   : 726d04f13a2217b78e2cbef48ef4c889f15b5b68    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 08:31:50 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 08:31:50 -0400    

Click here for diff

If the upper bound of the loop is 0 then nothing will be done, making the if condition extraneous.  

M src/common/memContext.c

Use contextParentIdx instead of a loop when moving a context.

commit   : d72d4415a7c79296e3160aede9df59cd5d959694    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 08:27:07 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 08:27:07 -0400    

Click here for diff

contextParentIdx was introduced in 90709dfd to improve the performance of mem context frees. memContextMove() did not get the message, however, and continued to use a loop to find the mem context in the old parent.  
  
Use contextParentIdx to find the mem context in the old parent to avoid a loop.  

M src/common/memContext.c
M test/src/module/common/memContextTest.c

Remove useless memFree() in regExpNew().

commit   : 8a9887f27cb46728e85801e075474928207e012f    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 07:58:59 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 07:58:59 -0400    

Click here for diff

The MEM_CONTEXT_NEW*() block will automatically free memory if there is an error.  
  
This code is pretty old so this may not have been true when it was written.  

M src/common/regExp.c

Update config.guess and config.sub to latest versions.

commit   : 5ffb7cc77127a60de98e2d9080f27ed28bf95b4f    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 07:14:22 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 07:14:22 -0400    

Click here for diff

M doc/RELEASE.md
M src/build/config.guess
M src/build/config.sub

Begin v2.36 development.

commit   : aee0e7bac746c5c68baee106d8d4809cf6d8edfd    
  
author   : David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 07:03:40 -0400    
  
committer: David Steele <david@pgmasters.net>    
date     : Mon, 23 Aug 2021 07:03:40 -0400    

Click here for diff

M doc/resource/git-history.cache
M doc/xml/release.xml
M src/build/configure.ac
M src/configure
M src/version.h