PostgreSQL 9.5.18 (upcoming) commit log

Fix some grammar in documentation of spgist and pgbench

commit   : cd703f5e17d50f105e110c99165ec734ae997133    
  
author   : Michael Paquier <michael@paquier.xyz>    
date     : Mon, 20 May 2019 09:48:50 +0900    
  
committer: Michael Paquier <michael@paquier.xyz>    
date     : Mon, 20 May 2019 09:48:50 +0900    

Click here for diff

Discussion: https://postgr.es/m/92961161-9b49-e42f-0a72-d5d47e0ed4de@postgrespro.ru  
Author: Liudmila Mantrova  
Reviewed-by: Jonathan Katz, Tom Lane, Michael Paquier  
Backpatch-through: 9.4  

M doc/src/sgml/ref/pgbench.sgml
M doc/src/sgml/spgist.sgml

Revert “In the pg_upgrade test suite, don’t write to src/test/regress.”

commit   : 2604751c73e59a1dd148361345b643676c9e4110    
  
author   : Noah Misch <noah@leadboat.com>    
date     : Sun, 19 May 2019 15:24:42 -0700    
  
committer: Noah Misch <noah@leadboat.com>    
date     : Sun, 19 May 2019 15:24:42 -0700    

Click here for diff

This reverts commit bd1592e8570282b1650af6b8eede0016496daecd.  It had  
multiple defects.  
  
Discussion: https://postgr.es/m/12717.1558304356@sss.pgh.pa.us  

M src/bin/pg_upgrade/test.sh
M src/test/regress/input/largeobject.source
M src/test/regress/output/largeobject.source
M src/test/regress/output/largeobject_1.source
M src/tools/msvc/vcregress.pl

In the pg_upgrade test suite, don’t write to src/test/regress.

commit   : e00b5f090fa04656dd1929786720f500741efbac    
  
author   : Noah Misch <noah@leadboat.com>    
date     : Sun, 19 May 2019 14:36:44 -0700    
  
committer: Noah Misch <noah@leadboat.com>    
date     : Sun, 19 May 2019 14:36:44 -0700    

Click here for diff

When this suite runs installcheck, redirect file creations from  
src/test/regress to src/bin/pg_upgrade/tmp_check/regress.  This closes a  
race condition in "make -j check-world".  If the pg_upgrade suite wrote  
to a given src/test/regress/results file in parallel with the regular  
src/test/regress invocation writing it, a test failed spuriously.  Even  
without parallelism, in "make -k check-world", the suite finishing  
second overwrote the other's regression.diffs.  This revealed test  
"largeobject" assuming @abs_builddir@ is getcwd(), so fix that, too.  
  
Buildfarm client REL_10, released forty-five days ago, supports saving  
regression.diffs from its new location.  When an older client reports a  
pg_upgradeCheck failure, it will no longer include regression.diffs.  
Back-patch to 9.5, where pg_upgrade moved to src/bin.  
  
Reviewed by Andrew Dunstan.  
  
Discussion: https://postgr.es/m/20181224034411.GA3224776@rfd.leadboat.com  

M src/bin/pg_upgrade/test.sh
M src/test/regress/input/largeobject.source
M src/test/regress/output/largeobject.source
M src/test/regress/output/largeobject_1.source
M src/tools/msvc/vcregress.pl

Doc: Refer to line pointers as item identifiers.

commit   : dd01bbf00cbfd58fae6195070c2440f8f2187045    
  
author   : Peter Geoghegan <pg@bowt.ie>    
date     : Mon, 13 May 2019 15:38:59 -0700    
  
committer: Peter Geoghegan <pg@bowt.ie>    
date     : Mon, 13 May 2019 15:38:59 -0700    

Click here for diff

An upcoming HEAD-only patch will standardize the terminology around  
ItemIdData variables/line pointers, ending the practice of referring to  
them as "item pointers".  Make the "Database Page Layout" docs  
consistent with the new policy.  The term "item identifier" is already  
used in the same section, so stick with that.  
  
Discussion: https://postgr.es/m/CAH2-Wz=c=MZQjUzde3o9+2PLAPuHTpVZPPdYxN=E4ndQ2--8ew@mail.gmail.com  
Backpatch: All supported branches.  

M doc/src/sgml/storage.sgml

Fix misuse of an integer as a bool.

commit   : 0ec3d2ab250ef696f5e32d4ff7779849bfb0770d    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Mon, 13 May 2019 10:53:19 -0400    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Mon, 13 May 2019 10:53:19 -0400    

Click here for diff

pgtls_read_pending is declared to return bool, but what the underlying  
SSL_pending function returns is a count of available bytes.  
  
This is actually somewhat harmless if we're using C99 bools, but in  
the back branches it's a live bug: if the available-bytes count happened  
to be a multiple of 256, it would get converted to a zero char value.  
On machines where char is signed, counts of 128 and up could misbehave  
as well.  The net effect is that when using SSL, libpq might block  
waiting for data even though some has already been received.  
  
Broken by careless refactoring in commit 4e86f1b16, so back-patch  
to 9.5 where that came in.  
  
Per bug #15802 from David Binderman.  
  
Discussion: https://postgr.es/m/15802-f0911a97f0346526@postgresql.org  

M src/interfaces/libpq/fe-misc.c
M src/interfaces/libpq/fe-secure-openssl.c

Fix misoptimization of “{1,1}” quantifiers in regular expressions.

commit   : d6de52636827ab64f537eb78657db125b082bb65    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sun, 12 May 2019 18:53:13 -0400    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sun, 12 May 2019 18:53:13 -0400    

Click here for diff

A bounded quantifier with m = n = 1 might be thought a no-op.  But  
according to our documentation (which traces back to Henry Spencer's  
original man page) it still imposes greediness, or non-greediness in the  
case of the non-greedy variant "{1,1}?", on whatever it's attached to.  
  
This turns out not to work though, because parseqatom() optimizes away  
the m = n = 1 case without regard for whether it's supposed to change  
the greediness of the argument RE.  
  
We can fix this by just not applying the optimization when the greediness  
needs to change; the subsequent general cases handle it fine.  
  
The three cases in which we can still apply the optimization are  
(a) no quantifier, or quantifier does not impose a preference;  
(b) atom has no greediness property, implying it cannot match a  
variable amount of text anyway; or  
(c) quantifier's greediness is same as atom's.  
Note that in most cases where one of these applies, we'd have exited  
earlier in the "not a messy case" fast path.  I think it's now only  
possible to get to the optimization when the atom involves capturing  
parentheses or a non-top-level backref.  
  
Back-patch to all supported branches.  I'd ordinarily be hesitant to  
put a subtle behavioral change into back branches, but in this case  
it's very hard to see a reason why somebody would write "{1,1}?" unless  
they're trying to get the documented change-of-greediness behavior.  
  
Discussion: https://postgr.es/m/5bb27a41-350d-37bf-901e-9d26f5592dd0@charter.net  

M src/backend/regex/regcomp.c
M src/test/regress/expected/regex.out
M src/test/regress/sql/regex.sql

Fail pgwin32_message_to_UTF16() for SQL_ASCII messages.

commit   : 9dc65bcf9dece8aa92f554db51112645c9751c9e    
  
author   : Noah Misch <noah@leadboat.com>    
date     : Sun, 12 May 2019 10:33:05 -0700    
  
committer: Noah Misch <noah@leadboat.com>    
date     : Sun, 12 May 2019 10:33:05 -0700    

Click here for diff

The function had been interpreting SQL_ASCII messages as UTF8, throwing  
an error when they were invalid UTF8.  The new behavior is consistent  
with pg_do_encoding_conversion().  This affects LOG_DESTINATION_STDERR  
and LOG_DESTINATION_EVENTLOG, which will send untranslated bytes to  
write() and ReportEventA().  On buildfarm member bowerbird, enabling  
log_connections caused an error whenever the role name was not valid  
UTF8.  Back-patch to 9.4 (all supported versions).  
  
Discussion: https://postgr.es/m/20190512015615.GD1124997@rfd.leadboat.com  

M src/backend/utils/mb/mbutils.c

Rearrange pgstat_bestart() to avoid failures within its critical section.

commit   : 899f943ca43ad2752e36a44c6ed9dd19cff02f69    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sat, 11 May 2019 21:27:13 -0400    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sat, 11 May 2019 21:27:13 -0400    

Click here for diff

We long ago decided to design the shared PgBackendStatus data structure to  
minimize the cost of writing status updates, which means that writers just  
have to increment the st_changecount field twice.  That isn't hooked into  
any sort of resource management mechanism, which means that if something  
were to throw error between the two increments, the st_changecount field  
would be left odd indefinitely.  That would cause readers to lock up.  
Now, since it's also a bad idea to leave the field odd for longer than  
absolutely necessary (because readers will spin while we have it set),  
the expectation was that we'd treat these segments like spinlock critical  
sections, with only short, more or less straight-line, code in them.  
  
That was fine as originally designed, but commit 9029f4b37 broke it  
by inserting a significant amount of non-straight-line code into  
pgstat_bestart(), code that is very capable of throwing errors, not to  
mention taking a significant amount of time during which readers will spin.  
We have a report from Neeraj Kumar of readers actually locking up, which  
I suspect was due to an encoding conversion error in X509_NAME_to_cstring,  
though conceivably it was just a garden-variety OOM failure.  
  
Subsequent commits have loaded even more dubious code into pgstat_bestart's  
critical section (and commit fc70a4b0d deserves some kind of booby prize  
for managing to miss the critical section entirely, although the negative  
consequences seem minimal given that the PgBackendStatus entry should be  
seen by readers as inactive at that point).  
  
The right way to fix this mess seems to be to compute all these values  
into a local copy of the process' PgBackendStatus struct, and then just  
copy the data back within the critical section proper.  This plan can't  
be implemented completely cleanly because of the struct's heavy reliance  
on out-of-line strings, which we must initialize separately within the  
critical section.  But still, the critical section is far smaller and  
safer than it was before.  
  
In hopes of forestalling future errors of the same ilk, rename the  
macros for st_changecount management to make it more apparent that  
the writer-side macros create a critical section.  And to prevent  
the worst consequences if we nonetheless manage to mess it up anyway,  
adjust those macros so that they really are a critical section, ie  
they now bump CritSectionCount.  That doesn't add much overhead, and  
it guarantees that if we do somehow throw an error while the counter  
is odd, it will lead to PANIC and a database restart to reset shared  
memory.  
  
Back-patch to 9.5 where the problem was introduced.  
  
In HEAD, also fix an oversight in commit b0b39f72b: it failed to teach  
pgstat_read_current_status to copy st_gssstatus data from shared memory to  
local memory.  Hence, subsequent use of that data within the transaction  
would potentially see changing data that it shouldn't see.  
  
Discussion: https://postgr.es/m/CAPR3Wj5Z17=+eeyrn_ZDG3NQGYgMEOY6JV6Y-WRRhGgwc16U3Q@mail.gmail.com  

M src/backend/postmaster/pgstat.c
M src/include/pgstat.h

Fix error reporting in reindexdb

commit   : e17fba8d8f13a52b544478481c9505e2efe99dc2    
  
author   : Michael Paquier <michael@paquier.xyz>    
date     : Sat, 11 May 2019 13:01:24 +0900    
  
committer: Michael Paquier <michael@paquier.xyz>    
date     : Sat, 11 May 2019 13:01:24 +0900    

Click here for diff

When failing to reindex a table or an index, reindexdb would generate an  
extra error message related to a database failure, which is misleading.  
  
Backpatch all the way down, as this has been introduced by 85e9a5a0.  
  
Discussion: https://postgr.es/m/CAOBaU_Yo61RwNO3cW6WVYWwH7EYMPuexhKqufb2nFGOdunbcHw@mail.gmail.com  
Author: Julien Rouhaud  
Reviewed-by: Daniel Gustafsson, Álvaro Herrera, Tom Lane, Michael  
Paquier  
Backpatch-through: 9.4  

M src/bin/scripts/reindexdb.c

Cope with EINVAL and EIDRM shmat() failures in PGSharedMemoryAttach.

commit   : 91a05390c33c74c8a6bbc7f89d77fe3a8ac5098f    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Fri, 10 May 2019 14:56:41 -0400    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Fri, 10 May 2019 14:56:41 -0400    

Click here for diff

There's a very old race condition in our code to see whether a pre-existing  
shared memory segment is still in use by a conflicting postmaster: it's  
possible for the other postmaster to remove the segment in between our  
shmctl() and shmat() calls.  It's a narrow window, and there's no risk  
unless both postmasters are using the same port number, but that's possible  
during parallelized "make check" tests.  (Note that while the TAP tests  
take some pains to choose a randomized port number, pg_regress doesn't.)  
If it does happen, we treated that as an unexpected case and errored out.  
  
To fix, allow EINVAL to be treated as segment-not-present, and the same  
for EIDRM on Linux.  AFAICS, the considerations here are basically  
identical to the checks for acceptable shmctl() failures, so I documented  
and coded it that way.  
  
While at it, adjust PGSharedMemoryAttach's API to remove its undocumented  
dependency on UsedShmemSegAddr in favor of passing the attach address  
explicitly.  This makes it easier to be sure we're using a null shmaddr  
when probing for segment conflicts (thus avoiding questions about what  
EINVAL means).  I don't think there was a bug there, but it required  
fragile assumptions about the state of UsedShmemSegAddr during  
PGSharedMemoryIsInUse.  
  
Commit c09850992 may have made this failure more probable by applying  
the conflicting-segment tests more often.  Hence, back-patch to all  
supported branches, as that was.  
  
Discussion: https://postgr.es/m/22224.1557340366@sss.pgh.pa.us  

M src/backend/port/sysv_shmem.c

Fix error status of vacuumdb when multiple jobs are used

commit   : a475b1e77e13ce282603e4133c7cbcbd537d508c    
  
author   : Michael Paquier <michael@paquier.xyz>    
date     : Thu, 9 May 2019 10:30:08 +0900    
  
committer: Michael Paquier <michael@paquier.xyz>    
date     : Thu, 9 May 2019 10:30:08 +0900    

Click here for diff

When running a batch of VACUUM or ANALYZE commands on a given database,  
there were cases where it is possible to have vacuumdb not report an  
error where it actually should, leading to incorrect status results.  
  
Author: Julien Rouhaud  
Reviewed-by: Amit Kapila, Michael Paquier  
Discussion: https://postgr.es/m/CAOBaU_ZuTwz7CtqLYJ1Ouuh272bTQPLN8b1bAPk0bCBm4PDMTQ@mail.gmail.com  
Backpatch-through: 9.5  

M src/bin/scripts/vacuumdb.c

Fix documentation for the privileges required for replication functions.

commit   : 2be2fee248396bbfdeced55f74fd23148e2745af    
  
author   : Fujii Masao <fujii@postgresql.org>    
date     : Thu, 9 May 2019 01:35:13 +0900    
  
committer: Fujii Masao <fujii@postgresql.org>    
date     : Thu, 9 May 2019 01:35:13 +0900    

Click here for diff

Previously it's documented that use of replication functions is  
restricted to superusers. This is true for the functions which  
use replication origin, but not for pg_logicl_emit_message() and  
functions which use replication slot. For example, not only  
superusers but also users with REPLICATION privilege is allowed  
to use the functions for replication slot. This commit fixes  
the documentation for the privileges required for those replication  
functions.  
  
Back-patch to 9.4 (all supported versions).  
  
Author: Matsumura Ryo  
Discussion: https://postgr.es/m/03040DFF97E6E54E88D3BFEE5F5480F74ABA6E16@G01JPEXMBYT04  

M doc/src/sgml/func.sgml

Remove leftover reference to old “flat file” mechanism in a comment.

commit   : f40fb143eddb4838c7309881ff929a65c28dd8d0    
  
author   : Heikki Linnakangas <heikki.linnakangas@iki.fi>    
date     : Wed, 8 May 2019 09:32:34 +0300    
  
committer: Heikki Linnakangas <heikki.linnakangas@iki.fi>    
date     : Wed, 8 May 2019 09:32:34 +0300    

Click here for diff

The flat file mechanism was removed in PostgreSQL 9.0.  

M src/backend/access/transam/xact.c

commit   : 6de005121c487152bdfa01a7c1a920ad665a4bd1    
  
author   : Michael Paquier <michael@paquier.xyz>    
date     : Tue, 7 May 2019 14:20:19 +0900    
  
committer: Michael Paquier <michael@paquier.xyz>    
date     : Tue, 7 May 2019 14:20:19 +0900    

Click here for diff

This code was broken as of 582edc3, and is most likely not used anymore.  
Note that pg_dump supports servers down to 8.0, and psql has code to  
support servers down to 7.4.  
  
Author: Julien Rouhaud  
Reviewed-by: Tom Lane  
Discussion: https://postgr.es/m/CAOBaU_Y5y=zo3+2gf+2NJC1pvMYPcbRXoQaPXx=U7+C8Qh4CzQ@mail.gmail.com  

M src/bin/scripts/common.c