PostgreSQL 12.2 (upcoming) commit log

Fix handling of OpenSSL's SSL_clear_options

commit   : 902276ff1309ce30522d2b1bc343898c786ca02d    
  
author   : Michael Paquier <michael@paquier.xyz>    
date     : Fri, 6 Dec 2019 15:14:26 +0900    
  
committer: Michael Paquier <michael@paquier.xyz>    
date     : Fri, 6 Dec 2019 15:14:26 +0900    

Click here for diff

This function is supported down to OpenSSL 0.9.8, which is the oldest  
version supported since 593d4e4 (from Postgres 10 onwards), and is used  
since e3bdb2d (from 11 onwards).  It is defined as a macro from OpenSSL  
0.9.8 to 1.0.2, and as a function in 1.1.0 and newer versions.  However,  
the configure check present is only adapted for functions.  So, even if  
the code would be able to compile, configure fails to detect the macro,  
causing it to be ignored when compiling the code with OpenSSL from 0.9.8  
to 1.0.2.  
  
The code needs a configure check as per a364dfa, which has fixed a  
compilation issue with a past version of LibreSSL in NetBSD 5.1.  On  
HEAD, just remove the configure check as the last release of NetBSD 5 is  
from 2014 (and we have no more buildfarm members for it).  In 11 and 12,  
improve the configure logic so as both macros and functions are  
correctly detected.  This makes NetBSD 5 still work on already-released  
branches, but not for 13 onwards.  
  
The patch for HEAD is from me, and Daniel has written the version to use  
for the back-branches.  
  
Author: Michael Paquier, Daniel Gustaffson  
Reviewed-by: Tom Lane  
Discussion: https://postgr.es/m/20191205083252.GE5064@paquier.xyz  
Discussion: https://postgr.es/m/98F7F99E-1129-41D8-B86B-FE3B1E286881@yesql.se  
Backpatch-through: 11  

M configure
M configure.in

Fix whitespace.

commit   : 0e5baa095176cbb77f8ea30054ab4c3a59409da7    
  
author   : Etsuro Fujita <efujita@postgresql.org>    
date     : Wed, 4 Dec 2019 12:45:01 +0900    
  
committer: Etsuro Fujita <efujita@postgresql.org>    
date     : Wed, 4 Dec 2019 12:45:01 +0900    

Click here for diff

M src/backend/executor/nodeModifyTable.c

Ensure maxlen is at leat 1 in dict_int

commit   : a8a8c6b296406bbf30948768939c08457bd468a4    
  
author   : Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Tue, 3 Dec 2019 16:55:51 +0100    
  
committer: Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Tue, 3 Dec 2019 16:55:51 +0100    

Click here for diff

The dict_int text search dictionary template accepts maxlen parameter,  
which is then used to cap the length of input strings. The value was  
not properly checked, and the code simply does  
  
    txt[d->maxlen] = '\0';  
  
to insert a terminator, leading to segfaults with negative values.  
  
This commit simply rejects values less than 1. The issue was there since  
dct_int was introduced in 9.3, so backpatch all the way back to 9.4  
which is the oldest supported version.  
  
Reported-by: cili  
Discussion: https://postgr.es/m/16144-a36a5bef7657047d@postgresql.org  
Backpatch-through: 9.4  

M contrib/dict_int/dict_int.c
M contrib/dict_int/expected/dict_int.out
M contrib/dict_int/sql/dict_int.sql

Fix failures with TAP tests of pg_ctl on Windows

commit   : ef30975b49702f9f8aaed7b1cd8f78470ed1b22d    
  
author   : Michael Paquier <michael@paquier.xyz>    
date     : Tue, 3 Dec 2019 13:01:40 +0900    
  
committer: Michael Paquier <michael@paquier.xyz>    
date     : Tue, 3 Dec 2019 13:01:40 +0900    

Click here for diff

On Windows, all the hosts spawned by the TAP tests bind to 127.0.0.1.  
Hence, if there is a port conflict, starting a cluster would immediately  
fail.  One of the test scripts of pg_ctl initializes a node without  
PostgresNode.pm, using the default port 5432.  This could cause  
unexpected startup failures in the tests if an independent server was up  
and running on the same host (the reverse is also possible, though more  
unlikely).  Fix this issue by assigning properly a free port to the node  
configured, in the same range used as for the other nodes part of the  
tests.  
  
Author: Michael Paquier  
Reviewed-by: Andrew Dunstan  
Discussion: https://postgr.es/m/20191202031444.GC1696@paquier.xyz  
Backpatch-through: 11  

M src/bin/pg_ctl/t/001_start_stop.pl

Fix misbehavior with expression indexes on ON COMMIT DELETE ROWS tables.

commit   : b154d70f743039452f2cadee9566d754e79e3653    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sun, 1 Dec 2019 13:09:26 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sun, 1 Dec 2019 13:09:26 -0500    

Click here for diff

We implement ON COMMIT DELETE ROWS by truncating tables marked that  
way, which requires also truncating/rebuilding their indexes.  But  
RelationTruncateIndexes asks the relcache for up-to-date copies of any  
index expressions, which may cause execution of eval_const_expressions  
on them, which can result in actual execution of subexpressions.  
This is a bad thing to have happening during ON COMMIT.  Manuel Rigger  
reported that use of a SQL function resulted in crashes due to  
expectations that ActiveSnapshot would be set, which it isn't.  
The most obvious fix perhaps would be to push a snapshot during  
PreCommit_on_commit_actions, but I think that would just open the door  
to more problems: CommitTransaction explicitly expects that no  
user-defined code can be running at this point.  
  
Fortunately, since we know that no tuples exist to be indexed, there  
seems no need to use the real index expressions or predicates during  
RelationTruncateIndexes.  We can set up dummy index expressions  
instead (we do need something that will expose the right data type,  
as there are places that build index tupdescs based on this), and  
just ignore predicates and exclusion constraints.  
  
In a green field it'd likely be better to reimplement ON COMMIT DELETE  
ROWS using the same "init fork" infrastructure used for unlogged  
relations.  That seems impractical without catalog changes though,  
and even without that it'd be too big a change to back-patch.  
So for now do it like this.  
  
Per private report from Manuel Rigger.  This has been broken forever,  
so back-patch to all supported branches.  

M src/backend/catalog/heap.c
M src/backend/catalog/index.c
M src/backend/utils/cache/relcache.c
M src/include/catalog/index.h
M src/include/utils/relcache.h
M src/test/regress/expected/temp.out
M src/test/regress/sql/temp.sql

Fix off-by-one error in PGTYPEStimestamp_fmt_asc

commit   : 0dafed6fedf4da0e0e38c5961ae4b1540b4a8a5c    
  
author   : Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Sat, 30 Nov 2019 14:51:27 +0100    
  
committer: Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Sat, 30 Nov 2019 14:51:27 +0100    

Click here for diff

When using %b or %B patterns to format a date, the code was simply using  
tm_mon as an index into array of month names. But that is wrong, because  
tm_mon is 1-based, while array indexes are 0-based. The result is we  
either use name of the next month, or a segfault (for December).  
  
Fix by subtracting 1 from tm_mon for both patterns, and add a regression  
test triggering the issue. Backpatch to all supported versions (the bug  
is there far longer, since at least 2003).  
  
Reported-by: Paul Spencer  
Backpatch-through: 9.4  
Discussion: https://postgr.es/m/16143-0d861eb8688d3fef%40postgresql.org  

M src/interfaces/ecpg/pgtypeslib/timestamp.c
M src/interfaces/ecpg/test/expected/pgtypeslib-dt_test.c
M src/interfaces/ecpg/test/expected/pgtypeslib-dt_test.stderr
M src/interfaces/ecpg/test/expected/pgtypeslib-dt_test.stdout
M src/interfaces/ecpg/test/pgtypeslib/dt_test.pgc

Remove unnecessary clauses_attnums variable

commit   : 79d6e6afabcb3766ea4d21e3e26d25a5f2f69553    
  
author   : Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Thu, 28 Nov 2019 23:25:14 +0100    
  
committer: Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Thu, 28 Nov 2019 23:25:14 +0100    

Click here for diff

Commit c676e659b2 reworked how choose_best_statistics() picks the best  
extended statistics, but failed to remove clauses_attnums which is now  
unnecessary. So get rid of it and backpatch to 12, same as c676e659b2.  
  
Author: Tomas Vondra  
Discussion: https://postgr.es/m/CA+u7OA7H5rcE2=8f263w4NZD6ipO_XOrYB816nuLXbmSTH9pQQ@mail.gmail.com  
Backpatch-through: 12  

M src/backend/statistics/extended_stats.c

Fix choose_best_statistics to check clauses individually

commit   : ef3fed2ce4a3018a992ef913a3333bb682b702ae    
  
author   : Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Thu, 28 Nov 2019 22:20:28 +0100    
  
committer: Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Thu, 28 Nov 2019 22:20:28 +0100    

Click here for diff

When picking the best extended statistics object for a list of clauses,  
it's not enough to look at attnums extracted from the clause list as a  
whole. Consider for example this query with OR clauses:  
  
   SELECT * FROM t WHERE (t.a = 1) OR (t.b = 1) OR (t.c = 1)  
  
with a statistics defined on columns (a,b). Relying on attnums extracted  
from the whole OR clause, we'd consider the statistics usable. That does  
not work, as we see the conditions as a single OR-clause, referencing an  
attribute not covered by the statistic, leading to empty list of clauses  
to be estimated using the statistics and an assert failure.  
  
This changes choose_best_statistics to check which clauses are actually  
covered, and only using attributes from the fully covered ones. For the  
previous example this means the statistics object will not be considered  
as compatible with the OR-clause.  
  
Backpatch to 12, where MCVs were introduced. The issue does not affect  
older versions because functional dependencies don't handle OR clauses.  
  
Author: Tomas Vondra  
Reviewed-by: Dean Rasheed  
Reported-By: Manuel Rigger  
Discussion: https://postgr.es/m/CA+u7OA7H5rcE2=8f263w4NZD6ipO_XOrYB816nuLXbmSTH9pQQ@mail.gmail.com  
Backpatch-through: 12  

M src/backend/statistics/dependencies.c
M src/backend/statistics/extended_stats.c
M src/include/statistics/statistics.h
M src/test/regress/expected/stats_ext.out
M src/test/regress/sql/stats_ext.sql

Fix typo in comment.

commit   : bf3cef24a31966686dfaae3085ce7c545c0019a4    
  
author   : Etsuro Fujita <efujita@postgresql.org>    
date     : Wed, 27 Nov 2019 16:00:46 +0900    
  
committer: Etsuro Fujita <efujita@postgresql.org>    
date     : Wed, 27 Nov 2019 16:00:46 +0900    

Click here for diff

M src/backend/optimizer/util/relnode.c

Allow access to child table statistics if user can read parent table.

commit   : 21a4edd1281de5efd09f1cf5c6073af14a2de409    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Tue, 26 Nov 2019 14:41:48 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Tue, 26 Nov 2019 14:41:48 -0500    

Click here for diff

The fix for CVE-2017-7484 disallowed use of pg_statistic data for  
planning purposes if the user would not be able to select the associated  
column and a non-leakproof function is to be applied to the statistics  
values.  That turns out to disable use of pg_statistic data in some  
common cases involving inheritance/partitioning, where the user does  
have permission to select from the parent table that was actually named  
in the query, but not from a child table whose stats are needed.  Since,  
in non-corner cases, the user *can* select the child table's data via  
the parent, this restriction is not actually useful from a security  
standpoint.  Improve the logic so that we also check the permissions of  
the originally-named table, and allow access if select permission exists  
for that.  
  
When checking access to stats for a simple child column, we can map  
the child column number back to the parent, and perform this test  
exactly (including not allowing access if the child column isn't  
exposed by the parent).  For expression indexes, the current logic  
just insists on whole-table select access, and this patch allows  
access if the user can select the whole parent table.  In principle,  
if the child table has extra columns, this might allow access to  
stats on columns the user can't read.  In practice, it's unlikely  
that the planner is going to do any stats calculations involving  
expressions that are not visible to the query, so we'll ignore that  
fine point for now.  Perhaps someday we'll improve that logic to  
detect exactly which columns are used by an expression index ...  
but today is not that day.  
  
Back-patch to v11.  The issue was created in 9.2 and up by the  
CVE-2017-7484 fix, but this patch depends on the append_rel_array[]  
planner data structure which only exists in v11 and up.  In  
practice the issue is most urgent with partitioned tables, so  
fixing v11 and later should satisfy much of the practical need.  
  
Dilip Kumar and Amit Langote, with some kibitzing by me  
  
Discussion: https://postgr.es/m/3876.1531261875@sss.pgh.pa.us  

M src/backend/utils/adt/selfuncs.c
M src/test/regress/expected/inherit.out
M src/test/regress/sql/inherit.sql

Don't shut down Gather[Merge] early under Limit.

commit   : 1cc3a90c7551cf1f33611152eba2bca56e0b721e    
  
author   : Amit Kapila <akapila@postgresql.org>    
date     : Mon, 18 Nov 2019 14:17:41 +0530    
  
committer: Amit Kapila <akapila@postgresql.org>    
date     : Mon, 18 Nov 2019 14:17:41 +0530    

Click here for diff

Revert part of commit 19df1702f5.  
  
Early shutdown was added by that commit so that we could collect  
statistics from workers, but unfortunately, it interacted badly with  
rescans.  The problem is that we ended up destroying the parallel context  
which is required for rescans.  This leads to rescans of a Limit node over  
a Gather node to produce unpredictable results as it tries to access  
destroyed parallel context.  By reverting the early shutdown code, we  
might lose statistics in some cases of Limit over Gather [Merge], but that  
will require further study to fix.  
  
Reported-by: Jerry Sievers  
Diagnosed-by: Thomas Munro  
Author: Amit Kapila, testcase by Vignesh C  
Backpatch-through: 9.6  
Discussion: https://postgr.es/m/87ims2amh6.fsf@jsievers.enova.com  

M src/backend/executor/nodeLimit.c
M src/test/regress/expected/select_parallel.out
M src/test/regress/sql/select_parallel.sql

Avoid assertion failure with LISTEN in a serializable transaction.

commit   : a24a4167aa3889d26c425d38a8c67cc39df35fbd    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sun, 24 Nov 2019 15:57:31 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sun, 24 Nov 2019 15:57:31 -0500    

Click here for diff

If LISTEN is the only action in a serializable-mode transaction,  
and the session was not previously listening, and the notify queue  
is not empty, predicate.c reported an assertion failure.  That  
happened because we'd acquire the transaction's initial snapshot  
during PreCommit_Notify, which was called *after* predicate.c  
expects any such snapshot to have been established.  
  
To fix, just swap the order of the PreCommit_Notify and  
PreCommit_CheckForSerializationFailure calls during CommitTransaction.  
This will imply holding the notify-insertion lock slightly longer,  
but the difference could only be meaningful in serializable mode,  
which is an expensive option anyway.  
  
It appears that this is just an assertion failure, with no  
consequences in non-assert builds.  A snapshot used only to scan  
the notify queue could not have been involved in any serialization  
conflicts, so there would be nothing for  
PreCommit_CheckForSerializationFailure to do except assign it a  
prepareSeqNo and set the SXACT_FLAG_PREPARED flag.  And given no  
conflicts, neither of those omissions affect the behavior of  
ReleasePredicateLocks.  This admittedly once-over-lightly analysis  
is backed up by the lack of field reports of trouble.  
  
Per report from Mark Dilger.  The bug is old, so back-patch to all  
supported branches; but the new test case only goes back to 9.6,  
for lack of adequate isolationtester infrastructure before that.  
  
Discussion: https://postgr.es/m/3ac7f397-4d5f-be8e-f354-440020675694@gmail.com  
Discussion: https://postgr.es/m/13881.1574557302@sss.pgh.pa.us  

M src/backend/access/transam/xact.c
M src/test/isolation/expected/async-notify.out
M src/test/isolation/specs/async-notify.spec

doc: Fix whitespace in syntax.

commit   : ec9f6be3b9a935864f28b9ef183a9a4d08662cc3    
  
author   : Thomas Munro <tmunro@postgresql.org>    
date     : Mon, 25 Nov 2019 09:20:28 +1300    
  
committer: Thomas Munro <tmunro@postgresql.org>    
date     : Mon, 25 Nov 2019 09:20:28 +1300    

Click here for diff

Back-patch to 10.  
  
Author: Andreas Karlsson  
Discussion: https://postgr.es/m/043acae2-a369-b7fa-be48-1933aa2e82d1%40proxel.se  

M doc/src/sgml/ref/insert.sgml

Stabilize NOTIFY behavior by transmitting notifies before ReadyForQuery.

commit   : c47f498c93f9ebbb44cd03cf82f817da15761350    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sun, 24 Nov 2019 14:42:59 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sun, 24 Nov 2019 14:42:59 -0500    

Click here for diff

This patch ensures that, if any notify messages were received during  
a just-finished transaction, they get sent to the frontend just before  
not just after the ReadyForQuery message.  With libpq and other client  
libraries that act similarly, this guarantees that the client will see  
the notify messages as available as soon as it thinks the transaction  
is done.  
  
This probably makes no difference in practice, since in realistic  
use-cases the application would have to cope with asynchronous  
arrival of notify events anyhow.  However, it makes it a lot easier  
to build cross-session-notify test cases with stable behavior.  
I'm a bit surprised now that we've not seen any buildfarm instability  
with the test cases added by commit b10f40bf0.  Tests that I intend  
to add in an upcoming bug fix are definitely unstable without this.  
  
Back-patch to 9.6, which is as far back as we can do NOTIFY testing  
with the isolationtester infrastructure.  
  
Discussion: https://postgr.es/m/13881.1574557302@sss.pgh.pa.us  

M src/backend/commands/async.c
M src/backend/tcop/postgres.c

Improve test coverage for LISTEN/NOTIFY.

commit   : 7d4c3118137a37dddcefe28145a3a2e4bccf59fd    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sat, 23 Nov 2019 17:30:00 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sat, 23 Nov 2019 17:30:00 -0500    

Click here for diff

Back-patch commit b10f40bf0 into older branches.  This adds reporting  
of NOTIFY messages to isolationtester.c, and extends the async-notify  
test to include direct tests of basic NOTIFY functionality.  
  
This provides useful infrastructure for testing a bug fix I'm about  
to back-patch, and there seems no good reason not to have better tests  
of LISTEN/NOTIFY in the back branches.  The commit's survived long  
enough in HEAD to make it unlikely that it will cause problems.  
  
Back-patch as far as 9.6.  isolationtester.c changed too much in 9.6  
to make it sane to try to fix older branches this way, and I don't  
really want to back-patch those changes too.  
  
Discussion: https://postgr.es/m/31304.1564246011@sss.pgh.pa.us  

M src/test/isolation/expected/async-notify.out
M src/test/isolation/isolationtester.c
M src/test/isolation/specs/async-notify.spec

Add test coverage for "unchanged toast column" replication code path.

commit   : 8047a7b9dd454a6514ab7d0fcbe27aca3ec2e11c    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Fri, 22 Nov 2019 12:52:26 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Fri, 22 Nov 2019 12:52:26 -0500    

Click here for diff

It seems pretty unacceptable to have no regression test coverage  
for this aspect of the logical replication protocol, especially  
given the bugs we've found in related code.  
  
Discussion: https://postgr.es/m/16129-a0c0f48e71741e5f@postgresql.org  

M src/test/subscription/t/001_rep_changes.pl

Fix bogus tuple-slot management in logical replication UPDATE handling.

commit   : a2aa224e05dca3caf45beb6720a8b9a25e68efad    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Fri, 22 Nov 2019 11:31:19 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Fri, 22 Nov 2019 11:31:19 -0500    

Click here for diff

slot_modify_cstrings seriously abused the TupleTableSlot API by relying  
on a slot's underlying data to stay valid across ExecClearTuple.  Since  
this abuse was also quite undocumented, it's little surprise that the  
case got broken during the v12 slot rewrites.  As reported in bug #16129  
from Ondřej Jirman, this could lead to crashes or data corruption when  
a logical replication subscriber processes a row update.  Problems would  
only arise if the subscriber's table contained columns of pass-by-ref  
types that were not being copied from the publisher.  
  
Fix by explicitly copying the datum/isnull arrays from the source slot  
that the old row was in already.  This ends up being about the same  
thing that happened pre-v12, but hopefully in a less opaque and  
fragile way.  
  
We might've caught the problem sooner if there were any test cases  
dealing with updates involving non-replicated or dropped columns.  
Now there are.  
  
Back-patch to v10 where this code came in.  Even though the failure  
does not manifest before v12, IMO this code is too fragile to leave  
as-is.  In any case we certainly want the additional test coverage.  
  
Patch by me; thanks to Tomas Vondra for initial investigation.  
  
Discussion: https://postgr.es/m/16129-a0c0f48e71741e5f@postgresql.org  

M src/backend/replication/logical/worker.c
M src/test/subscription/t/001_rep_changes.pl

Defend against self-referential views in relation_is_updatable().

commit   : 5186f7625e5fc0e3ae85419099a75d6aa93d8351    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Thu, 21 Nov 2019 16:21:43 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Thu, 21 Nov 2019 16:21:43 -0500    

Click here for diff

While a self-referential view doesn't actually work, it's possible  
to create one, and it turns out that this breaks some of the  
information_schema views.  Those views call relation_is_updatable(),  
which neglected to consider the hazards of being recursive.  In  
older PG versions you get a "stack depth limit exceeded" error,  
but since v10 it'd recurse to the point of stack overrun and crash,  
because commit a4c35ea1c took out the expression_returns_set() call  
that was incidentally checking the stack depth.  
  
Since this function is only used by information_schema views, it  
seems like it'd be better to return "not updatable" than suffer  
an error.  Hence, add tracking of what views we're examining,  
in just the same way that the nearby fireRIRrules() code detects  
self-referential views.  I added a check_stack_depth() call too,  
just to be defensive.  
  
Per private report from Manuel Rigger.  Back-patch to all  
supported versions.  

M src/backend/rewrite/rewriteHandler.c
M src/backend/utils/adt/misc.c
M src/include/rewrite/rewriteHandler.h

Provide statistics for hypothetical BRIN indexes

commit   : c644407f757c26c9037e4dfc67f57b841489afd4    
  
author   : Michael Paquier <michael@paquier.xyz>    
date     : Thu, 21 Nov 2019 10:23:38 +0900    
  
committer: Michael Paquier <michael@paquier.xyz>    
date     : Thu, 21 Nov 2019 10:23:38 +0900    

Click here for diff

Trying to use hypothetical indexes with BRIN currently fails when trying  
to access a relation that does not exist when looking for the  
statistics.  With the current API, it is not possible to easily pass  
a value for pages_per_range down to the hypothetical index, so this  
makes use of the default value of BRIN_DEFAULT_PAGES_PER_RANGE, which  
should be fine enough in most cases.  
  
Being able to refine or enforce the hypothetical costs in more  
optimistic ways would require more refactoring by filling in the  
statistics when building IndexOptInfo in plancat.c.  This would involve  
ABI breakages around the costing routines, something not fit for stable  
branches.  
  
This is broken since 7e534ad, so backpatch down to v10.  
  
Author: Julien Rouhaud, Heikki Linnakangas  
Reviewed-by: Álvaro Herrera, Tom Lane, Michael Paquier  
Discussion: https://postgr.es/m/CAOBaU_ZH0LKEA8VFCocr6Lpte1ab0b6FpvgS0y4way+RPSXfYg@mail.gmail.com  
Backpatch-through: 10  

M src/backend/utils/adt/selfuncs.c

Remove incorrect markup

commit   : 2c9772f5f5b4172d7332c7fbf742d773bdf44e1c    
  
author   : Magnus Hagander <magnus@hagander.net>    
date     : Wed, 20 Nov 2019 17:03:07 +0100    
  
committer: Magnus Hagander <magnus@hagander.net>    
date     : Wed, 20 Nov 2019 17:03:07 +0100    

Click here for diff

Author: Daniel Gustafsson <daniel@yesql.se>  

M doc/src/sgml/libpq.sgml

Handle ReadFile() EOF correctly on Windows.

commit   : 2189f49c420f2bf49dfac122f9fbae51c3565594    
  
author   : Thomas Munro <tmunro@postgresql.org>    
date     : Wed, 20 Nov 2019 17:52:15 +1300    
  
committer: Thomas Munro <tmunro@postgresql.org>    
date     : Wed, 20 Nov 2019 17:52:15 +1300    

Click here for diff

When ReadFile() encounters the end of a file while reading from  
a synchronous handle with an offset provided via OVERLAPPED, it  
reports an error instead of returning 0.  By not handling that  
(undocumented) result correctly, we caused some noisy LOG  
messages about an unknown error code.  Repair.  
  
Back-patch to 12, where we started using pread()/ReadFile() with  
an offset.  
  
Reported-by: ZhenHua Cai, Amit Kapila  
Diagnosed-by: Juan Jose Santamaria Flecha  
Tested-by: Amit Kapila  
Discussion: https://postgr.es/m/CAA4eK1LK3%2BWRtpz68TiRdpHwxxWm%3D%2Bt1BMf-G68hhQsAQ41PZg%40mail.gmail.com  

M src/port/pread.c

Doc: fix minor typo in func.sgml.

commit   : 135026653d450c28d458576a023dffc0e3cb0bf9    
  
author   : Tatsuo Ishii <ishii@postgresql.org>    
date     : Wed, 20 Nov 2019 09:13:12 +0900    
  
committer: Tatsuo Ishii <ishii@postgresql.org>    
date     : Wed, 20 Nov 2019 09:13:12 +0900    

Click here for diff

Discussion: https://postgr.es/m/20191119.222048.49467220816510881.t-ishii%40sraoss.co.jp  

M doc/src/sgml/func.sgml

Fix corner-case failure in match_pattern_prefix().

commit   : bffe18e3e7dbb3c159361e62478c324ebe3041ec    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Tue, 19 Nov 2019 17:03:26 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Tue, 19 Nov 2019 17:03:26 -0500    

Click here for diff

The planner's optimization code for LIKE and regex operators could  
error out with a complaint like "no = operator for opfamily NNN"  
if someone created a binary-compatible index (for example, a  
bpchar_ops index on a text column) on the LIKE's left argument.  
  
This is a consequence of careless refactoring in commit 74dfe58a5.  
The old code in match_special_index_operator only accepted specific  
combinations of the pattern operator and the index opclass, thereby  
indirectly guaranteeing that the opclass would have a comparison  
operator with the same LHS input type as the pattern operator.  
While moving the logic out to a planner support function, I simplified  
that test in a way that no longer guarantees that.  Really though we'd  
like an altogether weaker dependency on the opclass, so rather than  
put back exactly the old code, just allow lookup failure.  I have in  
mind now to rewrite this logic completely, but this is the minimum  
change needed to fix the bug in v12.  
  
Per report from Manuel Rigger.  Back-patch to v12 where the mistake  
came in.  
  
Discussion: https://postgr.es/m/CA+u7OA7nnGYy8rY0vdTe811NuA+Frr9nbcBO9u2Z+JxqNaud+g@mail.gmail.com  

M src/backend/utils/adt/like_support.c
M src/test/regress/expected/btree_index.out
M src/test/regress/sql/btree_index.sql

Fix page modification outside of critical section in GIN

commit   : a64e7e05a418ec26a76ecbf04c80e9ba7fe8aa90    
  
author   : Alexander Korotkov <akorotkov@postgresql.org>    
date     : Wed, 20 Nov 2019 00:12:33 +0300    
  
committer: Alexander Korotkov <akorotkov@postgresql.org>    
date     : Wed, 20 Nov 2019 00:12:33 +0300    

Click here for diff

By oversight 52ac6cd2d0 makes ginDeletePage() sets pd_prune_xid of page to be  
deleted before entering the critical section.  It appears that only versions 11  
and later were affected by this oversight.  
  
Backpatch-through: 11  

M src/backend/access/gin/ginvacuum.c

Revise GIN README

commit   : ca05fa5375eb3124660e24346f1bdc3990c118f2    
  
author   : Alexander Korotkov <akorotkov@postgresql.org>    
date     : Tue, 19 Nov 2019 23:11:24 +0300    
  
committer: Alexander Korotkov <akorotkov@postgresql.org>    
date     : Tue, 19 Nov 2019 23:11:24 +0300    

Click here for diff

We find GIN concurrency bugs from time to time.  One of the problems here is  
that concurrency of GIN isn't well-documented in README.  So, it might be even  
hard to distinguish design bugs from implementation bugs.  
  
This commit revised concurrency section in GIN README providing more details.  
Some examples are illustrated in ASCII art.  
  
Also, this commit add the explanation of how is tuple layout in internal GIN  
B-tree page different in comparison with nbtree.  
  
Discussion: https://postgr.es/m/CAPpHfduXR_ywyaVN4%2BOYEGaw%3DcPLzWX6RxYLBncKw8de9vOkqw%40mail.gmail.com  
Author: Alexander Korotkov  
Reviewed-by: Peter Geoghegan  
Backpatch-through: 9.4  

M src/backend/access/gin/README

Fix traversing to the deleted GIN page via downlink

commit   : ee437ca7408b67f2b785e4221c7e567aecf68b47    
  
author   : Alexander Korotkov <akorotkov@postgresql.org>    
date     : Tue, 19 Nov 2019 23:08:14 +0300    
  
committer: Alexander Korotkov <akorotkov@postgresql.org>    
date     : Tue, 19 Nov 2019 23:08:14 +0300    

Click here for diff

Current GIN code appears to don't handle traversing to the deleted page via  
downlink.  This commit fixes that by stepping right from the delete page like  
we do in nbtree.  
  
This commit also fixes setting 'deleted' flag to the GIN pages.  Now other page  
flags are not erased once page is deleted.  That helps to keep our assertions  
true if we arrive deleted page via downlink.  
  
Discussion: https://postgr.es/m/CAPpHfdvMvsw-NcE5bRS7R1BbvA4BxoDnVVjkXC5W0Czvy9LVrg%40mail.gmail.com  
Author: Alexander Korotkov  
Reviewed-by: Peter Geoghegan  
Backpatch-through: 9.4  

M src/backend/access/gin/ginbtree.c
M src/backend/access/gin/gindatapage.c
M src/backend/access/gin/ginvacuum.c
M src/backend/access/gin/ginxlog.c

Fix deadlock between ginDeletePage() and ginStepRight()

commit   : 051c50c01c32ceb498d53e78f48297713744b7cb    
  
author   : Alexander Korotkov <akorotkov@postgresql.org>    
date     : Tue, 19 Nov 2019 23:07:36 +0300    
  
committer: Alexander Korotkov <akorotkov@postgresql.org>    
date     : Tue, 19 Nov 2019 23:07:36 +0300    

Click here for diff

When ginDeletePage() is about to delete page it locks its left sibling to revise  
the rightlink.  So, it locks pages in right to left manner.  Int he same time  
ginStepRight() locks pages in left to right manner, and that could cause a  
deadlock.  
  
This commit makes ginScanToDelete() keep exclusive lock on left siblings of  
currently investigated path.  That elimites need to relock left sibling in  
ginDeletePage().  Thus, deadlock with ginStepRight() can't happen anymore.  
  
Reported-by: Chen Huajun  
Discussion: https://postgr.es/m/5c332bd1.87b6.16d7c17aa98.Coremail.chjischj%40163.com  
Author: Alexander Korotkov  
Reviewed-by: Peter Geoghegan  
Backpatch-through: 10  

M src/backend/access/gin/README
M src/backend/access/gin/ginvacuum.c

Doc: clarify use of RECURSIVE in WITH.

commit   : 823a551fe0c6e732f48cb54dc60d5cfd66b0aff4    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Tue, 19 Nov 2019 14:43:37 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Tue, 19 Nov 2019 14:43:37 -0500    

Click here for diff

Apparently some people misinterpreted the syntax as being that  
RECURSIVE is a prefix of individual WITH queries.  It's a modifier  
for the WITH clause as a whole, so state that more clearly.  
  
Discussion: https://postgr.es/m/ca53c6ce-a0c6-b14a-a8e3-162f0b2cc119@a-kretschmer.de  

M doc/src/sgml/ref/select.sgml

Doc: clarify behavior of ALTER DEFAULT PRIVILEGES ... IN SCHEMA.

commit   : 93b2bbede91b32d12d9e444e576be2efff6942b5    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Tue, 19 Nov 2019 14:21:41 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Tue, 19 Nov 2019 14:21:41 -0500    

Click here for diff

The existing text stated that "Default privileges that are specified  
per-schema are added to whatever the global default privileges are for  
the particular object type".  However, that bare-bones observation is  
not quite clear enough, as demonstrated by the complaint in bug #16124.  
Flesh it out by stating explicitly that you can't revoke built-in  
default privileges this way, and by providing an example to drive  
the point home.  
  
Back-patch to all supported branches, since it's been like this  
from the beginning.  
  
Discussion: https://postgr.es/m/16124-423d8ee4358421bc@postgresql.org  

M doc/src/sgml/ref/alter_default_privileges.sgml

Further fix dumping of views that contain just VALUES(...).

commit   : fcaf29d87a6ca08188fa9fe1ebba90c9654df006    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sat, 16 Nov 2019 20:00:19 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Sat, 16 Nov 2019 20:00:19 -0500    

Click here for diff

It turns out that commit e9f1c01b7 missed a case: we must print a  
VALUES clause in long format if get_query_def is given a resultDesc  
that would require the query's output column name(s) to be different  
from what the bare VALUES clause would produce.  
  
This applies in case an ALTER ... RENAME COLUMN has been done to  
a view that formerly could be printed in simple format, as shown  
in the added regression test case.  It also explains bug #16119  
from Dmitry Telpt, because it turns out that (unlike CREATE VIEW)  
CREATE MATERIALIZED VIEW fails to apply any column aliases it's  
given to the stored ON SELECT rule.  So to get them to be printed,  
we have to account for the resultDesc renaming.  It might be worth  
changing the matview code so that it creates the ON SELECT rule  
with the correct aliases; but we'd still need these messy checks in  
get_simple_values_rte to handle the case of a subsequent column  
rename, so any such change would be just neatnik-ism not a bug fix.  
  
Like the previous patch, back-patch to all supported branches.  
  
Discussion: https://postgr.es/m/16119-e64823f30a45a754@postgresql.org  

M src/backend/utils/adt/ruleutils.c
M src/test/regress/expected/rules.out
M src/test/regress/sql/rules.sql

Improve stability of tests for VACUUM (SKIP_LOCKED)

commit   : bbaa38e824236ac106fffbf5f61e5da9b3dc1ec9    
  
author   : Michael Paquier <michael@paquier.xyz>    
date     : Sat, 16 Nov 2019 15:23:50 +0900    
  
committer: Michael Paquier <michael@paquier.xyz>    
date     : Sat, 16 Nov 2019 15:23:50 +0900    

Click here for diff

Concurrent autovacuums running with the main regression test suite  
could cause the tests with VACUUM (SKIP_LOCKED) to generate randomly  
WARNING messages.  For these tests, set client_min_messages to ERROR to  
get rid of those random failures, as disabling autovacuum for the  
relations operated would not completely close the failure window.  
  
For isolation tests, disable autovacuum for the relations vacuumed with  
SKIP_LOCKED.  The tests are designed so as LOCK commands are taken  
in a first session before running a concurrent VACUUM (SKIP_LOCKED) in a  
second to generate WARNING messages, but a concurrent autovacuum could  
cause the tests to be slower.  
  
Reported-by: Tom Lane  
Author: Michael Paquier  
Reviewed-by: Andres Freund, Tom Lane  
Discussion: https://postgr.es/m/25294.1573077278@sss.pgh.pa.us  
Backpatch-through: 12  

M src/test/isolation/specs/vacuum-skip-locked.spec
M src/test/regress/expected/vacuum.out
M src/test/regress/sql/vacuum.sql

Skip system attributes when applying mvdistinct stats

commit   : 28555a53cb75d00ed0e73f63a6481b4fffcc3dea    
  
author   : Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Sat, 16 Nov 2019 01:17:15 +0100    
  
committer: Tomas Vondra <tomas.vondra@postgresql.org>    
date     : Sat, 16 Nov 2019 01:17:15 +0100    

Click here for diff

When estimating number of distinct groups, we failed to ignore system  
attributes when matching the group expressions to mvdistinct stats,  
causing failures like  
  
  ERROR: negative bitmapset member not allowed  
  
Fix that by simply skipping anything that is not a regular attribute.  
Backpatch to PostgreSQL 10, where the extended stats were introduced.  
  
Bug: #16111  
Reported-by: Tuomas Leikola  
Author: Tomas Vondra  
Backpatch-through: 10  
Discussion: https://postgr.es/m/16111-687799584c3a7e73@postgresql.org  

M src/backend/utils/adt/selfuncs.c
M src/test/regress/expected/stats_ext.out
M src/test/regress/sql/stats_ext.sql

Always call ExecShutdownNode() if appropriate.

commit   : 24897e1a1af27dc759fb41afba2a663ff9af4ef6    
  
author   : Thomas Munro <tmunro@postgresql.org>    
date     : Sat, 16 Nov 2019 10:04:52 +1300    
  
committer: Thomas Munro <tmunro@postgresql.org>    
date     : Sat, 16 Nov 2019 10:04:52 +1300    

Click here for diff

Call ExecShutdownNode() after ExecutePlan()'s loop, rather than at each  
break.  We had forgotten to do that in one case.  The omission caused  
intermittent "temporary file leak" warnings from multi-batch parallel  
hash joins with a LIMIT clause.  
  
Back-patch to 11.  Though the problem exists in theory in earlier  
parallel query releases, nothing really depended on it.  
  
Author: Kyotaro Horiguchi  
Reviewed-by: Thomas Munro, Amit Kapila  
Discussion: https://postgr.es/m/20191111.212418.2222262873417235945.horikyota.ntt%40gmail.com  

M src/backend/executor/execMain.c

Doc: in v12 release notes, explain how to replace uses of consrc and adsrc.

commit   : d61e7f174db9ee45df58afc103682d933048aa9a    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Fri, 15 Nov 2019 12:31:53 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Fri, 15 Nov 2019 12:31:53 -0500    

Click here for diff

While you can find that info if you drill down far enough, it seems more  
helpful to put something right in the compatibility notes.  Per a question  
from Ivan Sergio Borgonovo.  
  
Discussion: https://postgr.es/m/a6359855-2a5e-a56c-ebba-4ea46a1f0ebe@webthatworks.it  

M doc/src/sgml/release-12.sgml

Add missing check_collation_set call to bpcharne().

commit   : 5a6eea0926f4b732a803149a6c169b1f79993fa9    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 15:53:53 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 15:53:53 -0500    

Click here for diff

We should throw an error for indeterminate collation, but bpcharne()  
was missing that logic, resulting in a much less user-friendly error  
(either an assertion failure or "cache lookup failed for collation 0").  
  
Per report from Manuel Rigger.  Back-patch to v12 where the mistake  
came in, evidently in commit 5e1963fb7.  (Before non-deterministic  
collations, this function wasn't collation sensitive.)  
  
Discussion: https://postgr.es/m/CA+u7OA4HOjtymxAbuGNh4-X_2R0Lw5n01tzvP8E5-i-2gQXYWA@mail.gmail.com  

M src/backend/utils/adt/varchar.c

Fix silly initializations (cosmetic only).

commit   : 99cc47b1d85aabc4a80910c839073f83a4fb163b    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 15:26:54 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 15:26:54 -0500    

Click here for diff

Initializing a pointer to "false" isn't per project style,  
and reportedly some compilers warn about it (though I've  
not seen any such warnings in the buildfarm).  
  
Seems to have come in with commit ff11e7f4b, so back-patch  
to v12 where that was added.  
  
Didier Gautheron  
  
Discussion: https://postgr.es/m/CAJRYxu+XQuM0qnSqt1Ujztu6fBPzMMAT3VEn6W32rgKG6A2Fsw@mail.gmail.com  

M src/backend/commands/trigger.c

Avoid downcasing/truncation of RADIUS authentication parameters.

commit   : d9802590a1b32b5d3d071766f7814a31cc00fc87    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 13:41:04 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 13:41:04 -0500    

Click here for diff

Commit 6b76f1bb5 changed all the RADIUS auth parameters to be lists  
rather than single values.  But its use of SplitIdentifierString  
to parse the list format was not very carefully thought through,  
because that function thinks it's parsing SQL identifiers, which  
means it will (a) downcase the strings and (b) truncate them to  
be shorter than NAMEDATALEN.  While downcasing should be harmless  
for the server names and ports, it's just wrong for the shared  
secrets, and probably for the NAS Identifier strings as well.  
The truncation aspect is at least potentially a problem too,  
though typical values for these parameters would fit in 63 bytes.  
  
Fortunately, we now have a function SplitGUCList that is exactly  
the same except for not doing the two unwanted things, so fixing  
this is a trivial matter of calling that function instead.  
  
While here, improve the documentation to show how to double-quote  
the parameter values.  I failed to resist the temptation to do  
some copy-editing as well.  
  
Report and patch from Marcos David (bug #16106); doc changes by me.  
Back-patch to v10 where the aforesaid commit came in, since this is  
arguably a regression from our previous behavior with RADIUS auth.  
  
Discussion: https://postgr.es/m/16106-7d319e4295d08e70@postgresql.org  

M doc/src/sgml/client-auth.sgml
M src/backend/libpq/hba.c

Include TableFunc references when computing expression dependencies.

commit   : eec569fac7a5cae17670f777c5a31054bf89ecfb    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 12:11:49 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 12:11:49 -0500    

Click here for diff

The TableFunc node (i.e., XMLTABLE) includes type and collation OIDs  
that might not be referenced anywhere else in the expression tree,  
so they need to be accounted for when extracting dependencies.  
  
Fortunately, the practical effects of this are limited, since  
(a) it's somewhat unlikely that people would be extracting  
columns of non-builtin types from an XML document, and (b)  
in many scenarios, the query would contain other references  
to such types, or functions depending on them.  However, it's  
not hard to construct examples wherein the existing code lets  
one drop a type used in XMLTABLE and thereby break a view.  
  
This is evidently an original oversight in the XMLTABLE patch,  
so back-patch to v10 where that came in.  
  
Discussion: https://postgr.es/m/18427.1573508501@sss.pgh.pa.us  

M src/backend/catalog/dependency.c

Handle arrays and ranges in pg_upgrade's test for non-upgradable types.

commit   : 1cd57b05ef8bc1ff5862a004ef621884613a4f0d    
  
author   : Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 11:35:37 -0500    
  
committer: Tom Lane <tgl@sss.pgh.pa.us>    
date     : Wed, 13 Nov 2019 11:35:37 -0500    

Click here for diff

pg_upgrade needs to check whether certain non-upgradable data types  
appear anywhere on-disk in the source cluster.  It knew that it has  
to check for these types being contained inside domains and composite  
types; but it somehow overlooked that they could be contained in  
arrays and ranges, too.  Extend the existing recursive-containment  
query to handle those cases.  
  
We probably should have noticed this oversight while working on  
commit 0ccfc2822 and follow-ups, but we failed to :-(.  The whole  
thing's possibly a bit overdesigned, since we don't really expect  
that any of these types will appear on disk; but if we're going to  
the effort of doing a recursive search then it's silly not to cover  
all the possibilities.  
  
While at it, refactor so that we have only one copy of the search  
logic, not three-and-counting.  Also, to keep the branches looking  
more alike, back-patch the output wording change of commit 1634d3615.  
  
Back-patch to all supported branches.  
  
Discussion: https://postgr.es/m/31473.1573412838@sss.pgh.pa.us  

M src/bin/pg_upgrade/version.c

docs: clarify that only INSERT and UPDATE triggers can mod. NEW

commit   : 5f2cfe7f5fbd90f0d6ffef26c90808f4bd3e1d51    
  
author   : Bruce Momjian <bruce@momjian.us>    
date     : Tue, 12 Nov 2019 22:04:31 -0500    
  
committer: Bruce Momjian <bruce@momjian.us>    
date     : Tue, 12 Nov 2019 22:04:31 -0500    

Click here for diff

The point is that DELETE triggers cannot modify any values.  
  
Reported-by: Eugen Konkov, Liudmila Mantrova  
  
Discussion: https://postgr.es/m/919823407.20191029175436@yandex.ru  
  
Backpatch-through: 12 only, where commit as missing  

M doc/src/sgml/trigger.sgml