Stamp 13.5.
commit : 084346ccee8ead6b387a90cdf6a29036ae9ec77e
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 8 Nov 2021 17:00:24 -0500
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 8 Nov 2021 17:00:24 -0500
M configure
M configure.in
Last-minute updates for release notes.
commit : 402c3ba3954d81531b6c9ff9b5d554552a85c6a7
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 8 Nov 2021 14:02:16 -0500
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 8 Nov 2021 14:02:16 -0500
Security: CVE-2021-23214, CVE-2021-23222
M doc/src/sgml/release-13.sgml
libpq: reject extraneous data after SSL or GSS encryption handshake.
commit : 844b3169204c28cd086c1b4fae4a2cbdd0540640
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 8 Nov 2021 11:14:56 -0500
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 8 Nov 2021 11:14:56 -0500
libpq collects up to a bufferload of data whenever it reads data from
the socket. When SSL or GSS encryption is requested during startup,
any additional data received with the server's yes-or-no reply
remained in the buffer, and would be treated as already-decrypted data
once the encryption handshake completed. Thus, a man-in-the-middle
with the ability to inject data into the TCP connection could stuff
some cleartext data into the start of a supposedly encryption-protected
database session.
This could probably be abused to inject faked responses to the
client's first few queries, although other details of libpq's behavior
make that harder than it sounds. A different line of attack is to
exfiltrate the client's password, or other sensitive data that might
be sent early in the session. That has been shown to be possible with
a server vulnerable to CVE-2021-23214.
To fix, throw a protocol-violation error if the internal buffer
is not empty after the encryption handshake.
Our thanks to Jacob Champion for reporting this problem.
Security: CVE-2021-23222
M doc/src/sgml/protocol.sgml
M src/interfaces/libpq/fe-connect.c
Reject extraneous data after SSL or GSS encryption handshake.
commit : e92ed93e8eb76ee0701b42d4f0ce94e6af3fc741
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 8 Nov 2021 11:01:43 -0500
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 8 Nov 2021 11:01:43 -0500
The server collects up to a bufferload of data whenever it reads data
from the client socket. When SSL or GSS encryption is requested
during startup, any additional data received with the initial
request message remained in the buffer, and would be treated as
already-decrypted data once the encryption handshake completed.
Thus, a man-in-the-middle with the ability to inject data into the
TCP connection could stuff some cleartext data into the start of
a supposedly encryption-protected database session.
This could be abused to send faked SQL commands to the server,
although that would only work if the server did not demand any
authentication data. (However, a server relying on SSL certificate
authentication might well not do so.)
To fix, throw a protocol-violation error if the internal buffer
is not empty after the encryption handshake.
Our thanks to Jacob Champion for reporting this problem.
Security: CVE-2021-23214
M src/backend/libpq/pqcomm.c
M src/backend/postmaster/postmaster.c
M src/include/libpq/libpq.h
Fix typo
commit : 7c0a78f089980cec45927aa3160add1781aed402
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 8 Nov 2021 09:17:24 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 8 Nov 2021 09:17:24 -0300
Introduced in 1d97d3d0867f.
Co-authored-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/83641f59-d566-b33e-ef21-a272a98675aa@gmail.com
M src/backend/access/transam/xlog.c
M src/backend/po/de.po
M src/backend/po/fr.po
M src/backend/po/ru.po
M src/backend/po/sv.po
M src/test/recovery/t/026_overwrite_contrecord.pl
Translation updates
commit : 98da5cd0d1b1c47a19f8453413c0fd01c07b3830
author : Peter Eisentraut <peter@eisentraut.org>
date : Mon, 8 Nov 2021 10:08:56 +0100
committer: Peter Eisentraut <peter@eisentraut.org>
date : Mon, 8 Nov 2021 10:08:56 +0100
Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git
Source-Git-Hash: 027ff7dad8afb1a907cb4c59da4e13c3ace8d376
M src/backend/po/de.po
M src/backend/po/fr.po
M src/backend/po/ru.po
M src/backend/po/sv.po
M src/bin/initdb/po/fr.po
M src/bin/initdb/po/ru.po
M src/bin/pg_archivecleanup/po/fr.po
M src/bin/pg_basebackup/po/fr.po
M src/bin/pg_basebackup/po/ru.po
M src/bin/pg_checksums/po/fr.po
M src/bin/pg_checksums/po/ru.po
M src/bin/pg_checksums/po/sv.po
M src/bin/pg_config/po/fr.po
M src/bin/pg_config/po/ru.po
M src/bin/pg_controldata/po/fr.po
M src/bin/pg_controldata/po/ru.po
M src/bin/pg_ctl/po/fr.po
M src/bin/pg_ctl/po/ru.po
M src/bin/pg_dump/po/de.po
M src/bin/pg_dump/po/fr.po
M src/bin/pg_dump/po/ru.po
M src/bin/pg_dump/po/sv.po
M src/bin/pg_resetwal/po/fr.po
M src/bin/pg_resetwal/po/ru.po
M src/bin/pg_resetwal/po/sv.po
M src/bin/pg_rewind/po/fr.po
M src/bin/pg_rewind/po/ru.po
M src/bin/pg_rewind/po/sv.po
M src/bin/pg_test_fsync/po/ru.po
M src/bin/pg_test_timing/po/ru.po
M src/bin/pg_upgrade/po/fr.po
M src/bin/pg_upgrade/po/ru.po
M src/bin/pg_upgrade/po/sv.po
M src/bin/pg_verifybackup/po/fr.po
M src/bin/pg_verifybackup/po/ru.po
M src/bin/pg_waldump/po/fr.po
M src/bin/pg_waldump/po/ru.po
M src/bin/psql/po/de.po
M src/bin/psql/po/fr.po
M src/bin/psql/po/ru.po
M src/bin/psql/po/sv.po
M src/bin/scripts/po/de.po
M src/bin/scripts/po/fr.po
M src/bin/scripts/po/ru.po
M src/bin/scripts/po/sv.po
M src/interfaces/ecpg/ecpglib/po/ru.po
M src/interfaces/ecpg/preproc/po/fr.po
M src/interfaces/ecpg/preproc/po/ru.po
M src/interfaces/ecpg/preproc/po/sv.po
M src/interfaces/libpq/po/fr.po
M src/interfaces/libpq/po/ru.po
M src/interfaces/libpq/po/sv.po
M src/pl/plperl/po/fr.po
M src/pl/plpgsql/src/po/fr.po
M src/pl/plpgsql/src/po/ru.po
M src/pl/plpython/po/fr.po
M src/pl/tcl/po/fr.po
Release notes for 14.1, 13.5, 12.9, 11.14, 10.19, 9.6.24.
commit : 5d73415d20086406f083eb0929626036e10797a1
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sun, 7 Nov 2021 14:21:50 -0500
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sun, 7 Nov 2021 14:21:50 -0500
M doc/src/sgml/release-13.sgml
Reset lastOverflowedXid on standby when needed
commit : e1fee28a0441652f021045646808a772314798f9
author : Alexander Korotkov <akorotkov@postgresql.org>
date : Sat, 6 Nov 2021 18:31:21 +0300
committer: Alexander Korotkov <akorotkov@postgresql.org>
date : Sat, 6 Nov 2021 18:31:21 +0300
Currently, lastOverflowedXid is never reset. It's just adjusted on new
transactions known to be overflowed. But if there are no overflowed
transactions for a long time, snapshots could be mistakenly marked as
suboverflowed due to wraparound.
This commit fixes this issue by resetting lastOverflowedXid when needed
altogether with KnownAssignedXids.
Backpatch to all supported versions.
Reported-by: Stan Hu
Discussion: https://postgr.es/m/CAMBWrQ%3DFp5UAsU_nATY7EMY7NHczG4-DTDU%3DmCvBQZAQ6wa2xQ%40mail.gmail.com
Author: Kyotaro Horiguchi, Alexander Korotkov
Reviewed-by: Stan Hu, Simon Riggs, Nikolay Samokhvalov, Andrey Borodin, Dmitry Dolgov
M src/backend/storage/ipc/procarray.c
Avoid crash in rare case of concurrent DROP
commit : bf5cdcfd5e444756378e60307a89b8cad16f65c0
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Fri, 5 Nov 2021 12:29:34 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Fri, 5 Nov 2021 12:29:34 -0300
When a role being dropped contains is referenced by catalog objects that
are concurrently also being dropped, a crash can result while trying to
construct the string that describes the objects. Suppress that by
ignoring objects whose descriptions are returned as NULL.
The majority of relevant codesites were already cautious about this
already; we had just missed a couple.
This is an old bug, so backpatch all the way back.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17126-21887f04508cb5c8@postgresql.org
M src/backend/catalog/dependency.c
M src/backend/catalog/pg_shdepend.c
Update alternative expected output file.
commit : b7299b66469fe90c5cbb19c1d02d8e91fc95e4f2
author : Heikki Linnakangas <heikki.linnakangas@iki.fi>
date : Wed, 3 Nov 2021 19:38:17 +0200
committer: Heikki Linnakangas <heikki.linnakangas@iki.fi>
date : Wed, 3 Nov 2021 19:38:17 +0200
Previous commit added a test to 'largeobject', but neglected the
alternative expected output file 'largeobject_1.source'. Per failure
on buildfarm animal 'hamerkop'.
Discussion: https://www.postgresql.org/message-id/DBA08346-9962-4706-92D1-230EE5201C10@yesql.se
M src/test/regress/output/largeobject_1.source
Fix snapshot reference leak if lo_export fails.
commit : 07070c0082aacc9e9bf09d1d08bc3a60c999332e
author : Heikki Linnakangas <heikki.linnakangas@iki.fi>
date : Wed, 3 Nov 2021 10:28:52 +0200
committer: Heikki Linnakangas <heikki.linnakangas@iki.fi>
date : Wed, 3 Nov 2021 10:28:52 +0200
If lo_export() fails to open the target file or to write to it, it leaks
the created LargeObjectDesc and its snapshot in the top-transaction
context and resource owner. That's pretty harmless, it's a small leak
after all, but it gives the user a "Snapshot reference leak" warning.
Fix by using a short-lived memory context and no resource owner for
transient LargeObjectDescs that are opened and closed within one function
call. The leak is easiest to reproduce with lo_export() on a directory
that doesn't exist, but in principle the other lo_* functions could also
fail.
Backpatch to all supported versions.
Reported-by: Andrew B
Reviewed-by: Alvaro Herrera
Discussion: https://www.postgresql.org/message-id/32bf767a-2d65-71c4-f170-122f416bab7e@iki.fi
M src/backend/libpq/be-fsstubs.c
M src/backend/storage/large_object/inv_api.c
M src/test/regress/input/largeobject.source
M src/test/regress/output/largeobject.source
Fix variable lifespan in ExecInitCoerceToDomain().
commit : ada667b45496998674818e135e9cd6580f1bc019
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 2 Nov 2021 13:36:47 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 2 Nov 2021 13:36:47 -0400
This undoes a mistake in 1ec7679f1: domainval and domainnull were
meant to live across loop iterations, but they were incorrectly
moved inside the loop. The effect was only to emit useless extra
EEOP_MAKE_READONLY steps, so it's not a big deal; nonetheless,
back-patch to v13 where the mistake was introduced.
Ranier Vilela
Discussion: https://postgr.es/m/CAEudQAqXuhbkaAp-sGH6dR6Nsq7v28_0TPexHOm6FiDYqwQD-w@mail.gmail.com
M src/backend/executor/execExpr.c
Avoid O(N^2) behavior in SyncPostCheckpoint().
commit : 0151af40cd4e0321bc549cea9f0c631bce0303c5
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 2 Nov 2021 11:31:54 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 2 Nov 2021 11:31:54 -0400
As in commits 6301c3ada and e9d9ba2a4, avoid doing repetitive
list_delete_first() operations, since that would be expensive when
there are many files waiting to be unlinked. This is a slightly
larger change than in those cases. We have to keep the list state
valid for calls to AbsorbSyncRequests(), so it's necessary to invent a
"canceled" field instead of immediately deleting PendingUnlinkEntry
entries. Also, because we might not be able to process all the
entries, we need a new list primitive list_delete_first_n().
list_delete_first_n() is almost list_copy_tail(), but it modifies the
input List instead of making a new copy. I found a couple of existing
uses of the latter that could profitably use the new function. (There
might be more, but the other callers look like they probably shouldn't
overwrite the input List.)
As before, back-patch to v13.
Discussion: https://postgr.es/m/CD2F0E7F-9822-45EC-A411-AE56F14DEA9F@amazon.com
M src/backend/nodes/list.c
M src/backend/optimizer/util/clauses.c
M src/backend/parser/parse_func.c
M src/backend/storage/sync/sync.c
M src/include/nodes/pg_list.h
Avoid some other O(N^2) hazards in list manipulation.
commit : e477642a1ba8041c94cae2f8bbdb689f05cb0260
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 1 Nov 2021 16:24:40 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 1 Nov 2021 16:24:40 -0400
In the same spirit as 6301c3ada, fix some more places where we were
using list_delete_first() in a loop and thereby risking O(N^2)
behavior. It's not clear that the lists manipulated in these spots
can get long enough to be really problematic ... but it's not clear
that they can't, either, and the fixes are simple enough.
As before, back-patch to v13.
Discussion: https://postgr.es/m/CD2F0E7F-9822-45EC-A411-AE56F14DEA9F@amazon.com
M contrib/pg_trgm/trgm_regexp.c
M src/backend/executor/nodeAgg.c
M src/backend/jit/llvm/llvmjit.c
Handle XLOG_OVERWRITE_CONTRECORD in DecodeXLogOp
commit : 17227825ca4288c70af45038ac6af26be7fde570
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 1 Nov 2021 13:07:23 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 1 Nov 2021 13:07:23 -0300
Failing to do so results in inability of logical decoding to process the
WAL stream. Handle it by doing nothing.
Backpatch all the way back.
Reported-by: Petr Jelínek <petr.jelinek@enterprisedb.com>
M src/backend/replication/logical/decode.c
Preserve opclass parameters across REINDEX CONCURRENTLY
commit : 77f7909a409eac9ee9a5728a4f80741925580fb6
author : Michael Paquier <michael@paquier.xyz>
date : Mon, 1 Nov 2021 11:40:29 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Mon, 1 Nov 2021 11:40:29 +0900
The opclass parameter Datums from the old index are fetched in the same
way as for predicates and expressions, by grabbing them directly from
the system catalogs. They are then copied into the new IndexInfo that
will be used for the creation of the new copy.
This caused the new index to be rebuilt with default parameters rather
than the ones pre-defined by a user. The only way to get back a new
index with correct opclass parameters would be to recreate a new index
from scratch.
The issue has been introduced by 911e702.
Author: Michael Paquier
Reviewed-by: Zhihong Yu
Discussion: https://postgr.es/m/YX0CG/QpLXcPr8HJ@paquier.xyz
Backpatch-through: 13
M src/backend/catalog/index.c
M src/test/regress/expected/create_index.out
M src/test/regress/sql/create_index.sql
Don't try to read a multi-GB pg_stat_statements file in one call.
commit : 3a5b313ce74864303a8a5d6556b49d642c422922
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sun, 31 Oct 2021 19:13:48 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sun, 31 Oct 2021 19:13:48 -0400
Windows fails on a request to read() more than INT_MAX bytes,
and perhaps other platforms could have similar issues. Let's
adjust this code to read at most 1GB per call.
(One would not have thought the file could get that big, but now
we have a field report of trouble, so it can. We likely ought to
add some mechanism to limit the size of the query-texts file
separately from the size of the hash table. That is not this
patch, though.)
Per bug #17254 from Yusuke Egashira. It's been like this for
awhile, so back-patch to all supported branches.
Discussion: https://postgr.es/m/17254-a926c89dc03375c2@postgresql.org
M contrib/pg_stat_statements/pg_stat_statements.c
Avoid O(N^2) behavior when the standby process releases many locks.
commit : df238aed1090249a2bcd21e1688a398c6f513dfd
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sun, 31 Oct 2021 15:31:29 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sun, 31 Oct 2021 15:31:29 -0400
When replaying a transaction that held many exclusive locks on the
primary, a standby server's startup process would expend O(N^2)
effort on manipulating the list of locks. This code was fine when
written, but commit 1cff1b95a made repetitive list_delete_first()
calls inefficient, as explained in its commit message. Fix by just
iterating the list normally, and releasing storage only when done.
(This'd be inadequate if we needed to recover from an error occurring
partway through; but we don't.)
Back-patch to v13 where 1cff1b95a came in.
Nathan Bossart
Discussion: https://postgr.es/m/CD2F0E7F-9822-45EC-A411-AE56F14DEA9F@amazon.com
M src/backend/storage/ipc/standby.c
Update time zone data files to tzdata release 2021e.
commit : 4cd72add0579d33e741cb1a2effe6ee727a994dd
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 29 Oct 2021 11:38:18 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 29 Oct 2021 11:38:18 -0400
DST law changes in Fiji, Jordan, Palestine, and Samoa. Historical
corrections for Barbados, Cook Islands, Guyana, Niue, Portugal, and
Tonga.
Also, the Pacific/Enderbury zone has been renamed to Pacific/Kanton.
The following zones have been merged into nearby, more-populous zones
whose clocks have agreed since 1970: Africa/Accra, America/Atikokan,
America/Blanc-Sablon, America/Creston, America/Curacao,
America/Nassau, America/Port_of_Spain, Antarctica/DumontDUrville,
and Antarctica/Syowa.
M src/timezone/data/tzdata.zi
M src/timezone/known_abbrevs.txt
Improve contrib/amcheck's tests for CREATE INDEX CONCURRENTLY.
commit : 5a4b8a8a720ca699c5cbcfc04069dfe089f218c8
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 28 Oct 2021 11:45:14 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 28 Oct 2021 11:45:14 -0400
Commits fdd965d07 and 3cd9c3b92 tested CREATE INDEX CONCURRENTLY by
launching two separate pgbench runs concurrently. This was needed so
that only a single client thread would run CREATE INDEX CONCURRENTLY,
avoiding deadlock between two CICs. However, there's a better way,
which is to use an advisory lock to prevent concurrent CICs. That's
better in part because the test code is shorter and more readable, but
mostly because it automatically scales things to launch an appropriate
number of CICs relative to the number of INSERT transactions.
As committed, typically half to three-quarters of the CIC transactions
were pointless because the INSERT transactions had already stopped.
In passing, remove background_pgbench, which was added to support
these tests and isn't needed anymore. We can always put it back
if we find a use for it later.
Back-patch to v12; older pgbench versions lack the
conditional-execution features needed for this method.
Tom Lane and Andrey Borodin
Discussion: https://postgr.es/m/139687.1635277318@sss.pgh.pa.us
M contrib/amcheck/t/002_cic.pl
M contrib/amcheck/t/003_cic_2pc.pl
M src/test/perl/PostgresNode.pm
doc: Fix link to SELinux user guide in sepgsql page
commit : 0a75e1186afe58e140cc5eb0f6f0c13f5b7d8c3b
author : Michael Paquier <michael@paquier.xyz>
date : Thu, 28 Oct 2021 09:26:18 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Thu, 28 Oct 2021 09:26:18 +0900
Reported-by: Anton Voloshin
Discussion: https://postgr.es/m/15a86d4e-a237-1acd-18a2-fd69730f1ab9@postgrespro.ru
Backpatch-through: 10
M doc/src/sgml/sepgsql.sgml
Fix ordering of items in nbtree error message.
commit : d5a2ffbce534e4fe9d207da14c9b7cf7a00a7953
author : Peter Geoghegan <pg@bowt.ie>
date : Wed, 27 Oct 2021 13:09:00 -0700
committer: Peter Geoghegan <pg@bowt.ie>
date : Wed, 27 Oct 2021 13:09:00 -0700
Oversight in commit a5213adf.
Backpatch: 13-, just like commit a5213adf.
M src/backend/access/nbtree/nbtinsert.c
Further harden nbtree posting split code.
commit : f8cce4a3d88ccce830086bc80b563bd152f4955f
author : Peter Geoghegan <pg@bowt.ie>
date : Wed, 27 Oct 2021 12:10:43 -0700
committer: Peter Geoghegan <pg@bowt.ie>
date : Wed, 27 Oct 2021 12:10:43 -0700
Add more defensive checks around posting list split code. These should
detect corruption involving duplicate table TIDs earlier and more
reliably than any existing check.
Follow up to commit 8f72bbac.
Discussion: https://postgr.es/m/CAH2-WzkrSY_kjyd1_M5xJK1uM0govJXMxPn8JUSvwcUOiHuWVw@mail.gmail.com
Backpatch: 13-, where nbtree deduplication was introduced.
M src/backend/access/nbtree/nbtinsert.c
M src/backend/access/nbtree/nbtsearch.c
Clarify that --system reindexes system catalogs *only*
commit : dd111887fbaead778d1077dfbc92b520a5188b51
author : Magnus Hagander <magnus@hagander.net>
date : Wed, 27 Oct 2021 16:20:02 +0200
committer: Magnus Hagander <magnus@hagander.net>
date : Wed, 27 Oct 2021 16:20:02 +0200
Make this more clear both in the help message and docs.
Reviewed-By: Michael Paquier
Backpatch-through: 9.6
Discussion: https://postgr.es/m/CABUevEw6Je0WUFTLhPKOk4+BoBuDrE-fKw3N4ckqgDBMFu4paA@mail.gmail.com
M doc/src/sgml/ref/reindexdb.sgml
M src/bin/scripts/reindexdb.c
Reject huge_pages=on if shared_memory_type=sysv.
commit : 24b7cf8a5cff0b0e2dba12a3a1f5c7638ace9986
author : Thomas Munro <tmunro@postgresql.org>
date : Tue, 26 Oct 2021 12:54:55 +1300
committer: Thomas Munro <tmunro@postgresql.org>
date : Tue, 26 Oct 2021 12:54:55 +1300
It doesn't work (it could, but hasn't been implemented).
Back-patch to 12, where shared_memory_type arrived.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/163271880203.22789.1125998876173795966@wrigleys.postgresql.org
M doc/src/sgml/config.sgml
M src/backend/port/sysv_shmem.c
Fix CREATE INDEX CONCURRENTLY for the newest prepared transactions.
commit : a9d0a54094153d8714a66444e89e565bb3eb21e4
author : Noah Misch <noah@leadboat.com>
date : Sat, 23 Oct 2021 18:36:38 -0700
committer: Noah Misch <noah@leadboat.com>
date : Sat, 23 Oct 2021 18:36:38 -0700
The purpose of commit 8a54e12a38d1545d249f1402f66c8cde2837d97c was to
fix this, and it sufficed when the PREPARE TRANSACTION completed before
the CIC looked for lock conflicts. Otherwise, things still broke. As
before, in a cluster having used CIC while having enabled prepared
transactions, queries that use the resulting index can silently fail to
find rows. It may be necessary to reindex to recover from past
occurrences; REINDEX CONCURRENTLY suffices. Fix this for future index
builds by making CIC wait for arbitrarily-recent prepared transactions
and for ordinary transactions that may yet PREPARE TRANSACTION. As part
of that, have PREPARE TRANSACTION transfer locks to its dummy PGPROC
before it calls ProcArrayClearTransaction(). Back-patch to 9.6 (all
supported versions).
Andrey Borodin, reviewed (in earlier versions) by Andres Freund.
Discussion: https://postgr.es/m/01824242-AA92-4FE9-9BA7-AEBAFFEA3D0C@yandex-team.ru
A contrib/amcheck/t/003_cic_2pc.pl
M src/backend/access/transam/twophase.c
M src/backend/access/transam/xact.c
M src/backend/storage/lmgr/lmgr.c
M src/backend/storage/lmgr/lock.c
M src/backend/utils/cache/inval.c
M src/include/access/twophase.h
M src/include/storage/lock.h
M src/test/perl/PostgresNode.pm
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
commit : 2e33b43599ad3868ee180f149050444bf0a7ca15
author : Noah Misch <noah@leadboat.com>
date : Sat, 23 Oct 2021 18:36:38 -0700
committer: Noah Misch <noah@leadboat.com>
date : Sat, 23 Oct 2021 18:36:38 -0700
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
M contrib/amcheck/Makefile
A contrib/amcheck/t/002_cic.pl
M src/backend/utils/cache/inval.c
M src/backend/utils/cache/relcache.c
M src/include/utils/inval.h
M src/include/utils/relcache.h
M src/test/perl/PostgresNode.pm
M src/tools/pgindent/typedefs.list
doc: Describe calculation method of streaming start for pg_receivewal
commit : 7c949f1b3aab8f0cf02afa42c44a9dfc66188c70
author : Michael Paquier <michael@paquier.xyz>
date : Sat, 23 Oct 2021 14:43:45 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Sat, 23 Oct 2021 14:43:45 +0900
The documentation was imprecise about the starting LSN used for WAL
streaming if nothing can be found in the local archive directory
defined with the pg_receivewal command, so be more talkative on this
matter.
Extracted from a larger patch by the same author.
Author: Ronan Dunklau, Michael Paquier
Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan
Backpatch-through: 10
M doc/src/sgml/ref/pg_receivewal.sgml
Fix frontend version of sh_error() in simplehash.h.
commit : 2e01d050d989d67f491349018a09574427d1d0ba
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 22 Oct 2021 16:43:38 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 22 Oct 2021 16:43:38 -0400
The code does not expect sh_error() to return, but the patch
that made this header usable in frontend didn't get that memo.
While here, plaster unlikely() on the tests that decide whether
to invoke sh_error(), and add our standard copyright notice.
Noted by Andres Freund. Back-patch to v13 where this frontend
support came in.
Discussion: https://postgr.es/m/0D54435C-1199-4361-9D74-2FBDCF8EA164@anarazel.de
M src/include/lib/simplehash.h
pg_dump: fix mis-dumping of non-global default privileges.
commit : 476006023538730d2291125b883bfe07d03f3dfa
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 22 Oct 2021 15:22:26 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 22 Oct 2021 15:22:26 -0400
Non-global default privilege entries should be dumped as-is,
not made relative to the default ACL for their object type.
This would typically only matter if one had revoked some
on-by-default privileges in a global entry, and then wanted
to grant them again in a non-global entry.
Per report from Boris Korzun. This is an old bug, so back-patch
to all supported branches.
Neil Chen, test case by Masahiko Sawada
Discussion: https://postgr.es/m/111621616618184@mail.yandex.ru
Discussion: https://postgr.es/m/CAA3qoJnr2+1dVJObNtfec=qW4Z0nz=A9+r5bZKoTSy5RDjskMw@mail.gmail.com
M src/bin/pg_dump/pg_dump.c
M src/bin/pg_dump/t/002_pg_dump.pl
Back-patch "Add parent table name in an error in reorderbuffer.c."
commit : 23469b867ac000b8ed36fdeaab5c40ba9be23772
author : Amit Kapila <akapila@postgresql.org>
date : Thu, 21 Oct 2021 09:36:27 +0530
committer: Amit Kapila <akapila@postgresql.org>
date : Thu, 21 Oct 2021 09:36:27 +0530
This was originally done in commit 5e77625b26 for 15 only, as a
troubleshooting aid but multiple people showed interest in back-patching
this.
Author: Jeremy Schneider
Reviewed-by: Amit Kapila
Backpatch-through: 9.6
Discussion: https://postgr.es/m/808ed65b-994c-915a-361c-577f088b837f@amazon.com
M src/backend/replication/logical/reorderbuffer.c
Protect against collation variations in test
commit : a73a3671daea34f244e1c51321d714be3200f2d9
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Wed, 20 Oct 2021 13:05:42 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Wed, 20 Oct 2021 13:05:42 -0300
Discussion: https://postgr.es/m/YW/MYdSRQZtPFBWR@paquier.xyz
M src/test/regress/expected/alter_table.out
M src/test/regress/sql/alter_table.sql
Fix build of MSVC with OpenSSL 3.0.0
commit : abb9ee92c5070d2b742aea9175c6a59ee403049f
author : Michael Paquier <michael@paquier.xyz>
date : Wed, 20 Oct 2021 16:49:00 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Wed, 20 Oct 2021 16:49:00 +0900
The build scripts of Visual Studio would fail to detect properly a 3.0.0
build as the check on the second digit was failing. This is adjusted
where needed, allowing the builds to complete. Note that the MSIs of
OpenSSL mentioned in the documentation have not changed any library
names for Win32 and Win64, making this change straight-forward.
Reported-by: htalaco, via github
Reviewed-by: Daniel Gustafsson
Discussion: https://postgr.es/m/YW5XKYkq6k7OtrFq@paquier.xyz
Backpatch-through: 9.6
M src/tools/msvc/Solution.pm
Ensure correct lock level is used in ALTER ... RENAME
commit : 842fe6123c8acd09fac904ffb89bee883b36ac12
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Tue, 19 Oct 2021 19:08:45 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Tue, 19 Oct 2021 19:08:45 -0300
Commit 1b5d797cd4f7 intended to relax the lock level used to rename
indexes, but inadvertently allowed *any* relation to be renamed with a
lowered lock level, as long as the command is spelled ALTER INDEX.
That's undesirable for other relation types, so retry the operation with
the higher lock if the relation turns out not to be an index.
After this fix, ALTER INDEX <sometable> RENAME will require access
exclusive lock, which it didn't before.
Author: Nathan Bossart <bossartn@amazon.com>
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Reported-by: Onder Kalaci <onderk@microsoft.com>
Discussion: https://postgr.es/m/PH0PR21MB1328189E2821CDEC646F8178D8AE9@PH0PR21MB1328.namprd21.prod.outlook.com
M src/backend/commands/tablecmds.c
M src/test/regress/expected/alter_table.out
M src/test/regress/sql/alter_table.sql
Adapt src/test/ldap/t/001_auth.pl to work with openldap 2.5.
commit : 246a035f05c6accca34113de8e78f7afad90b329
author : Andres Freund <andres@anarazel.de>
date : Tue, 19 Oct 2021 10:14:49 -0700
committer: Andres Freund <andres@anarazel.de>
date : Tue, 19 Oct 2021 10:14:49 -0700
ldapsearch's deprecated -h/-p arguments were removed, need to use -H now -
which has been around for over 20 years.
As perltidy insists on reflowing the parameters anyway, change order and
"phrasing" to yield a less confusing layout (per suggestion from Tom Lane).
Discussion: https://postgr.es/m/20211009233850.wvr6apcrw2ai6cnj@alap3.anarazel.de
Backpatch: 11-, where the tests were added.
M src/test/ldap/t/001_auth.pl
Fix assignment to array of domain over composite.
commit : 30e61a8cdc8624102ebdb1abbbb1e45457877c5c
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 19 Oct 2021 13:54:46 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 19 Oct 2021 13:54:46 -0400
An update such as "UPDATE ... SET fld[n].subfld = whatever"
failed if the array elements were domains rather than plain
composites. That's because isAssignmentIndirectionExpr()
failed to cope with the CoerceToDomain node that would appear
in the expression tree in this case. The result would typically
be a crash, and even if we accidentally didn't crash, we'd not
correctly preserve other fields of the same array element.
Per report from Onder Kalaci. Back-patch to v11 where arrays of
domains came in.
Discussion: https://postgr.es/m/PH0PR21MB132823A46AA36F0685B7A29AD8BD9@PH0PR21MB1328.namprd21.prod.outlook.com
M src/backend/executor/execExpr.c
M src/test/regress/expected/domain.out
M src/test/regress/sql/domain.sql
Remove bogus assertion in transformExpressionList().
commit : cf33fb7f4a199213a0a83209b81caa728f5c7750
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 19 Oct 2021 11:35:15 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 19 Oct 2021 11:35:15 -0400
I think when I added this assertion (in commit 8f889b108), I was only
thinking of the use of transformExpressionList at top level of INSERT
and VALUES. But it's also called by transformRowExpr(), which can
certainly occur in an UPDATE targetlist, so it's inappropriate to
suppose that p_multiassign_exprs must be empty. Besides, since the
input is not expected to contain ResTargets, there's no reason it
should contain MultiAssignRefs either. Hence this code need not
be concerned about the state of p_multiassign_exprs, and we should
just drop the assertion.
Per bug #17236 from ocean_li_996. It's been wrong for years,
so back-patch to all supported branches.
Discussion: https://postgr.es/m/17236-3210de9bcba1d7ca@postgresql.org
M src/backend/parser/parse_target.c
Fix bug in TOC file error message printing
commit : 687fe8a9d7a636d0792da39eb4dc1630741c7937
author : Daniel Gustafsson <dgustafsson@postgresql.org>
date : Tue, 19 Oct 2021 12:59:54 +0200
committer: Daniel Gustafsson <dgustafsson@postgresql.org>
date : Tue, 19 Oct 2021 12:59:54 +0200
If the blob TOC file cannot be parsed, the error message was failing
to print the filename as the variable holding it was shadowed by the
destination buffer for parsing. When the filename fails to parse,
the error will print an empty string:
./pg_restore -d foo -F d dump
pg_restore: error: invalid line in large object TOC file "": ..
..instead of the intended error message:
./pg_restore -d foo -F d dump
pg_restore: error: invalid line in large object TOC file "dump/blobs.toc": ..
Fix by renaming both variables as the shared name was too generic to
store either and still convey what the variable held.
Backpatch all the way down to 9.6.
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/A2B151F5-B32B-4F2C-BA4A-6870856D9BDE@yesql.se
Backpatch-through: 9.6
M src/bin/pg_dump/pg_backup_directory.c
Fix sscanf limits in pg_basebackup and pg_dump
commit : d3a4c1eb3d2188e8df58b4f2837cd17acc8629e1
author : Daniel Gustafsson <dgustafsson@postgresql.org>
date : Tue, 19 Oct 2021 12:59:50 +0200
committer: Daniel Gustafsson <dgustafsson@postgresql.org>
date : Tue, 19 Oct 2021 12:59:50 +0200
Make sure that the string parsing is limited by the size of the
destination buffer.
In pg_basebackup the available values sent from the server
is limited to two characters so there was no risk of overflow.
In pg_dump the buffer is bounded by MAXPGPATH, and thus the limit
must be inserted via preprocessor expansion and the buffer increased
by one to account for the terminator. There is no risk of overflow
here, since in this case, the buffer scanned is smaller than the
destination buffer.
Backpatch the pg_basebackup fix to 11 where it was introduced, and
the pg_dump fix all the way down to 9.6.
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/B14D3D7B-F98C-4E20-9459-C122C67647FB@yesql.se
Backpatch-through: 11 and 9.6
M src/bin/pg_basebackup/streamutil.c
M src/bin/pg_dump/pg_backup_directory.c
Block ALTER INDEX/TABLE index_name ALTER COLUMN colname SET (options)
commit : 85dc4292a7a1c0b9943cd285cbb3a90a5e368b3f
author : Michael Paquier <michael@paquier.xyz>
date : Tue, 19 Oct 2021 11:04:04 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Tue, 19 Oct 2021 11:04:04 +0900
The grammar of this command run on indexes with column names has always
been authorized by the parser, and it has never been documented.
Since 911e702, it is possible to define opclass parameters as of CREATE
INDEX, which actually broke the old case of ALTER INDEX/TABLE where
relation-level parameters n_distinct and n_distinct_inherited could be
defined for an index (see 76a47c0 and its thread where this point has
been touched, still remained unused). Attempting to do that in v13~
would cause the index to become unusable, as there is a new dedicated
code path to load opclass parameters instead of the relation-level ones
previously available. Note that it is possible to fix things with a
manual catalog update to bring the relation back online.
This commit disables this command for now as the use of column names for
indexes does not make sense anyway, particularly when it comes to index
expressions where names are automatically computed. One way to properly
support this case properly in the future would be to use column numbers
when it comes to indexes, in the same way as ALTER INDEX .. ALTER COLUMN
.. SET STATISTICS.
Partitioned indexes were already blocked, but not indexes. Some tests
are added for both cases.
There was some code in ANALYZE to enforce n_distinct to be used for an
index expression if the parameter was defined, but just remove it for
now until/if there is support for this (note that index-level parameters
never had support in pg_dump either, previously), so this was just dead
code.
Reported-by: Matthijs van der Vleuten
Author: Nathan Bossart, Michael Paquier
Reviewed-by: Vik Fearing, Dilip Kumar
Discussion: https://postgr.es/m/17220-15d684c6c2171a83@postgresql.org
Backpatch-through: 13
M src/backend/commands/analyze.c
M src/backend/commands/tablecmds.c
M src/test/regress/expected/btree_index.out
M src/test/regress/sql/btree_index.sql
Invalidate partitions of table being attached/detached
commit : fe35528a5ed6934e0d60c19a02798236160d93f1
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 18 Oct 2021 19:08:25 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 18 Oct 2021 19:08:25 -0300
Failing to do that, any direct inserts/updates of those partitions
would fail to enforce the correct constraint, that is, one that
considers the new partition constraint of their parent table.
Backpatch to 10.
Reported by: Hou Zhijie <houzj.fnst@fujitsu.com>
Author: Amit Langote <amitlangote09@gmail.com>
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Nitin Jadhav <nitinjadhavpostgres@gmail.com>
Reviewed-by: Pavel Borisov <pashkin.elfe@gmail.com>
Discussion: https://postgr.es/m/OS3PR01MB5718DA1C4609A25186D1FBF194089%40OS3PR01MB5718.jpnprd01.prod.outlook.com
M src/backend/commands/tablecmds.c
M src/test/regress/expected/alter_table.out
M src/test/regress/sql/alter_table.sql
Reset properly snapshot export state during transaction abort
commit : 8f4fe8d7f8dc790545c84be4ed7af5d5d47d5ea7
author : Michael Paquier <michael@paquier.xyz>
date : Mon, 18 Oct 2021 11:56:52 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Mon, 18 Oct 2021 11:56:52 +0900
During a replication slot creation, an ERROR generated in the same
transaction as the one creating a to-be-exported snapshot would have
left the backend in an inconsistent state, as the associated static
export snapshot state was not being reset on transaction abort, but only
on the follow-up command received by the WAL sender that created this
snapshot on replication slot creation. This would trigger inconsistency
failures if this session tried to export again a snapshot, like during
the creation of a replication slot.
Note that a snapshot export cannot happen in a transaction block, so
there is no need to worry resetting this state for subtransaction
aborts. Also, this inconsistent state would very unlikely show up to
users. For example, one case where this could happen is an
out-of-memory error when building the initial snapshot to-be-exported.
Dilip found this problem while poking at a different patch, that caused
an error in this code path for reasons unrelated to HEAD.
Author: Dilip Kumar
Reviewed-by: Michael Paquier, Zhihong Yu
Discussion: https://postgr.es/m/CAFiTN-s0zA1Kj0ozGHwkYkHwa5U0zUE94RSc_g81WrpcETB5=w@mail.gmail.com
Backpatch-through: 9.6
M src/backend/access/transam/xact.c
M src/backend/replication/logical/snapbuild.c
M src/include/replication/snapbuild.h
Avoid core dump in pg_dump when dumping from pre-8.3 server.
commit : 0b5f557b7e87527c0f14230326b7e279c9fd26c1
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 16 Oct 2021 15:02:55 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 16 Oct 2021 15:02:55 -0400
Commit f0e21f2f6 missed adding a tgisinternal output column
to getTriggers' query for pre-8.3 servers. Back-patch to v11,
like that commit.
M src/bin/pg_dump/pg_dump.c
Make pg_dump acquire lock on partitioned tables that are to be dumped.
commit : 6a262ba8c8648d7f2bfb419313d2d5d6daefeeb7
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 16 Oct 2021 12:23:57 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 16 Oct 2021 12:23:57 -0400
It was clearly the intent to do so all along, but the original coding
fat-fingered this by checking the wrong array element. We fixed it
in passing in 403a3d91c, but that later got reverted, and we forgot
to keep this bug fix.
Most of the time this'd be relatively harmless, since once we lock
any of the partitioned table's leaf partitions, that would suffice
to prevent major DDL on the partitioned table itself. However, a
childless partitioned table would get dumped with no relevant lock
whatsoever, possibly allowing dump failure or inconsistent output.
Unlike 403a3d91c, there are no versioning concerns, since every server
version that has partitioned tables will allow you to lock one.
Back-patch to v10 where partitioned tables were introduced.
Discussion: https://postgr.es/m/1018205.1634346327@sss.pgh.pa.us
M src/bin/pg_dump/pg_dump.c
Check criticalSharedRelcachesBuilt in GetSharedSecurityLabel().
commit : 20f785732fdd8872d9942761e2fbc1677adec9b7
author : Jeff Davis <jdavis@postgresql.org>
date : Thu, 14 Oct 2021 12:24:47 -0700
committer: Jeff Davis <jdavis@postgresql.org>
date : Thu, 14 Oct 2021 12:24:47 -0700
An extension may want to call GetSecurityLabel() on a shared object
before the shared relcaches are fully initialized. For instance, a
ClientAuthentication_hook might want to retrieve the security label on
a role.
Discussion: https://postgr.es/m/ecb7af0b26e3be1d96d291c8453a86f1f82d9061.camel@j-davis.com
Backpatch-through: 9.6
M src/backend/commands/seclabel.c
Fix planner error with pulling up subquery expressions into function RTEs.
commit : fdd6a4d8d90aee36d2d81f0ab29399021228183a
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 14 Oct 2021 12:43:43 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 14 Oct 2021 12:43:43 -0400
If a function-in-FROM laterally references the output of some sub-SELECT
earlier in the FROM clause, and we are able to flatten that sub-SELECT
into the outer query, the expression(s) copied into the function RTE
missed being processed by eval_const_expressions. This'd lead to trouble
and probable crashes at execution if such expressions contained
named-argument function call syntax or functions with defaulted arguments.
The bug is masked if the query contains any explicit JOIN syntax, which
may help explain why we'd not noticed.
Per bug #17227 from Bernd Dorn. This is an oversight in commit 7266d0997,
so back-patch to v13 where that came in.
Discussion: https://postgr.es/m/17227-5a28ed1512189fa4@postgresql.org
M src/backend/optimizer/plan/planner.c
M src/test/regress/expected/rangefuncs.out
M src/test/regress/sql/rangefuncs.sql
Change recently added test code for stability
commit : 2cdf97fd1ec86301c70ccffb2414a6047c9f8dd1
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Wed, 13 Oct 2021 18:49:27 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Wed, 13 Oct 2021 18:49:27 -0300
The test code added with ff9f111bce24 fails under valgrind, and probably
other slow cases too, because if (say) autovacuum runs in between and
produces WAL of its own, the large INSERT fails to account for that in
the LSN calculations. Rewrite to use a DO loop.
Per complaint from Andres Freund
Backpatch to all branches.
Discussion: https://postgr.es/m/20211013180338.5guyqzpkcisqugrl@alap3.anarazel.de
M src/test/recovery/t/026_overwrite_contrecord.pl
postgres_fdw: Move comments about elog level in (sub)abort cleanup.
commit : cdab0a80d28501692584e798abbbfcfe9b5a6b48
author : Etsuro Fujita <efujita@postgresql.org>
date : Wed, 13 Oct 2021 19:00:03 +0900
committer: Etsuro Fujita <efujita@postgresql.org>
date : Wed, 13 Oct 2021 19:00:03 +0900
The comments were misplaced when adding postgres_fdw. Fix that by
moving the comments to more appropriate functions.
Author: Etsuro Fujita
Backpatch-through: 9.6
Discussion: https://postgr.es/m/CAPmGK164sAXQtC46mDFyu6d-T25Mzvh5qaRNkit06VMmecYnOA%40mail.gmail.com
M contrib/postgres_fdw/connection.c
Fix tests of pg_upgrade across different major versions
commit : 2a8dee6a67ccde6074f463786d9152dbb87c7eb9
author : Michael Paquier <michael@paquier.xyz>
date : Wed, 13 Oct 2021 09:22:00 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Wed, 13 Oct 2021 09:22:00 +0900
This fixes a set of issues that cause different breakages or annoyances
when using pg_upgrade's test.sh to do upgrades across different major
versions:
- test.sh is completely broken when using v14 as new version because of
the removal of testtablespace/ as Makefile rule. Older versions of
pg_regress don't support --make-tablespacedir, blocking the creation of
the tablespace. In order to fix that, it is simple enough to create
those directories in the script itself, but only do that when an old
version is involved. This fix is needed on HEAD and REL_14_STABLE.
- The script would fail when using PG <= v11 as old version because of
WITH OIDS relations not supported in v12. In order to fix this, this
steals a method from the buildfarm that uses a DO block to change all
the relations marked as WITH OIDS, allowing pg_upgrade to pass. This is
more portable than using ALTER TABLE queries on the relations causing
issues. This is fixed down to v12, and authored originally by Andrew
Dunstan.
- Not using --extra-float-digits=0 with v11 as old version causes
a lot of diffs in the dumps, making the whole unreadable. This gets
only done when using v11 as old version. This is fixed down to v12.
The buildfarm code uses that already.
Note that the addition of --wal-segsize and --allow-group-access breaks
the script when using v10 or older at initdb time as these got added in
11. 10 would be EOL'd next year and nobody has complained about those
problems yet, so nothing is done about that. This means that this
commit fixes upgrade tests using test.sh with v11 as minimum older
version, up to HEAD, and that it is enough to apply this change down to
12. The old and new dumps still generate diffs, still require manual
checks, and more could be done to reduce the noise, but this allows the
tests to run with a rather minimal amount of them.
I have tested this commit and test.sh with v11 as minimum across all the
branches where this is applied. Note that this commit has no impact on
the normal pg_upgrade test run with a simple "make check".
Author: Justin Pryzby, Andrew Dunstan, Michael Paquier
Discussion: https://postgr.es/m/20201206180248.GI24052@telsasoft.com
Backpatch-through: 12
M src/bin/pg_upgrade/test.sh
Add more $Test::Builder::Level in the TAP tests
commit : bab0ff2e44b781ec1b1180c5e96bdb95c385c80b
author : Michael Paquier <michael@paquier.xyz>
date : Tue, 12 Oct 2021 11:16:25 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Tue, 12 Oct 2021 11:16:25 +0900
Incrementing the level of the call stack reported is useful for
debugging purposes as it allows to control which part of the test is
exactly failing, especially if a test is structured with subroutines
that call routines from Test::More.
This adds more incrementations of $Test::Builder::Level where debugging
gets improved (for example it does not make sense for some paths like
pg_rewind where long subroutines are used).
A note is added to src/test/perl/README about that, based on a
suggestion from Andrew Dunstan and a wording coming from both of us.
Usage of Test::Builder::Level has spread in 12, so a backpatch down to
this version is done.
Reviewed-by: Andrew Dunstan, Peter Eisentraut, Daniel Gustafsson
Discussion: https://postgr.es/m/YV1CCFwgM1RV1LeS@paquier.xyz
Backpatch-through: 12
M src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
M src/bin/pg_verifybackup/t/005_bad_manifest.pl
M src/bin/psql/t/010_tab_completion.pl
M src/test/kerberos/t/001_auth.pl
M src/test/perl/README
M src/test/recovery/t/001_stream_rep.pl
M src/test/recovery/t/003_recovery_targets.pl
M src/test/recovery/t/007_sync_rep.pl
M src/test/recovery/t/009_twophase.pl
M src/test/recovery/t/018_wal_optimize.pl
Add missing word to comment in joinrels.c.
commit : 08f37e2592920a6df9584821614aa8deb93ed39f
author : Etsuro Fujita <efujita@postgresql.org>
date : Thu, 7 Oct 2021 17:45:03 +0900
committer: Etsuro Fujita <efujita@postgresql.org>
date : Thu, 7 Oct 2021 17:45:03 +0900
Author: Amit Langote
Backpatch-through: 13
Discussion: https://postgr.es/m/CA%2BHiwqGQNbtamQ_9DU3osR1XiWR4wxWFZurPmN6zgbdSZDeWmw%40mail.gmail.com
M src/backend/optimizer/path/joinrels.c
Fix null-pointer crash in postgres_fdw's conversion_error_callback.
commit : aee83f39a86bb66baa66d65e87ca8415ac26800b
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 6 Oct 2021 15:50:24 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 6 Oct 2021 15:50:24 -0400
Commit c7b7311f6 adjusted conversion_error_callback to always use
information from the query's rangetable, to avoid doing catalog lookups
in an already-failed transaction. However, as a result of the utterly
inadequate documentation for make_tuple_from_result_row, I failed to
realize that fsstate could be NULL in some contexts. That led to a
crash if we got a conversion error in such a context. Fix by falling
back to the previous coding when fsstate is NULL. Improve the
commentary, too.
Per report from Andrey Borodin. Back-patch to 9.6, like the previous
patch.
Discussion: https://postgr.es/m/08916396-55E4-4D68-AB3A-BD6066F9E5C0@yandex-team.ru
M contrib/postgres_fdw/expected/postgres_fdw.out
M contrib/postgres_fdw/postgres_fdw.c
M contrib/postgres_fdw/sql/postgres_fdw.sql
Fix corner-case loss of precision in numeric_power().
commit : 9ab94ccb15af288f9ff295b4f04b28d86115b803
author : Dean Rasheed <dean.a.rasheed@gmail.com>
date : Wed, 6 Oct 2021 13:20:23 +0100
committer: Dean Rasheed <dean.a.rasheed@gmail.com>
date : Wed, 6 Oct 2021 13:20:23 +0100
This fixes a loss of precision that occurs when the first input is
very close to 1, so that its logarithm is very small.
Formerly, during the initial low-precision calculation to estimate the
result weight, the logarithm was computed to a local rscale that was
capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be
as close as 1e-16383 to 1, hence its logarithm may be as small as
1e-16383, and so the local rscale needs to be allowed to exceed 16383,
otherwise all precision is lost, leading to a poor choice of rscale
for the full-precision calculation.
Fix this by removing the cap on the local rscale during the initial
low-precision calculation, as we already do in the full-precision
calculation. This doesn't change the fact that the initial calculation
is a low-precision approximation, computing the logarithm to around 8
significant digits, which is very fast, especially when the base is
very close to 1.
Patch by me, reviewed by Alvaro Herrera.
Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com
M src/backend/utils/adt/numeric.c
M src/test/regress/expected/numeric.out
M src/test/regress/sql/numeric.sql
Fix warning in TAP test of pg_verifybackup
commit : d6d68e223379985661df45c728c32c3b6d462629
author : Michael Paquier <michael@paquier.xyz>
date : Wed, 6 Oct 2021 13:28:35 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Wed, 6 Oct 2021 13:28:35 +0900
Oversight in a3fcbcd.
Reported-by: Thomas Munro
Discussion: https://postgr.es/m/CA+hUKGKnajZEwe91OTjro9kQLCMGGFHh2vvFn8tgHgbyn4bF9w@mail.gmail.com
Backpatch-through: 13
M src/bin/pg_verifybackup/t/007_wal.pl
Doc: improve description of UNION/INTERSECT/EXCEPT syntax.
commit : 9024a35c11d7e8c42c2489ae2f912e4d0572f1df
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 5 Oct 2021 10:24:14 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 5 Oct 2021 10:24:14 -0400
queries.sgml failed to mention the rather important point that
INTERSECT binds more tightly than UNION or EXCEPT. I thought
it could also use more discussion of the role of parentheses
in these constructs.
Per gripe from Christopher Painter-Wakefield.
Discussion: https://postgr.es/m/163338891727.12510.3939775743980651160@wrigleys.postgresql.org
M doc/src/sgml/queries.sgml
doc: remove URL for ICU explorer/locexp
commit : 0a6549316ee852ebea843671b755dc6d54b0e85b
author : Bruce Momjian <bruce@momjian.us>
date : Mon, 4 Oct 2021 17:10:59 -0400
committer: Bruce Momjian <bruce@momjian.us>
date : Mon, 4 Oct 2021 17:10:59 -0400
The old URL was HTTP 404 and the git link didn't build. Also update two
other ICU links. If we ever get a good link we will add it back.
Reported-by: Anton Voloshin
Author: Laurenz Albe
Backpatch-through: 10
M doc/src/sgml/charset.sgml
Fix TestLib::slurp_file() with offset on windows.
commit : 5ba397f7404e911c7822e85441d42673080c5b98
author : Andres Freund <andres@anarazel.de>
date : Mon, 4 Oct 2021 13:28:06 -0700
committer: Andres Freund <andres@anarazel.de>
date : Mon, 4 Oct 2021 13:28:06 -0700
3c5b0685b921 used setFilePointer() to set the position of the filehandle, but
passed the wrong filehandle, always leaving the position at 0. Instead of just
fixing that, remove use of setFilePointer(), we have a perl fd at this point,
so we can just use perl's seek().
Additionally, the perl filehandle wasn't closed, just the windows filehandle.
Reviewed-By: Andrew Dunstan <andrew@dunslane.net>
Author: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20211003173038.64mmhgxctfqn7wl6@alap3.anarazel.de
Backpatch: 9.6-, like 3c5b0685b921
M src/test/perl/TestLib.pm
Update our mapping of Windows time zone names some more.
commit : c53ff69e1fd24ae4905fe50ce8d3ca3a08d6a3c2
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 4 Oct 2021 14:52:17 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 4 Oct 2021 14:52:17 -0400
Per discussion, let's just follow CLDR's default zone mappings
faithfully. There are two changes here that are clear improvements:
* Mapping "Greenwich Standard Time" to Atlantic/Reykjavik is actually
a better fit than using London, because Iceland hasn't observed DST
since 1968, so this is more nearly what people might expect.
* Since the "Samoa" zone is specified to be UTC+13:00, we must map
it to Pacific/Apia not Pacific/Samoa; the latter refers to American
Samoa which is now on the other side of the date line.
The rest of these changes look like they're choosing the most populous
IANA zone as representative. Whatever the details, we're just going
to say "if you don't like this mapping, complain to CLDR".
Discussion: https://postgr.es/m/3266414.1633045628@sss.pgh.pa.us
M src/bin/initdb/findtimezone.c
Fix snapshot builds during promotion of hot standby node with 2PC
commit : 194e535a07b3c36738b67f59aaa1291276929e9b
author : Michael Paquier <michael@paquier.xyz>
date : Mon, 4 Oct 2021 14:05:52 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Mon, 4 Oct 2021 14:05:52 +0900
Some specific logic is done at the end of recovery when involving 2PC
transactions:
1) Call RecoverPreparedTransactions(), to recover the state of 2PC
transactions into memory (re-acquire locks, etc.).
2) ShutdownRecoveryTransactionEnvironment(), to move back to normal
operations, mainly cleaning up recovery locks and KnownAssignedXids
(including any 2PC transaction tracked previously).
3) Switch XLogCtl->SharedRecoveryState to RECOVERY_STATE_DONE, which is
the tipping point for any process calling RecoveryInProgress() to check
if the cluster is still in recovery or not.
Any snapshot taken between steps 2) and 3) would be empty, causing any
transaction relying on a snapshot at this point to potentially corrupt
data as there could still be some 2PC transactions to track, with
RecentXmin moving backwards on successive calls to GetSnapshotData() in
the same transaction.
As SharedRecoveryState is the point to take into account to know if it
is safe to discard KnownAssignedXids, this commit moves step 2) after
step 3), so as we can never finish with empty snapshots.
This exists since the introduction of hot standby, so backpatch all the
way down. The window with incorrect snapshots is extremely small, but I
have seen it when running 023_pitr_prepared_xact.pl, as did buildfarm
member fairywren. Thomas Munro also found it independently. Special
thanks to Andres Freund for taking the time to analyze this issue.
Reported-by: Thomas Munro, Michael Paquier
Analyzed-by: Andres Freund
Discussion: https://postgr.es/m/20210422203603.fdnh3fu2mmfp2iov@alap3.anarazel.de
Backpatch-through: 9.6
M src/backend/access/transam/xlog.c
Update our mapping of Windows time zone names using CLDR info.
commit : 9c76689de442fc5849e383cc370d0769f8196bf6
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 2 Oct 2021 16:05:42 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 2 Oct 2021 16:05:42 -0400
This corrects a bunch of entries in win32_tzmap[], and adds a few
new ones, based on the CLDR project's windowsZones.xml file.
Non-cosmetic changes fall into four main categories:
* Flat-out errors:
US/Aleutan doesn't exist
America/Salvador doesn't exist
Asia/Baku is wrong for Yerevan
Asia/Dhaka (Bangladesh) is wrong for Astana (Kazakhstan)
Europe/Bucharest is wrong for Chisinau
America/Mexico_City is wrong for Chetumal
America/Buenos_Aires is wrong for Cayenne
America/Caracas has its own zone, so poor fit for La Paz
US/Eastern is wrong for Haiti
US/Eastern is wrong for Indiana (East)
Asia/Karachi is wrong for Tashkent
Etc/UTC+12 doesn't exist
Signs of Etc/GMT zones were backwards
* Judgment calls:
(These changes follow CLDR's choices, except for the first one)
Use Europe/London for "Greenwich Standard Time", since that seems much
more likely than Africa/Casablanca to be what people will think that
zone name means. CLDR has Atlantic/Reykjavik here, but that's no better.
Asia/Shanghai seems a better fit than Hong Kong for "China Standard
Time".
Europe/Sarajevo is now a link to Belgrade, ie "Central Europe Standard
Time"; so use Warsaw for "Central European Standard Time".
America/Sao_Paulo seems more representative than Araguaina for
"E. South America Standard Time".
Africa/Johannesburg seems more representative than Harare for
"South Africa Standard Time".
* New Windows zone names:
"Israel Standard Time"
"Kaliningrad Standard Time"
"Russia Time Zone N" for various N
"Singapore Standard Time"
"South Sudan Standard Time"
"W. Central Africa Standard Time"
"West Bank Standard Time"
"Yukon Standard Time"
Some of these replace older spellings, but I kept the older spellings
too in case our code runs on a machine with the older data.
* Replace aliases (tzdb Links) with underlying city-named zones:
(This tracks tzdb's longstanding practice, and reduces inconsistency
with the rest of the entries, as well as with CLDR.)
US/Alaska
Asia/Kuwait
Asia/Muscat
Canada/Atlantic
Australia/Canberra
Canada/Saskatchewan
US/Central
US/Eastern
US/Hawaii
US/Mountain
Canada/Newfoundland
US/Pacific
Back-patch to all supported branches, as is our usual practice for
time zone data updates.
Discussion: https://postgr.es/m/3266414.1633045628@sss.pgh.pa.us
M src/bin/initdb/findtimezone.c
Re-alphabetize the win32_tzmap[] array.
commit : 7ba8eb81f6b79e7c1aaa3001137f38a61de80aba
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 2 Oct 2021 16:05:10 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 2 Oct 2021 16:05:10 -0400
The original intent seems to have been to sort case-insensitively
by the Windows zone name, but various changes over the years did
not get that memo. This commit just moves a few entries to
restore exact alphabetic order, to ease comparison to the outputs
of processing scripts.
Back-patch to all supported branches, as is our usual practice for
time zone data updates.
Discussion: https://postgr.es/m/3266414.1633045628@sss.pgh.pa.us
M src/bin/initdb/findtimezone.c
Error out if SKIP LOCKED and WITH TIES are both specified
commit : 170206e458bce1d4be34bb805f884b2735430519
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Fri, 1 Oct 2021 18:29:18 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Fri, 1 Oct 2021 18:29:18 -0300
Both bugs #16676[1] and #17141[2] illustrate that the combination of
SKIP LOCKED and FETCH FIRST WITH TIES break expectations when it comes
to rows returned to other sessions accessing the same row. Since this
situation is detectable from the syntax and hard to fix otherwise,
forbid for now, with the potential to fix in the future.
[1] https://postgr.es/m/16676-fd62c3c835880da6@postgresql.org
[2] https://postgr.es/m/17141-913d78b9675aac8e@postgresql.org
Backpatch-through: 13, where WITH TIES was introduced
Author: David Christensen <david.christensen@crunchydata.com>
Discussion: https://postgr.es/m/CAOxo6XLPccCKru3xPMaYDpa+AXyPeWFs+SskrrL+HKwDjJnLhg@mail.gmail.com
M doc/src/sgml/ref/select.sgml
M src/backend/commands/matview.c
M src/backend/parser/gram.y
M src/test/regress/expected/limit.out
M src/test/regress/sql/limit.sql
Avoid believing incomplete MCV-only stats in get_variable_range().
commit : 7adbe186f7fd7e036d2cab7d8ed8ee99354ec219
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 1 Oct 2021 14:59:35 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 1 Oct 2021 14:59:35 -0400
get_variable_range() would incautiously believe that statistics
containing only an MCV list are sufficient to derive a range estimate.
That's okay for an enum-like column that contains only MCVs, but
otherwise the estimate could be pretty bad. Make it report that the
range is indeterminate unless the MCVs plus nullfrac account for
the whole table.
I don't think this needs a dedicated test case, since a quick code
coverage check verifies that the existing regression tests traverse
all the alternatives. There is room to doubt that a future-proof
test case could be built anyway, given that the submitted example
accidentally doesn't fail before v11.
Per bug #17207 from Simon Perepelitsa. Back-patch to v10.
In principle this has been broken all along, but I'm hesitant to
make such changes in 9.6, since if anyone is unhappy with 9.6.24's
behavior there will be no second chance to fix it.
Discussion: https://postgr.es/m/17207-5265aefa79e333b4@postgresql.org
M src/backend/utils/adt/selfuncs.c
Fix Portal snapshot tracking to handle subtransactions properly.
commit : 04ef2021e3ca7207b87ffce437b0b4db73a7f6b2
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 1 Oct 2021 11:10:12 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 1 Oct 2021 11:10:12 -0400
Commit 84f5c2908 forgot to consider the possibility that
EnsurePortalSnapshotExists could run inside a subtransaction with
lifespan shorter than the Portal's. In that case, the new active
snapshot would be popped at the end of the subtransaction, leaving
a dangling pointer in the Portal, with mayhem ensuing.
To fix, make sure the ActiveSnapshot stack entry is marked with
the same subtransaction nesting level as the associated Portal.
It's certainly safe to do so since we won't be here at all unless
the stack is empty; hence we can't create an out-of-order stack.
Let's also apply this logic in the case where PortalRunUtility
sets portalSnapshot, just to be sure that path can't cause similar
problems. It's slightly less clear that that path can't create
an out-of-order stack, so add an assertion guarding it.
Report and patch by Bertrand Drouvot (with kibitzing by me).
Back-patch to v11, like the previous commit.
Discussion: https://postgr.es/m/ff82b8c5-77f4-3fe7-6028-fcf3303e82dd@amazon.com
M src/backend/access/transam/xact.c
M src/backend/tcop/pquery.c
M src/backend/utils/mmgr/portalmem.c
M src/backend/utils/time/snapmgr.c
M src/include/utils/portal.h
M src/include/utils/snapmgr.h
M src/pl/plpgsql/src/expected/plpgsql_transaction.out
M src/pl/plpgsql/src/sql/plpgsql_transaction.sql
Remove gratuitous environment dependency in 002_types.pl test.
commit : 649e561f65c579630e5c31ee65308eefdd21ec93
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 30 Sep 2021 16:23:10 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 30 Sep 2021 16:23:10 -0400
Computing related timestamps by subtracting "N days" is sensitive
to the prevailing timezone, since we interpret that as "same local
time on the N'th prior day". Even though the intervals in question
are only two to four days, through remarkable bad luck they managed
to cross the end of Ramadan in 2014, causing the test's output to
change if timezone is set to Africa/Casablanca. (Maybe in other
Muslim areas as well; I didn't check.) There's absolutely no reason
for this test to exercise interval subtraction, so just get rid of
that and use plain timestamptz constants representing the intended
values.
Per report from Andres Freund. Back-patch to v10 where this test
script came in.
Discussion: https://postgr.es/m/20210930183641.7lh4jhvpipvromca@alap3.anarazel.de
M src/test/subscription/t/002_types.pl
Fix WAL replay in presence of an incomplete record
commit : 1d97d3d0867faf967a3ab977b6a0b675b6a451da
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Wed, 29 Sep 2021 11:21:51 -0300
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Wed, 29 Sep 2021 11:21:51 -0300
Physical replication always ships WAL segment files to replicas once
they are complete. This is a problem if one WAL record is split across
a segment boundary and the primary server crashes before writing down
the segment with the next portion of the WAL record: WAL writing after
crash recovery would happily resume at the point where the broken record
started, overwriting that record ... but any standby or backup may have
already received a copy of that segment, and they are not rewinding.
This causes standbys to stop following the primary after the latter
crashes:
LOG: invalid contrecord length 7262 at A8/D9FFFBC8
because the standby is still trying to read the continuation record
(contrecord) for the original long WAL record, but it is not there and
it will never be. A workaround is to stop the replica, delete the WAL
file, and restart it -- at which point a fresh copy is brought over from
the primary. But that's pretty labor intensive, and I bet many users
would just give up and re-clone the standby instead.
A fix for this problem was already attempted in commit 515e3d84a0b5, but
it only addressed the case for the scenario of WAL archiving, so
streaming replication would still be a problem (as well as other things
such as taking a filesystem-level backup while the server is down after
having crashed), and it had performance scalability problems too; so it
had to be reverted.
This commit fixes the problem using an approach suggested by Andres
Freund, whereby the initial portion(s) of the split-up WAL record are
kept, and a special type of WAL record is written where the contrecord
was lost, so that WAL replay in the replica knows to skip the broken
parts. With this approach, we can continue to stream/archive segment
files as soon as they are complete, and replay of the broken records
will proceed across the crash point without a hitch.
Because a new type of WAL record is added, users should be careful to
upgrade standbys first, primaries later. Otherwise they risk the standby
being unable to start if the primary happens to write such a record.
A new TAP test that exercises this is added, but the portability of it
is yet to be seen.
This has been wrong since the introduction of physical replication, so
backpatch all the way back. In stable branches, keep the new
XLogReaderState members at the end of the struct, to avoid an ABI
break.
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Reviewed-by: Nathan Bossart <bossartn@amazon.com>
Discussion: https://postgr.es/m/202108232252.dh7uxf6oxwcy@alvherre.pgsql
M src/backend/access/rmgrdesc/xlogdesc.c
M src/backend/access/transam/xlog.c
M src/backend/access/transam/xlogreader.c
M src/include/access/xlog_internal.h
M src/include/access/xlogreader.h
M src/include/catalog/pg_control.h
A src/test/recovery/t/026_overwrite_contrecord.pl
M src/tools/pgindent/typedefs.list
pgbench: Fix handling of socket errors during benchmark.
commit : 8cf4f7118596a2a3f5b51c04f73f1a06b4e363da
author : Fujii Masao <fujii@postgresql.org>
date : Wed, 29 Sep 2021 21:01:10 +0900
committer: Fujii Masao <fujii@postgresql.org>
date : Wed, 29 Sep 2021 21:01:10 +0900
Previously socket errors such as invalid socket or socket wait method failures
during benchmark caused pgbench to exit with status 0. Instead, errors during
the run should result in exit status 2.
Back-patch to v12 where pgbench started reporting exit status.
Original complaint and patch by Hayato Kuroda.
Author: Yugo Nagata, Fabien COELHO
Reviewed-by: Kyotaro Horiguchi, Fujii Masao
Discussion: https://postgr.es/m/TYCPR01MB5870057375ACA8A73099C649F5349@TYCPR01MB5870.jpnprd01.prod.outlook.com
M src/bin/pgbench/pgbench.c
pgbench: Correct log level of message output when socket wait method fails.
commit : 3cc85d7d5353945bf5bdb26bb7771f40aeacaf0b
author : Fujii Masao <fujii@postgresql.org>
date : Wed, 29 Sep 2021 20:35:00 +0900
committer: Fujii Masao <fujii@postgresql.org>
date : Wed, 29 Sep 2021 20:35:00 +0900
The failure of socket wait method like "select()" doesn't terminate pgbench.
So the log level of error message when that failure happens should be ERROR.
But previously FATAL was used in that case.
Back-patch to v13 where pgbench started using common logging API.
Author: Yugo Nagata, Fabien COELHO
Reviewed-by: Kyotaro Horiguchi, Fujii Masao
Discussion: https://postgr.es/m/20210617005934.8bd37bf72efd5f1b38e6f482@sraoss.co.jp
M src/bin/pgbench/pgbench.c
Fix instability in contrib/bloom TAP tests.
commit : cf26a8d8a75ff4ae78bf0818ebb2bf6cf90db69c
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 28 Sep 2021 17:34:31 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 28 Sep 2021 17:34:31 -0400
It turns out that the instability complained of in commit d3c09b9b1
has an embarrassingly simple explanation. The test script waits for
the standby to flush incoming WAL to disk, but it should wait for
the WAL to be replayed, since we are testing for the effects of that
to be visible.
While at it, use wait_for_catchup instead of reinventing that logic,
and adjust $Test::Builder::Level to improve future error reports.
Back-patch to v12 where the necessary infrastructure came in
(cf. aforesaid commit). Also back-patch 7d1aa6bf1 so that the
test will actually get run.
Discussion: https://postgr.es/m/2854602.1632852664@sss.pgh.pa.us
M contrib/bloom/Makefile
M contrib/bloom/t/001_wal.pl
Fix typos in docs
commit : 78515b89f2985ef78136732cd239609c569797b3
author : Michael Paquier <michael@paquier.xyz>
date : Sun, 26 Sep 2021 19:18:27 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Sun, 26 Sep 2021 19:18:27 +0900
Author: Justin Pryzby
Discussion: https://postgr.es/m/20210924215827.GS831@telsasoft.com
Backpatch-through: 9.6
M doc/src/sgml/catalogs.sgml
M doc/src/sgml/datatype.sgml
Doc: extend warnings about collation-mismatch hazards in postgres_fdw.
commit : 739b872f6c98459a460b522e726f369b6fc67ca3
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 25 Sep 2021 10:53:54 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 25 Sep 2021 10:53:54 -0400
Be a little more vocal about the risks of remote collations not
matching local ones. Actually fixing these risks seems hard,
and I've given up on the idea that it might be back-patchable.
So the best we can do for the back branches is add documentation.
Per discussion of bug #16583 from Jiří Fejfar.
Discussion: https://postgr.es/m/2438715.1632510693@sss.pgh.pa.us
M doc/src/sgml/postgres-fdw.sgml
Add alternative output for OpenSSL 3 without legacy loaded
commit : 8e7199453bf9fe142f3f4a5e17010320c24867e7
author : Daniel Gustafsson <dgustafsson@postgresql.org>
date : Sat, 25 Sep 2021 11:27:28 +0200
committer: Daniel Gustafsson <dgustafsson@postgresql.org>
date : Sat, 25 Sep 2021 11:27:28 +0200
OpenSSL 3 introduced the concept of providers to support modularization,
and moved the outdated ciphers to the new legacy provider. In case it's
not loaded in the users openssl.cnf file there will be a lot of regress
test failures, so add alternative outputs covering those.
Also document the need to load the legacy provider in order to use older
ciphers with OpenSSL-enabled pgcrypto.
This will be backpatched to all supported version once there is sufficient
testing in the buildfarm of OpenSSL 3.
Reviewed-by: Michael Paquier
Discussion: https://postgr.es/m/FEF81714-D479-4512-839B-C769D2605F8A@yesql.se
Backpatch-through: 9.6
A contrib/pgcrypto/expected/blowfish_1.out
A contrib/pgcrypto/expected/cast5_1.out
A contrib/pgcrypto/expected/des_1.out
A contrib/pgcrypto/expected/pgp-decrypt_1.out
A contrib/pgcrypto/expected/pgp-pubkey-decrypt_1.out
M doc/src/sgml/pgcrypto.sgml
Disable OpenSSL EVP digest padding in pgcrypto
commit : 135d8687adf12a0d4cd7c94d1095ed5a7a08f7ed
author : Daniel Gustafsson <dgustafsson@postgresql.org>
date : Sat, 25 Sep 2021 11:27:20 +0200
committer: Daniel Gustafsson <dgustafsson@postgresql.org>
date : Sat, 25 Sep 2021 11:27:20 +0200
The PX layer in pgcrypto is handling digest padding on its own uniformly
for all backend implementations. Starting with OpenSSL 3.0.0, DecryptUpdate
doesn't flush the last block in case padding is enabled so explicitly
disable it as we don't use it.
This will be backpatched to all supported version once there is sufficient
testing in the buildfarm of OpenSSL 3.
Reviewed-by: Peter Eisentraut, Michael Paquier
Discussion: https://postgr.es/m/FEF81714-D479-4512-839B-C769D2605F8A@yesql.se
Backpatch-through: 9.6
M contrib/pgcrypto/openssl.c
pgcrypto: Check for error return of px_cipher_decrypt()
commit : a69e1506f618d4577bf7fdbfea51924a44c6e7de
author : Daniel Gustafsson <dgustafsson@postgresql.org>
date : Sat, 25 Sep 2021 11:25:48 +0200
committer: Daniel Gustafsson <dgustafsson@postgresql.org>
date : Sat, 25 Sep 2021 11:25:48 +0200
This has previously not been a problem (that anyone ever reported),
but in future OpenSSL versions (3.0.0), where legacy ciphers are/can
be disabled, this is the place where this is reported. So we need to
catch the error here, otherwise the higher-level functions would
return garbage. The nearby encryption code already handled errors
similarly.
Author: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://www.postgresql.org/message-id/9e9c431c-0adc-7a6d-9b1a-915de1ba3fe7@enterprisedb.com
Backpatch-through: 9.6
M contrib/pgcrypto/px.c
doc: Improve description of index vacuuming with GUCs
commit : 52f8575a9e75e67ed7a7ae1585be28a85e85ae0e
author : Michael Paquier <michael@paquier.xyz>
date : Sat, 25 Sep 2021 15:12:00 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Sat, 25 Sep 2021 15:12:00 +0900
Index vacuums may happen multiple times depending on the number of dead
tuples stored, as of maintenance_work_mem for a manual VACUUM. For
autovacuum, this is controlled by autovacuum_work_mem instead, if set.
The documentation mentioned the former, but not the latter in the
context of autovacuum.
Reported-by: Nikolai Berkoff
Author: Laurenz Albe, Euler Taveira
Discussion: https://postgr.es/m/161545365522.10134.12195402324485546870@wrigleys.postgresql.org
Backpatch-through: 9.6
M doc/src/sgml/monitoring.sgml
doc: Add missing markup in CREATE EVENT TRIGGER page
commit : ca925fe3c4d3727679f00deedde83765b89c32bb
author : Michael Paquier <michael@paquier.xyz>
date : Sat, 25 Sep 2021 14:48:13 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Sat, 25 Sep 2021 14:48:13 +0900
Reported-by: rir
Discussion: https://postgr.es/m/20210924183658.3syyitp3yuxjv2fp@localhost
Backpatch-through: 9.6
M doc/src/sgml/ref/create_event_trigger.sgml
Split macros from visibilitymap.h into a separate header
commit : fd22aec631af9f2b967af20e23206840c6594c72
author : Alexander Korotkov <akorotkov@postgresql.org>
date : Thu, 23 Sep 2021 19:59:03 +0300
committer: Alexander Korotkov <akorotkov@postgresql.org>
date : Thu, 23 Sep 2021 19:59:03 +0300
That allows to include just visibilitymapdefs.h from file.c, and in turn,
remove include of postgres.h from relcache.h.
Reported-by: Andres Freund
Discussion: https://postgr.es/m/20210913232614.czafiubr435l6egi%40alap3.anarazel.de
Author: Alexander Korotkov
Reviewed-by: Andres Freund, Tom Lane, Alvaro Herrera
Backpatch-through: 13
M src/bin/pg_upgrade/file.c
M src/include/access/visibilitymap.h
A src/include/access/visibilitymapdefs.h
M src/include/utils/relcache.h
Release memory allocated by dependency_degree
commit : c0386f403a832f50d61fc7bb9bb265f7700273f3
author : Tomas Vondra <tomas.vondra@postgresql.org>
date : Tue, 21 Sep 2021 01:13:11 +0200
committer: Tomas Vondra <tomas.vondra@postgresql.org>
date : Tue, 21 Sep 2021 01:13:11 +0200
Calculating degree of a functional dependency may allocate a lot of
memory - we have released mot of the explicitly allocated memory, but
e.g. detoasted varlena values were left behind. That may be an issue,
because we consider a lot of dependencies (all combinations), and the
detoasting may happen for each one again.
Fixed by calling dependency_degree() in a dedicated context, and
resetting it after each call. We only need the calculated dependency
degree, so we don't need to copy anything.
Backpatch to PostgreSQL 10, where extended statistics were introduced.
Backpatch-through: 10
Discussion: https://www.postgresql.org/message-id/20210915200928.GP831%40telsasoft.com
M src/backend/statistics/dependencies.c
Free memory after building each statistics object
commit : b564eb0181e678ee58c1be2e9f3c53eeb68f8c4b
author : Tomas Vondra <tomas.vondra@postgresql.org>
date : Tue, 21 Sep 2021 01:14:11 +0200
committer: Tomas Vondra <tomas.vondra@postgresql.org>
date : Tue, 21 Sep 2021 01:14:11 +0200
Until now, all extended statistics on a given relation were built in the
same memory context, without resetting. Some of the memory was released
explicitly, but not all of it - for example memory allocated while
detoasting values is hard to free. This is how it worked since extended
statistics were introduced in PostgreSQL 10, but adding support for
extended stats on expressions made the issue somewhat worse as it
increases the number of statistics to build.
Fixed by adding a memory context which gets reset after building each
statistics object (all the statistics kinds included in it). Resetting
it after building each statistics kind would be even better, but it
would require more invasive changes and copying of results, making it
harder to backpatch.
Backpatch to PostgreSQL 10, where extended statistics were introduced.
Author: Justin Pryzby
Reported-by: Justin Pryzby
Reviewed-by: Tomas Vondra
Backpatch-through: 10
Discussion: https://www.postgresql.org/message-id/20210915200928.GP831%40telsasoft.com
M src/backend/statistics/extended_stats.c
Invalidate all partitions for a partitioned table in publication.
commit : f09a81f1c08fa5c2fed12bed88d53a11a72ff993
author : Amit Kapila <akapila@postgresql.org>
date : Wed, 22 Sep 2021 08:24:20 +0530
committer: Amit Kapila <akapila@postgresql.org>
date : Wed, 22 Sep 2021 08:24:20 +0530
Updates/Deletes on a partition were allowed even without replica identity
after the parent table was added to a publication. This would later lead
to an error on subscribers. The reason was that we were not invalidating
the partition's relcache and the publication information for partitions
was not getting rebuilt. Similarly, we were not invalidating the
partitions' relcache after dropping a partitioned table from a publication
which will prohibit Updates/Deletes on its partition without replica
identity even without any publication.
Reported-by: Haiying Tang
Author: Hou Zhijie and Vignesh C
Reviewed-by: Vignesh C and Amit Kapila
Backpatch-through: 13
Discussion: https://postgr.es/m/OS0PR01MB6113D77F583C922F1CEAA1C3FBD29@OS0PR01MB6113.jpnprd01.prod.outlook.com
M src/backend/catalog/pg_publication.c
M src/backend/commands/publicationcmds.c
M src/include/catalog/pg_publication.h
M src/include/commands/publicationcmds.h
M src/test/regress/expected/publication.out
M src/test/regress/sql/publication.sql
Fix places in TestLib.pm in need of adaptation to the output of Msys perl
commit : 583e15af9976f8905e2b27f985d619e758270e16
author : Michael Paquier <michael@paquier.xyz>
date : Wed, 22 Sep 2021 08:43:10 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Wed, 22 Sep 2021 08:43:10 +0900
Contrary to the output of native perl, Msys perl generates outputs with
CRLFs characters. There are already places in the TAP code where CRLFs
(\r\n) are automatically converted to LF (\n) on Msys, but we missed a
couple of places when running commands and using their output for
comparison, that would lead to failures.
This problem has been found thanks to the test added in 5adb067 using
TestLib::command_checks_all(), but after a closer look more code paths
were missing a filter.
This is backpatched all the way down to prevent any surprises if a new
test is introduced in stable branches.
Reviewed-by: Andrew Dunstan, Álvaro Herrera
Discussion: https://postgr.es/m/1252480.1631829409@sss.pgh.pa.us
Backpatch-through: 9.6
M src/test/perl/TestLib.pm
Fix misevaluation of STABLE parameters in CALL within plpgsql.
commit : 5f0a073cbbf8bdc276e570ffa006e762fc815e44
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 21 Sep 2021 19:06:33 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 21 Sep 2021 19:06:33 -0400
Before commit 84f5c2908, a STABLE function in a plpgsql CALL
statement's argument list would see an up-to-date snapshot,
because exec_stmt_call would push a new snapshot. I got rid of
that because the possibility of the snapshot disappearing within
COMMIT made it too hard to manage a snapshot across the CALL
statement. That's fine so far as the procedure itself goes,
but I forgot to think about the possibility of STABLE functions
within the CALL argument list. As things now stand, those'll
be executed with the Portal's snapshot as ActiveSnapshot,
keeping them from seeing updates more recent than Portal startup.
(VOLATILE functions don't have a problem because they take their
own snapshots; which indeed is also why the procedure itself
doesn't have a problem. There are no STABLE procedures.)
We can fix this by pushing a new snapshot transiently within
ExecuteCallStmt itself. Popping the snapshot before we get
into the procedure proper eliminates the management problem.
The possibly-useless extra snapshot-grab is slightly annoying,
but it's no worse than what happened before 84f5c2908.
Per bug #17199 from Alexander Nawratil. Back-patch to v11,
like the previous patch.
Discussion: https://postgr.es/m/17199-1ab2561f0d94af92@postgresql.org
M src/backend/commands/functioncmds.c
M src/pl/plpgsql/src/expected/plpgsql_transaction.out
M src/pl/plpgsql/src/sql/plpgsql_transaction.sql
Remove overzealous index deletion assertion.
commit : a1708ab652eaef3e9405d5119721a9a4ecb6fcbd
author : Peter Geoghegan <pg@bowt.ie>
date : Mon, 20 Sep 2021 14:26:22 -0700
committer: Peter Geoghegan <pg@bowt.ie>
date : Mon, 20 Sep 2021 14:26:22 -0700
A broken HOT chain is not an unexpected condition, even when the offset
number points past the end of the page's line pointer array.
heap_prune_chain() does not (and never has) treated this condition as
unexpected, so derivative code in heap_index_delete_tuples() shouldn't
do so either.
Oversight in commit 4228817449.
The assertion can probably only fail on Postgres 14 and master. Earlier
releases don't have commit 3c3b8a4b, which taught VACUUM to truncate the
line pointer array of heap pages. Backpatch all the same, just to be
consistent.
Author: Peter Geoghegan <pg@bowt.ie>
Reported-By: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/17197-9438f31f46705182@postgresql.org
Backpatch: 12-, just like commit 4228817449.
M src/backend/access/heap/heapam.c
Don't elide casting to typmod -1.
commit : dede1439978120d2196e0d3963bb809a8b262e5b
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 20 Sep 2021 11:48:52 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 20 Sep 2021 11:48:52 -0400
Casting a value that's already of a type with a specific typmod
to an unspecified typmod doesn't do anything so far as run-time
behavior is concerned. However, it really ought to change the
exposed type of the expression to match. Up to now,
coerce_type_typmod hasn't bothered with that, which creates gotchas
in contexts such as recursive unions. If for example one side of
the union is numeric(18,3), but it needs to be plain numeric to
match the other side, there's no direct way to express that.
This is easy enough to fix, by inserting a RelabelType to update the
exposed type of the expression. However, it's a bit nervous-making
to change this behavior, because it's stood for a really long time.
But no complaints have emerged about 14beta3, so go ahead and
back-patch.
Back-patch of 5c056b0c2 into previous supported branches.
Discussion: https://postgr.es/m/CABNQVagu3bZGqiTjb31a8D5Od3fUMs7Oh3gmZMQZVHZ=uWWWfQ@mail.gmail.com
Discussion: https://postgr.es/m/1488389.1631984807@sss.pgh.pa.us
M src/backend/parser/parse_coerce.c
M src/test/regress/expected/expressions.out
M src/test/regress/sql/expressions.sql
Doc: fix typos.
commit : 89b5676b65b5f27fcab4f972390197983d7a9bfe
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sun, 19 Sep 2021 11:36:53 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sun, 19 Sep 2021 11:36:53 -0400
"PGcon" should be "PGconn". Noted by D. Frey.
Discussion: https://postgr.es/m/163191739352.4680.16994248583642672629@wrigleys.postgresql.org
M doc/src/sgml/lobj.sgml
Fix pull_varnos to cope with translated PlaceHolderVars.
commit : e0b0d1eab48c85f961a39c263d2074ad11931920
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 17 Sep 2021 15:41:16 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 17 Sep 2021 15:41:16 -0400
Commit 55dc86eca changed pull_varnos to use (if possible) the associated
ph_eval_at for a PlaceHolderVar. I missed a fine point though: we might
be looking at a PHV in the quals or tlist of a child appendrel, in which
case we need to compute a ph_eval_at value that's been translated in the
same way that the PHV itself has been (cf. adjust_appendrel_attrs).
Fortunately, enough info is available in the PlaceHolderInfo to make
such translation possible without additional outside data, so we don't
need another round of uglification of planner APIs. This is a little
bit complicated, but since it's a hard-to-hit corner case, I'm not much
worried about adding cycles here.
Per report from Jaime Casanova. Back-patch to v12, like the previous
commit.
Discussion: https://postgr.es/m/20210915230959.GB17635@ahch-to
M src/backend/optimizer/util/var.c
M src/test/regress/expected/join.out
M src/test/regress/sql/join.sql
Fix variable shadowing in procarray.c.
commit : 8767a86fa2b1d24537cca318a26fc68d7e457911
author : Fujii Masao <fujii@postgresql.org>
date : Thu, 16 Sep 2021 13:07:29 +0900
committer: Fujii Masao <fujii@postgresql.org>
date : Thu, 16 Sep 2021 13:07:29 +0900
ProcArrayGroupClearXid function has a parameter named "proc",
but the same name was used for its local variables. This commit fixes
this variable shadowing, to improve code readability.
Back-patch to all supported versions, to make future back-patching
easy though this patch is classified as refactoring only.
Reported-by: Ranier Vilela
Author: Ranier Vilela, Aleksander Alekseev
https://postgr.es/m/CAEudQAqyoTZC670xWi6w-Oe2_Bk1bfu2JzXz6xRfiOUzm7xbyQ@mail.gmail.com
M src/backend/storage/ipc/procarray.c
Disallow LISTEN in background workers.
commit : e06cc024bd3372cb4280246e114d7508e3e2c20a
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 15 Sep 2021 12:31:56 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 15 Sep 2021 12:31:56 -0400
It's possible to execute user-defined SQL in some background processes;
for example, logical replication workers can fire triggers. This opens
the possibility that someone would try to execute LISTEN in such a
context. But since only regular backends ever call
ProcessNotifyInterrupt, no messages would actually be received, and
thus the registered listener would simply prevent the message queue
from being cleaned. Eventually NOTIFY would stop working, which is bad.
Perhaps someday somebody will invent infrastructure to make listening
in a background worker actually useful. In the meantime, forbid it.
Back-patch to v13, which is where we introduced the MyBackendType
variable. It'd be a lot harder to implement the check without that,
and it doesn't seem worth the trouble.
Discussion: https://postgr.es/m/153243441449.1404.2274116228506175596@wrigleys.postgresql.org
M src/backend/tcop/utility.c
Send NOTIFY signals during CommitTransaction.
commit : 63f28776cb9fbefd99c5e2134f02c80057b8b717
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 14 Sep 2021 17:18:25 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 14 Sep 2021 17:18:25 -0400
Formerly, we sent signals for outgoing NOTIFY messages within
ProcessCompletedNotifies, which was also responsible for sending
relevant ones of those messages to our connected client. It therefore
had to run during the main-loop processing that occurs just before
going idle. This arrangement had two big disadvantages:
* Now that procedures allow intra-command COMMITs, it would be
useful to send NOTIFYs to other sessions immediately at COMMIT
(though, for reasons of wire-protocol stability, we still shouldn't
forward them to our client until end of command).
* Background processes such as replication workers would not send
NOTIFYs at all, since they never execute the client communication
loop. We've had requests to allow triggers running in replication
workers to send NOTIFYs, so that's a problem.
To fix these things, move transmission of outgoing NOTIFY signals
into AtCommit_Notify, where it will happen during CommitTransaction.
Also move the possible call of asyncQueueAdvanceTail there, to
ensure we don't bloat the async SLRU if a background worker sends
many NOTIFYs with no one listening.
We can also drop the call of asyncQueueReadAllNotifications,
allowing ProcessCompletedNotifies to go away entirely. That's
because commit 790026972 added a call of ProcessNotifyInterrupt
adjacent to PostgresMain's call of ProcessCompletedNotifies,
and that does its own call of asyncQueueReadAllNotifications,
meaning that we were uselessly doing two such calls (inside two
separate transactions) whenever inbound notify signals coincided
with an outbound notify. We need only set notifyInterruptPending
to ensure that ProcessNotifyInterrupt runs, and we're done.
The existing documentation suggests that custom background workers
should call ProcessCompletedNotifies if they want to send NOTIFY
messages. To avoid an ABI break in the back branches, reduce it
to an empty routine rather than removing it entirely. Removal
will occur in v15.
Although the problems mentioned above have existed for awhile,
I don't feel comfortable back-patching this any further than v13.
There was quite a bit of churn in adjacent code between 12 and 13.
At minimum we'd have to also backpatch 51004c717, and a good deal
of other adjustment would also be needed, so the benefit-to-risk
ratio doesn't look attractive.
Per bug #15293 from Michael Powers (and similar gripes from others).
Artur Zakirov and Tom Lane
Discussion: https://postgr.es/m/153243441449.1404.2274116228506175596@wrigleys.postgresql.org
M doc/src/sgml/bgworker.sgml
M src/backend/access/transam/xact.c
M src/backend/commands/async.c
M src/backend/tcop/postgres.c
M src/include/commands/async.h
jit: Do not try to shut down LLVM state in case of LLVM triggered errors.
commit : c49e6f9d974966c7b0fc3eac43a6be922cf6eaee
author : Andres Freund <andres@anarazel.de>
date : Mon, 13 Sep 2021 18:07:19 -0700
committer: Andres Freund <andres@anarazel.de>
date : Mon, 13 Sep 2021 18:07:19 -0700
If an allocation failed within LLVM it is not safe to call back into LLVM as
LLVM is not generally safe against exceptions / stack-unwinding. Thus errors
while in LLVM code are promoted to FATAL. However llvm_shutdown() did call
back into LLVM even in such cases, while llvm_release_context() was careful
not to do so.
We cannot generally skip shutting down LLVM, as that can break profiling. But
it's OK to do so if there was an error from within LLVM.
Reported-By: Jelte Fennema <Jelte.Fennema@microsoft.com>
Author: Andres Freund <andres@anarazel.de>
Author: Justin Pryzby <pryzby@telsasoft.com>
Discussion: https://postgr.es/m/AM5PR83MB0178C52CCA0A8DEA0207DC14F7FF9@AM5PR83MB0178.EURPRD83.prod.outlook.com
Backpatch: 11-, where jit was introduced
M src/backend/jit/llvm/llvmjit.c
M src/backend/jit/llvm/llvmjit_error.cpp
M src/include/jit/llvmjit.h
Fix EXIT out of outermost block in plpgsql.
commit : 745abdd951ab31f3276adbbbf67bbc3b7dac0923
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 13 Sep 2021 12:42:03 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 13 Sep 2021 12:42:03 -0400
Ordinarily, using EXIT this way would draw "control reached end of
function without RETURN". However, if the function is one where we
don't require an explicit RETURN (such as a DO block), that should
not happen. It did anyway, because add_dummy_return() neglected to
account for the case.
Per report from Herwig Goemans. Back-patch to all supported branches.
Discussion: https://postgr.es/m/868ae948-e3ca-c7ec-95a6-83cfc08ef750@gmail.com
M src/pl/plpgsql/src/expected/plpgsql_control.out
M src/pl/plpgsql/src/pl_comp.c
M src/pl/plpgsql/src/sql/plpgsql_control.sql
Doc: Remove type information for import_generated in postgres-fdw.sgml.
commit : b7b8e2c4a5cb9d520c907f12fe9ebd3837dffc49
author : Etsuro Fujita <efujita@postgresql.org>
date : Mon, 13 Sep 2021 17:30:02 +0900
committer: Etsuro Fujita <efujita@postgresql.org>
date : Mon, 13 Sep 2021 17:30:02 +0900
The type information for FDW options is only added to HEAD; remove this
from back branches. Oversight in commit aa769f80e.
Apply the patch to v12, v13, and v14.
Discussion: https://postgr.es/m/CAPmGK14z92twaKwRoccHbbh5Va5vbRDZcTYYTx50+0JTQ8xx_g@mail.gmail.com
M doc/src/sgml/postgres-fdw.sgml
Fix reorder buffer memory accounting for toast changes.
commit : 58cf794ca68d90ea11888f200e0b74d01d0c7093
author : Amit Kapila <akapila@postgresql.org>
date : Mon, 13 Sep 2021 10:46:58 +0530
committer: Amit Kapila <akapila@postgresql.org>
date : Mon, 13 Sep 2021 10:46:58 +0530
While processing toast changes in logical decoding, we rejigger the
tuple change to point to in-memory toast tuples instead to on-disk toast
tuples. And, to make sure the memory accounting is correct, we were
subtracting the old change size and then after re-computing the new tuple,
re-adding its size at the end. Now, if there is any error before we add
the new size, we will release the changes and that will update the
accounting info (subtracting the size from the counters). And we were
underflowing there which leads to an assertion failure in assert enabled
builds and wrong memory accounting in reorder buffer otherwise.
Author: Bertrand Drouvot
Reviewed-by: Amit Kapila
Backpatch-through: 13, where memory accounting was introduced
Discussion: https://postgr.es/m/92b0ee65-b8bd-e42d-c082-4f3f4bf12d34@amazon.com
M src/backend/replication/logical/reorderbuffer.c
Fix error handling with threads on OOM in ECPG connection logic
commit : b589d212fdd7e449ec0d5da8e46c0d86eee943d6
author : Michael Paquier <michael@paquier.xyz>
date : Mon, 13 Sep 2021 13:24:20 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Mon, 13 Sep 2021 13:24:20 +0900
An out-of-memory failure happening when allocating the structures to
store the connection parameter keywords and values would mess up with
the set of connections saved, as on failure the pthread mutex would
still be hold with the new connection object listed but free()'d.
Rather than just unlocking the mutex, which would leave the static list
of connections into an inconsistent state, move the allocation for the
structures of the connection parameters before beginning the test
manipulation. This ensures that the list of connections and the
connection mutex remain consistent all the time in this code path.
This error is unlikely going to happen, but this could mess up badly
with ECPG clients in surprising ways, so backpatch all the way down.
Reported-by: ryancaicse
Discussion: https://postgr.es/m/17186-b4cfd8f0eb4d1dee@postgresql.org
Backpatch-through: 9.6
M src/interfaces/ecpg/ecpglib/connect.c
Make pg_regexec() robust against out-of-range search_start.
commit : 7e420072ea4a9f873d3378c8b35f2d939d925022
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 11 Sep 2021 15:19:31 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 11 Sep 2021 15:19:31 -0400
If search_start is greater than the length of the string, we should just
return REG_NOMATCH immediately. (Note that the equality case should
*not* be rejected, since the pattern might be able to match zero
characters.) This guards various internal assumptions that the min of a
range of string positions is not more than the max. Violation of those
assumptions could allow an attempt to fetch string[search_start-1],
possibly causing a crash.
Jaime Casanova pointed out that this situation is reachable with the
new regexp_xxx functions that accept a user-specified start position.
I don't believe it's reachable via any in-core call site in v14 and
below. However, extensions could possibly call pg_regexec with an
out-of-range search_start, so let's back-patch the fix anyway.
Discussion: https://postgr.es/m/20210911180357.GA6870@ahch-to
M src/backend/regex/regexec.c
Fix some anomalies with NO SCROLL cursors.
commit : fa5d0415f10fac3c2aa4c301840fde1780b91b79
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 10 Sep 2021 13:18:32 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 10 Sep 2021 13:18:32 -0400
We have long forbidden fetching backwards from a NO SCROLL cursor,
but the prohibition didn't extend to cases in which we rewind the
query altogether and then re-fetch forwards. I think the reason is
that this logic was mainly meant to protect plan nodes that can't
be run in the reverse direction. However, re-reading the query output
is problematic if the query is volatile (which includes SELECT FOR
UPDATE, not just queries with volatile functions): the re-read can
produce different results, which confuses the cursor navigation logic
completely. Another reason for disliking this approach is that some
code paths will either fetch backwards or rewind-and-fetch-forwards
depending on the distance to the target row; so that seemingly
identical use-cases may or may not draw the "cursor can only scan
forward" error. Hence, let's clean things up by disallowing rewind
as well as fetch-backwards in a NO SCROLL cursor.
Ordinarily we'd only make such a definitional change in HEAD, but
there is a third reason to consider this change now. Commit ba2c6d6ce
created some new user-visible anomalies for non-scrollable cursors
WITH HOLD, in that navigation in the cursor result got confused if the
cursor had been partially read before committing. The only good way
to resolve those anomalies is to forbid rewinding such a cursor, which
allows removal of the incorrect cursor state manipulations that
ba2c6d6ce added to PersistHoldablePortal.
To minimize the behavioral change in the back branches (including
v14), refuse to rewind a NO SCROLL cursor only when it has a holdStore,
ie has been held over from a previous transaction due to WITH HOLD.
This should avoid breaking most applications that have been sloppy
about whether to declare cursors as scrollable. We'll enforce the
prohibition across-the-board beginning in v15.
Back-patch to v11, as ba2c6d6ce was.
Discussion: https://postgr.es/m/3712911.1631207435@sss.pgh.pa.us
M src/backend/commands/portalcmds.c
M src/backend/tcop/pquery.c
M src/test/regress/expected/portals.out
M src/test/regress/sql/portals.sql
Avoid fetching from an already-terminated plan.
commit : d8d93bc8baa934fb23c90271d200e061386b546d
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 9 Sep 2021 13:36:31 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 9 Sep 2021 13:36:31 -0400
Some plan node types don't react well to being called again after
they've already returned NULL. PortalRunSelect() has long dealt
with this by calling the executor with NoMovementScanDirection
if it sees that we've already run the portal to the end. However,
commit ba2c6d6ce overlooked this point, so that persisting an
already-fully-fetched cursor would fail if it had such a plan.
Per report from Tomas Barton. Back-patch to v11, as the faulty
commit was. (I've omitted a test case because the type of plan
that causes a problem isn't all that stable.)
Discussion: https://postgr.es/m/CAPV2KRjd=ErgVGbvO2Ty20tKTEZZr6cYsYLxgN_W3eAo9pf5sw@mail.gmail.com
M src/backend/commands/portalcmds.c
Check for relation length overrun soon enough.
commit : 04118de78fe9a8a7160775b47e89bf21a5efb620
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 9 Sep 2021 11:45:48 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 9 Sep 2021 11:45:48 -0400
We don't allow relations to exceed 2^32-1 blocks, because block
numbers are 32 bits and the last possible block number is reserved
to mean InvalidBlockNumber. There is a check for this in mdextend,
but that's really way too late, because the smgr API requires us to
create a buffer for the block-to-be-added, and we do not want to
have any buffer with blocknum InvalidBlockNumber. (Such a case
can trigger assertions in bufmgr.c, plus I think it might confuse
ReadBuffer's logic for data-past-EOF later on.) So put the check
into ReadBuffer.
Per report from Christoph Berg. It's been like this forever,
so back-patch to all supported branches.
Discussion: https://postgr.es/m/YTn1iTkUYBZfcODk@msg.credativ.de
M src/backend/storage/buffer/bufmgr.c
M src/backend/storage/smgr/md.c
Fix issue with WAL archiving in standby.
commit : dd9b3fced83edb51a3e2f44d3d4476a45d0f5a24
author : Fujii Masao <fujii@postgresql.org>
date : Thu, 9 Sep 2021 23:58:05 +0900
committer: Fujii Masao <fujii@postgresql.org>
date : Thu, 9 Sep 2021 23:58:05 +0900
Previously, walreceiver always closed the currently-opened WAL segment
and created its archive notification file, after it finished writing
the current segment up and received any WAL data that should be
written into the next segment. If walreceiver exited just before
any WAL data in the next segment arrived at standby, it did not
create the archive notification file of the current segment
even though that's known completed. This behavior could cause
WAL archiving of the segment to be delayed until subsequent
restartpoints or checkpoints created its notification file.
To fix the issue, this commit changes walreceiver so that it creates
an archive notification file of a current WAL segment immediately
if that's known completed before receiving next WAL data.
Back-patch to all supported branches.
Reported-by: Kyotaro Horiguchi
Author: Fujii Masao
Reviewed-by: Kyotaro Horiguchi
Discussion: https://postgr.es/m/20200630.165503.1465894182551545886.horikyota.ntt@gmail.com
M src/backend/replication/walreceiver.c
Avoid useless malloc/free traffic around getFormattedTypeName().
commit : da7d81dd0579a2579741bcfc816cf614c58d0aab
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 8 Sep 2021 15:09:42 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 8 Sep 2021 15:09:42 -0400
Coverity complained that one caller of getFormattedTypeName() failed
to free the returned string. Which is true, but rather than fixing
that one, let's get rid of this tedious and error-prone requirement.
Now that getFormattedTypeName() caches its result, strdup'ing that
result and expecting the caller to free it accomplishes little except
to waste cycles. We do create a leak in the case where getTypes didn't
make a TypeInfo for the type, but that basically shouldn't ever happen.
Back-patch, as commit 6c450a861 was. This isn't a particularly
interesting bug fix, but the API change seems like a hazard for
future back-patching activity if we don't back-patch it.
M src/bin/pg_dump/pg_dump.c
Fix rewriter to set hasModifyingCTE correctly on rewritten queries.
commit : cbba6ba3a0628fdc47185f71d5f06ed7af95c147
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 8 Sep 2021 12:05:43 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 8 Sep 2021 12:05:43 -0400
If we copy data-modifying CTEs from the original query to a replacement
query (from a DO INSTEAD rule), we must set hasModifyingCTE properly
in the replacement query. Failure to do this can cause various
unpleasantness, such as unsafe usage of parallel plans. The code also
neglected to propagate hasRecursive, though that's only cosmetic at
the moment.
A difficulty arises if the rule action is an INSERT...SELECT. We
attach the original query's RTEs and CTEs to the sub-SELECT Query, but
data-modifying CTEs are only allowed to appear in the topmost Query.
For the moment, throw an error in such cases. It would probably be
possible to avoid this error by attaching the CTEs to the top INSERT
Query instead; but that would require a bunch of new code to adjust
ctelevelsup references. Given the narrowness of the use-case, and
the need to back-patch this fix, it does not seem worth the trouble
for now. We can revisit this if we get field complaints.
Per report from Greg Nancarrow. Back-patch to all supported branches.
(The test case added here does not fail before v10, but there are
plenty of places checking top-level hasModifyingCTE in 9.6, so I have
no doubt that this code change is necessary there too.)
Greg Nancarrow and Tom Lane
Discussion: https://postgr.es/m/CAJcOf-f68DT=26YAMz_i0+Au3TcLO5oiHY5=fL6Sfuits6r+_w@mail.gmail.com
Discussion: https://postgr.es/m/CAJcOf-fAdj=nDKMsRhQzndm-O13NY4dL6xGcEvdX5Xvbbi0V7g@mail.gmail.com
M src/backend/rewrite/rewriteHandler.c
M src/test/regress/expected/with.out
M src/test/regress/sql/with.sql
Invalidate relcache for publications defined for all tables.
commit : ddfc7299d0a3a41f0c178a8157bd751430428e8a
author : Amit Kapila <akapila@postgresql.org>
date : Wed, 8 Sep 2021 10:20:02 +0530
committer: Amit Kapila <akapila@postgresql.org>
date : Wed, 8 Sep 2021 10:20:02 +0530
Updates/Deletes on a relation were allowed even without replica identity
after we define the publication for all tables. This would later lead to
an error on subscribers. The reason was that for such publications we were
not invalidating the relcache and the publication information for
relations was not getting rebuilt. Similarly, we were not invalidating the
relcache after dropping of such publications which will prohibit
Updates/Deletes without replica identity even without any publication.
Author: Vignesh C and Hou Zhijie
Reviewed-by: Hou Zhijie, Kyotaro Horiguchi, Amit Kapila
Backpatch-through: 10, where it was introduced
Discussion: https://postgr.es/m/CALDaNm0pF6zeWqCA8TCe2sDuwFAy8fCqba=nHampCKag-qLixg@mail.gmail.com
M src/backend/commands/publicationcmds.c
M src/test/regress/expected/publication.out
M src/test/regress/sql/publication.sql
AIX: Fix missing libpq symbols by respecting SHLIB_EXPORTS.
commit : aae398a87cfafcb1d62fd1e464b2c4d2525a039a
author : Noah Misch <noah@leadboat.com>
date : Mon, 6 Sep 2021 11:27:59 -0700
committer: Noah Misch <noah@leadboat.com>
date : Mon, 6 Sep 2021 11:27:59 -0700
We make each AIX shared library export all globals found in .o files
that originate in the library. That doesn't include symbols acquired by
-lpgcommon_shlib. That is good on average, but it became a problem for
libpq when commit e6afa8918c461c1dd80c5063a950518fa4e950cd moved five
official libpq API symbols into src/common. Fix this by implementing
the SHLIB_EXPORTS mechanism for AIX, so affected libraries export the
same symbols that they export on Linux. This reintroduces symbols
pg_encoding_to_char, pg_utf_mblen, pg_char_to_encoding,
pg_valid_server_encoding, and pg_valid_server_encoding_id. Back-patch
to v13, where the aforementioned commit first appeared. While a minor
release is usually the wrong time to add or remove symbol exports in
libpq or libecpg, we should expect users to want each documented symbol.
Tony Reix
Discussion: https://postgr.es/m/PR3PR02MB6396742E2FC3E77D37A920BC86C79@PR3PR02MB6396.eurprd02.prod.outlook.com
M src/Makefile.shlib
M src/port/README
Fix bogus timetz_zone() results for DYNTZ abbreviations.
commit : d8a266c5e1bd50785d4619ecb3e686f86771685f
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 6 Sep 2021 11:29:52 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 6 Sep 2021 11:29:52 -0400
timetz_zone() delivered completely wrong answers if the zone was
specified by a dynamic TZ abbreviation, because it failed to account
for the difference between the POSIX conventions for field values in
struct pg_tm and the conventions used in PG-specific datetime code.
As a stopgap fix, just adjust the tm_year and tm_mon fields to match
PG conventions. This is fixed in a different way in HEAD (388e71af8)
but I don't want to back-patch the change of reference point.
Discussion: https://postgr.es/m/CAJ7c6TOMG8zSNEZtCn5SPe+cCk3Lfxb71ZaQwT2F4T7PJ_t=KA@mail.gmail.com
M src/backend/utils/adt/date.c
Fix pkg-config files for static linking
commit : 9f9ae019d194d012c74084cec3722ba2d67af8eb
author : Peter Eisentraut <peter@eisentraut.org>
date : Mon, 6 Sep 2021 09:41:03 +0200
committer: Peter Eisentraut <peter@eisentraut.org>
date : Mon, 6 Sep 2021 09:41:03 +0200
Since ea53100d5 (PostgreSQL 12), the shipped pkg-config files have
been broken for statically linking libpq because libpgcommon and
libpgport are missing. This patch adds those two missing private
dependencies (in a non-hardcoded way).
Reported-by: Filip Gospodinov <f@gospodinov.ch>
Discussion: https://www.postgresql.org/message-id/flat/c7108bde-e051-11d5-a234-99beec01ce2a@gospodinov.ch
M src/Makefile.shlib
Further portability tweaks for float4/float8 hash functions.
commit : 2c0dd669c3c909484c24b40a797bd99b352c6f0b
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 4 Sep 2021 16:29:08 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Sat, 4 Sep 2021 16:29:08 -0400
Attempting to make hashfloat4() look as much as possible like
hashfloat8(), I'd figured I could replace NaNs with get_float4_nan()
before widening to float8. However, results from protosciurus
and topminnow show that on some platforms that produces a different
bit-pattern from get_float8_nan(), breaking the intent of ce773f230.
Rearrange so that we use the result of get_float8_nan() for all NaN
cases. As before, back-patch.
M src/backend/access/hash/hashfunc.c
Revert "Avoid creating archive status ".ready" files too early"
commit : 518621c40b23b4188478578d21de62d25582d9e3
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Sat, 4 Sep 2021 12:14:30 -0400
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Sat, 4 Sep 2021 12:14:30 -0400
This reverts commit 515e3d84a0b5 and equivalent commits in back
branches. This solution to the problem has a number of problems, so
we'll try again with a different approach.
Per note from Andres Freund
Discussion: https://postgr.es/m/20210831042949.52eqp5xwbxgrfank@alap3.anarazel.de
M src/backend/access/transam/xlog.c
M src/backend/postmaster/walwriter.c
M src/include/access/xlog.h
M src/include/access/xlogdefs.h
Remove arbitrary MAXPGPATH limit on command lengths in pg_ctl.
commit : 742b30caee65a8d54388c1a249e93f27c65315f5
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 3 Sep 2021 21:04:44 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 3 Sep 2021 21:04:44 -0400
Replace fixed-length command buffers with psprintf() calls. We didn't
have anything as convenient as psprintf() when this code was written,
but now that we do, there's little reason for the limitation to
stand. Removing it eliminates some corner cases where (for example)
starting the postmaster with a whole lot of options fails.
Most individual file names that pg_ctl deals with are still restricted
to MAXPGPATH, but we've seldom had complaints about that limitation
so long as it only applies to one filename.
Back-patch to all supported branches.
Phil Krylov
Discussion: https://postgr.es/m/567e199c6b97ee19deee600311515b86@krylov.eu
M src/bin/pg_ctl/pg_ctl.c
Disallow creating an ICU collation if the DB encoding won't support it.
commit : 132be60006b334df00093973465215cd8b892f37
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 3 Sep 2021 16:38:55 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 3 Sep 2021 16:38:55 -0400
Previously this was allowed, but the collation effectively vanished
into the ether because of the way lookup_collation() works: you could
not use the collation, nor even drop it. Seems better to give an
error up front than to leave the user wondering why it doesn't work.
(Because this test is in DefineCollation not CreateCollation, it does
not prevent pg_import_system_collations from creating ICU collations,
regardless of the initially-chosen encoding.)
Per bug #17170 from Andrew Bille. Back-patch to v10 where ICU support
was added.
Discussion: https://postgr.es/m/17170-95845cf3f0a9c36d@postgresql.org
M src/backend/commands/collationcmds.c
Fix portability issue in tests from commit ce773f230.
commit : 9089f1543ef50414e1cecf89bc34eeb3ab1cb968
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 3 Sep 2021 10:01:02 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 3 Sep 2021 10:01:02 -0400
Modern POSIX seems to require strtod() to accept "-NaN", but there's
nothing about NaN in SUSv2, and some of our oldest buildfarm members
don't like it. Let's try writing it as -'NaN' instead; that seems
to produce the same result, at least on Intel hardware.
Per buildfarm.
M src/test/regress/expected/hash_func.out
M src/test/regress/sql/hash_func.sql
Fix float4/float8 hash functions to produce uniform results for NaNs.
commit : be2beadaff1ad87792efa58273698898d84afd4e
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 2 Sep 2021 17:24:42 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 2 Sep 2021 17:24:42 -0400
The IEEE 754 standard allows a wide variety of bit patterns for NaNs,
of which at least two ("NaN" and "-NaN") are pretty easy to produce
from SQL on most machines. This is problematic because our btree
comparison functions deem all NaNs to be equal, but our float hash
functions know nothing about NaNs and will happily produce varying
hash codes for them. That causes unexpected results from queries
that hash a column containing different NaN values. It could also
produce unexpected lookup failures when using a hash index on a
float column, i.e. "WHERE x = 'NaN'" will not find all the rows
it should.
To fix, special-case NaN in the float hash functions, not too much
unlike the existing special case that forces zero and minus zero
to hash the same. I arranged for the most vanilla sort of NaN
(that coming from the C99 NAN constant) to still have the same
hash code as before, to reduce the risk to existing hash indexes.
I dithered about whether to back-patch this into stable branches,
but ultimately decided to do so. It's a clear improvement for
queries that hash internally. If there is anybody who has -NaN
in a hash index, they'd be well advised to re-index after applying
this patch ... but the misbehavior if they don't will not be much
worse than the misbehavior they had before.
Per bug #17172 from Ma Liangzhu.
Discussion: https://postgr.es/m/17172-7505bea9e04e230f@postgresql.org
M src/backend/access/hash/hashfunc.c
M src/test/regress/expected/hash_func.out
M src/test/regress/sql/hash_func.sql
doc: Replace some uses of "which" by "that" in parallel.sgml
commit : e976cc4a79af63b3f1453a8e2d9eb9bcdeb91616
author : Michael Paquier <michael@paquier.xyz>
date : Thu, 2 Sep 2021 11:36:01 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Thu, 2 Sep 2021 11:36:01 +0900
This makes the documentation more accurate grammatically.
Author: Elena Indrupskaya
Discussion: https://postgr.es/m/1c994b3d-951e-59bb-1ac2-7b9221c0e4cf@postgrespro.ru
Backpatch-through: 9.6
M doc/src/sgml/parallel.sgml
Fix the random test failure in 001_rep_changes.
commit : b51985d8a0e1a7487d17559cbeefc87892c4d916
author : Amit Kapila <akapila@postgresql.org>
date : Wed, 1 Sep 2021 09:16:35 +0530
committer: Amit Kapila <akapila@postgresql.org>
date : Wed, 1 Sep 2021 09:16:35 +0530
The check to test whether the subscription workers were restarting after a
change in the subscription was failing. The reason was that the test was
assuming the walsender started before it reaches the 'streaming' state and
the walsender was exiting due to an error before that. Now, the walsender
was erroring out before reaching the 'streaming' state because it tries to
acquire the slot before the previous walsender has exited.
In passing, improve the die messages so that it is easier to investigate
the failures in the future if any.
Reported-by: Michael Paquier, as per buildfarm
Author: Ajin Cherian
Reviewed-by: Masahiko Sawada, Amit Kapila
Backpatch-through: 10, where this test was introduced
Discussion: https://postgr.es/m/YRnhFxa9bo73wfpV@paquier.xyz
M src/test/subscription/t/001_rep_changes.pl
In pg_dump, avoid doing per-table queries for RLS policies.
commit : db11b4a3db5f6613c844c1b3a6c203bbceb05532
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 31 Aug 2021 15:04:05 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 31 Aug 2021 15:04:05 -0400
For no particularly good reason, getPolicies() queried pg_policy
separately for each table. We can collect all the policies in
a single query instead, and attach them to the correct TableInfo
objects using findTableByOid() lookups. On the regression
database, this reduces the number of queries substantially, and
provides a visible savings even when running against a local
server.
Per complaint from Hubert Depesz Lubaczewski. Since this is such
a simple fix and can have a visible performance benefit, back-patch
to all supported branches.
Discussion: https://postgr.es/m/20210826084430.GA26282@depesz.com
M src/bin/pg_dump/pg_dump.c
Cache the results of format_type() queries in pg_dump.
commit : 904ce45bfa8862d4ae04065c8033182ad812b009
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 31 Aug 2021 13:53:33 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 31 Aug 2021 13:53:33 -0400
There's long been a "TODO: there might be some value in caching
the results" annotation on pg_dump's getFormattedTypeName function;
but we hadn't gotten around to checking what it was costing us to
repetitively look up type names. It turns out that when dumping the
current regression database, about 10% of the total number of queries
issued are duplicative format_type() queries. However, Hubert Depesz
Lubaczewski reported a not-unusual case where these account for over
half of the queries issued by pg_dump. Individually these queries
aren't expensive, but when network lag is a factor, they add up to a
problem. We can very easily add some caching to getFormattedTypeName
to solve it.
Since this is such a simple fix and can have a visible performance
benefit, back-patch to all supported branches.
Discussion: https://postgr.es/m/20210826084430.GA26282@depesz.com
M src/bin/pg_dump/pg_dump.c
M src/bin/pg_dump/pg_dump.h
Rename the role in stats_ext to have regress_ prefix
commit : c8213aa949626deb19f098332eded3586624ff59
author : Tomas Vondra <tomas.vondra@postgresql.org>
date : Tue, 31 Aug 2021 19:21:29 +0200
committer: Tomas Vondra <tomas.vondra@postgresql.org>
date : Tue, 31 Aug 2021 19:21:29 +0200
Commit 5be8ce82e8 added a new role to the stats_ext regression suite,
but the role name did not start with regress_ causing failures when
running with ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS. Fixed by
renaming the role to start with the expected regress_ prefix.
Backpatch-through: 10, same as the new regression test
Discussion: https://postgr.es/m/1F238937-7CC2-4703-A1B1-6DC225B8978A%40enterprisedb.com
M src/test/regress/expected/stats_ext.out
M src/test/regress/sql/stats_ext.sql
Fix lookup error in extended stats ownership check
commit : 1fe1a04af81de4c659ab2a85e578a6bbd4903035
author : Tomas Vondra <tomas.vondra@postgresql.org>
date : Tue, 31 Aug 2021 18:03:05 +0200
committer: Tomas Vondra <tomas.vondra@postgresql.org>
date : Tue, 31 Aug 2021 18:03:05 +0200
When an ownership check on extended statistics object failed, the code
was calling aclcheck_error_type to report the failure, which is clearly
wrong, resulting in cache lookup errors. Fix by calling aclcheck_error.
This issue exists since the introduction of extended statistics, so
backpatch all the way back to PostgreSQL 10. It went unnoticed because
there were no tests triggering the error, so add one.
Reported-by: Mark Dilger
Backpatch-through: 10, where extended stats were introduced
Discussion: https://postgr.es/m/1F238937-7CC2-4703-A1B1-6DC225B8978A%40enterprisedb.com
M src/backend/catalog/objectaddress.c
M src/test/regress/expected/stats_ext.out
M src/test/regress/sql/stats_ext.sql
Report tuple address in data-corruption error message
commit : 6197d7b5383e69383801a4e1c6eed7b7dbc086a3
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 30 Aug 2021 16:29:12 -0400
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 30 Aug 2021 16:29:12 -0400
Most data-corruption reports mention the location of the problem, but
this one failed to. Add it.
Backpatch all the way back. In 12 and older, also assign the
ERRCODE_DATA_CORRUPTED error code as was done in commit fd6ec93bf890 for
13 and later.
Discussion: https://postgr.es/m/202108191637.oqyzrdtnheir@alvherre.pgsql
M src/backend/access/heap/heapam_handler.c
Fix incorrect error code in StartupReplicationOrigin().
commit : 8ba3bad4c362ef71bf6a13ecf4ae5e26066a4876
author : Amit Kapila <akapila@postgresql.org>
date : Mon, 30 Aug 2021 09:26:49 +0530
committer: Amit Kapila <akapila@postgresql.org>
date : Mon, 30 Aug 2021 09:26:49 +0530
ERRCODE_CONFIGURATION_LIMIT_EXCEEDED was used for checksum failure, use
ERRCODE_DATA_CORRUPTED instead.
Reported-by: Tatsuhito Kasahara
Author: Tatsuhito Kasahara
Backpatch-through: 9.6, where it was introduced
Discussion: https://postgr.es/m/CAP0=ZVLHtYffs8SOWcFJWrBGoRzT9QQbk+_aP+E5AHLNXiOorA@mail.gmail.com
M src/backend/replication/logical/origin.c
psql \dP: reference regclass with "pg_catalog." prefix
commit : 9a33ed8fa10354bdb3f12d87ccd9f387bec338d1
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Sat, 28 Aug 2021 11:45:47 -0400
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Sat, 28 Aug 2021 11:45:47 -0400
Strictly speaking this isn't a bug, but since all references to catalog
objects are schema-qualified, we might as well be consistent. The
omission first appeared in commit 1c5d9270e339, so backpatch to 12.
Author: Justin Pryzby <pryzbyj@telsasoft.com>
Discussion: https://postgr.es/m/20210827193151.GN26465@telsasoft.com
M src/bin/psql/describe.c
Fix data loss in wal_level=minimal crash recovery of CREATE TABLESPACE.
commit : b18669f5e652f1d0c1dbf332fd7cd24a38c7ed31
author : Noah Misch <noah@leadboat.com>
date : Fri, 27 Aug 2021 23:33:23 -0700
committer: Noah Misch <noah@leadboat.com>
date : Fri, 27 Aug 2021 23:33:23 -0700
If the system crashed between CREATE TABLESPACE and the next checkpoint,
the result could be some files in the tablespace unexpectedly containing
no rows. Affected files would be those for which the system did not
write WAL; see the wal_skip_threshold documentation. Before v13, a
different set of conditions governed the writing of WAL; see v12's
<sect2 id="populate-pitr">. (The v12 conditions were broader in some
ways and narrower in others.) Users may want to audit non-default
tablespaces for unexpected short files. The bug could have truncated an
index without affecting the associated table, and reindexing the index
would fix that particular problem.
This fixes the bug by making create_tablespace_directories() more like
TablespaceCreateDbspace(). create_tablespace_directories() was
recursively removing tablespace contents, reasoning that WAL redo would
recreate everything removed that way. That assumption holds for other
wal_level values. Under wal_level=minimal, the old approach could
delete files for which no other copy existed. Back-patch to 9.6 (all
supported versions).
Reviewed by Robert Haas and Prabhat Sahu. Reported by Robert Haas.
Discussion: https://postgr.es/m/CA+TgmoaLO9ncuwvr2nN-J4VEP5XyAcy=zKiHxQzBbFRxxGxm0w@mail.gmail.com
M src/backend/commands/tablespace.c
M src/test/recovery/t/018_wal_optimize.pl
Count SP-GiST index scans in pg_stat statistics.
commit : dbb239d518eef791ec6e8f2180e71078e7f89e38
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 27 Aug 2021 19:42:42 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 27 Aug 2021 19:42:42 -0400
Somehow, spgist overlooked the need to call pgstat_count_index_scan().
Hence, pg_stat_all_indexes.idx_scan and equivalent columns never
became nonzero for an SP-GiST index, although the related per-tuple
counters worked fine.
This fix works a bit differently from other index AMs, in that the
counter increment occurs in spgrescan not spggettuple/spggetbitmap.
It looks like this won't make the user-visible semantics noticeably
different, so I won't go to the trouble of introducing an is-this-
the-first-call flag just to make the counter bumps happen in the
same places.
Per bug #17163 from Christian Quest. Back-patch to all supported
versions.
Discussion: https://postgr.es/m/17163-b8c5cc88322a5e92@postgresql.org
M src/backend/access/spgist/spgscan.c
docs: clarify bgw_restart_time documentation
commit : 53597fd6c3cc2042cad9909b962459fa26c26e20
author : Daniel Gustafsson <dgustafsson@postgresql.org>
date : Fri, 27 Aug 2021 22:50:19 +0200
committer: Daniel Gustafsson <dgustafsson@postgresql.org>
date : Fri, 27 Aug 2021 22:50:19 +0200
Author: Dave Cramer <davecramer@gmail.com>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/CADK3HHLZmqAQZ2ByPDQQ9yhGqax36kksq6sDkV0yYzsxw6ipvQ@mail.gmail.com
M doc/src/sgml/bgworker.sgml
Fix broken snapshot handling in parallel workers.
commit : bc062cb938234d8e815d14fecd8d3b2abff3fe88
author : Robert Haas <rhaas@postgresql.org>
date : Wed, 25 Aug 2021 08:32:04 -0400
committer: Robert Haas <rhaas@postgresql.org>
date : Wed, 25 Aug 2021 08:32:04 -0400
Pengchengliu reported an assertion failure in a parallel woker while
performing a parallel scan using an overflowed snapshot. The proximate
cause is that TransactionXmin was set to an incorrect value. The
underlying cause is incorrect snapshot handling in parallel.c.
In particular, InitializeParallelDSM() was unconditionally calling
GetTransactionSnapshot(), because I (rhaas) mistakenly thought that
was always retrieving an existing snapshot whereas, at isolation
levels less than REPEATABLE READ, it's actually taking a new one. So
instead do this only at higher isolation levels where there actually
is a single snapshot for the whole transaction.
By itself, this is not a sufficient fix, because we still need to
guarantee that TransactionXmin gets set properly in the workers. The
easiest way to do that seems to be to install the leader's active
snapshot as the transaction snapshot if the leader did not serialize a
transaction snapshot. This doesn't affect the results of future
GetTrasnactionSnapshot() calls since those have to take a new snapshot
anyway; what we care about is the side effect of setting TransactionXmin.
Report by Pengchengliu. Patch by Greg Nancarrow, except for some comment
text which I supplied.
Discussion: https://postgr.es/m/002f01d748ac$eaa781a0$bff684e0$@tju.edu.cn
M src/backend/access/transam/parallel.c
Fix toast rewrites in logical decoding.
commit : 794025eff0db181e463cb92d4b1aedd0879c88a6
author : Amit Kapila <akapila@postgresql.org>
date : Wed, 25 Aug 2021 09:23:27 +0530
committer: Amit Kapila <akapila@postgresql.org>
date : Wed, 25 Aug 2021 09:23:27 +0530
Commit 325f2ec555 introduced pg_class.relwrite to skip operations on
tables created as part of a heap rewrite during DDL. It links such
transient heaps to the original relation OID via this new field in
pg_class but forgot to do anything about toast tables. So, logical
decoding was not able to skip operations on internally created toast
tables. This leads to an error when we tried to decode the WAL for the
next operation for which it appeared that there is a toast data where
actually it didn't have any toast data.
To fix this, we set pg_class.relwrite for internally created toast tables
as well which allowed skipping operations on them during logical decoding.
Author: Bertrand Drouvot
Reviewed-by: David Zhang, Amit Kapila
Backpatch-through: 11, where it was introduced
Discussion: https://postgr.es/m/b5146fb1-ad9e-7d6e-f980-98ed68744a7c@amazon.com
M contrib/test_decoding/expected/toast.out
M contrib/test_decoding/sql/toast.sql
M src/backend/catalog/toasting.c
M src/backend/commands/cluster.c
M src/backend/commands/tablecmds.c
M src/include/catalog/toasting.h
M src/include/commands/tablecmds.h
Avoid using ambiguous word "positive" in error message.
commit : 7d9026cbfd340a4de6c96ff7c728e53c3791e1cc
author : Fujii Masao <fujii@postgresql.org>
date : Wed, 25 Aug 2021 11:46:25 +0900
committer: Fujii Masao <fujii@postgresql.org>
date : Wed, 25 Aug 2021 11:46:25 +0900
There are two identical error messages about valid value of modulus for
hash partition, in PostgreSQL source code. Commit 0e1275fb07 improved
only one of them so that ambiguous word "positive" was avoided there,
and forgot to improve the other. This commit improves the other.
Which would reduce translator burden.
Back-pach to v11 where the error message exists.
Author: Kyotaro Horiguchi
Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/20210819.170315.1413060634876301811.horikyota.ntt@gmail.com
M src/backend/parser/parse_utilcmd.c
M src/test/regress/expected/alter_table.out
M src/test/regress/expected/create_table.out
Improve error message about valid value for distance in phrase operator.
commit : 81fa1bce2770ee9c9ffe92baa7c5abae70939c59
author : Fujii Masao <fujii@postgresql.org>
date : Wed, 25 Aug 2021 11:43:56 +0900
committer: Fujii Masao <fujii@postgresql.org>
date : Wed, 25 Aug 2021 11:43:56 +0900
The distance in phrase operator must be an integer value between zero
and MAXENTRYPOS inclusive. But previously the error message about
its valid value included the information about its upper limit
but not lower limit (i.e., zero). This commit improves the error message
so that it also includes the information about its lower limit.
Back-patch to v9.6 where full-text phrase search was supported.
Author: Kyotaro Horiguchi
Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/20210819.170315.1413060634876301811.horikyota.ntt@gmail.com
M src/backend/utils/adt/tsquery.c
Fix regexp misbehavior with capturing parens inside "{0}".
commit : 071146184a59de4ac20f5508903fdbc21f963b6c
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 24 Aug 2021 16:37:27 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 24 Aug 2021 16:37:27 -0400
Regexps like "(.){0}...\1" drew an "invalid backreference number".
That's not unreasonable on its face, since the capture group will
never be matched if it's iterated zero times. However, other engines
such as Perl's don't complain about this, nor do we throw an error for
related cases such as "(.)|\1", even though that backref can never
succeed either. Also, if the zero-iterations case happens at runtime
rather than compile time --- say, "(x)*...\1" when there's no "x" to
be found --- that's not an error, we just deem the backref to not
match. Making this even less defensible, no error was thrown for
nested cases such as "((.)){0}...\2"; and to add insult to injury,
those cases could result in assertion failures instead. (It seems
that nothing especially bad happened in non-assert builds, though.)
Let's just fix it so that no error is thrown and instead the backref
is deemed to never match, so that compile-time detection of no
iterations behaves the same as run-time detection.
Per report from Mark Dilger. This appears to be an aboriginal error
in Spencer's library, so back-patch to all supported versions.
Pre-v14, it turns out to also be necessary to back-patch one aspect of
commits cb76fbd7e/00116dee5, namely to create capture-node subREs with
the begin/end states of their subexpressions, not the current lp/rp
of the outer parseqatom invocation. Otherwise delsub complains that
we're trying to disconnect a state from itself. This is a bit scary
but code examination shows that it's safe: in the pre-v14 code, if we
want to wrap iteration around the subexpression, the first thing we do
is overwrite the atom's begin/end fields with new states. So the
bogus values didn't survive long enough to be used for anything, except
if no iteration is required, in which case it doesn't matter.
Discussion: https://postgr.es/m/A099E4A8-4377-4C64-A98C-3DEDDC075502@enterprisedb.com
M src/backend/regex/regcomp.c
M src/test/regress/expected/regex.out
M src/test/regress/sql/regex.sql
Prevent regexp back-refs from sometimes matching when they shouldn't.
commit : 9a327179c824712b35eda01a1edef33ad674b188
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 23 Aug 2021 17:41:07 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Mon, 23 Aug 2021 17:41:07 -0400
The recursion in cdissect() was careless about clearing match data
for capturing parentheses after rejecting a partial match. This
could allow a later back-reference to succeed when by rights it
should fail for lack of a defined referent.
To fix, think a little more rigorously about what the contract
between different levels of cdissect's recursion needs to be.
With the right spec, we can fix this using fewer rather than more
resets of the match data; the key decision being that a failed
sub-match is now explicitly responsible for clearing any matches
it may have set.
There are enough other cross-checks and optimizations in the code
that it's not especially easy to exhibit this problem; usually, the
match will fail as-expected. Plus, regexps that are even potentially
vulnerable are most likely user errors, since there's just not much
point in writing a back-ref that doesn't always have a referent.
These facts perhaps explain why the issue hasn't been detected,
even though it's almost certainly a couple of decades old.
Discussion: https://postgr.es/m/151435.1629733387@sss.pgh.pa.us
M src/backend/regex/regexec.c
M src/test/regress/expected/regex.out
M src/test/regress/sql/regex.sql
Avoid creating archive status ".ready" files too early
commit : ad1231171f5d4319024a2f48c52d267bbee43503
author : Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 23 Aug 2021 15:50:35 -0400
committer: Alvaro Herrera <alvherre@alvh.no-ip.org>
date : Mon, 23 Aug 2021 15:50:35 -0400
WAL records may span multiple segments, but XLogWrite() does not
wait for the entire record to be written out to disk before
creating archive status files. Instead, as soon as the last WAL page of
the segment is written, the archive status file is created, and the
archiver may process it. If PostgreSQL crashes before it is able to
write and flush the rest of the record (in the next WAL segment), the
wrong version of the first segment file lingers in the archive, which
causes operations such as point-in-time restores to fail.
To fix this, keep track of records that span across segments and ensure
that segments are only marked ready-for-archival once such records have
been completely written to disk.
This has always been wrong, so backpatch all the way back.
Author: Nathan Bossart <bossartn@amazon.com>
Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Reviewed-by: Ryo Matsumura <matsumura.ryo@fujitsu.com>
Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru>
Discussion: https://postgr.es/m/CBDDFA01-6E40-46BB-9F98-9340F4379505@amazon.com
M src/backend/access/transam/xlog.c
M src/backend/postmaster/walwriter.c
M src/include/access/xlog.h
M src/include/access/xlogdefs.h
Fix backup manifests to generate correct WAL-Ranges across timelines
commit : 29f94232517bb5c5f22d1deebecfe2160711150f
author : Michael Paquier <michael@paquier.xyz>
date : Mon, 23 Aug 2021 11:09:57 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Mon, 23 Aug 2021 11:09:57 +0900
In a backup manifest, WAL-Ranges stores the range of WAL that is
required for the backup to be valid. pg_verifybackup would then
internally use pg_waldump for the checks based on this data.
When the timeline where the backup started was more than 1 with a
history file looked at for the manifest data generation, the calculation
of the WAL range for the first timeline to check was incorrect. The
previous logic used as start LSN the start position of the first
timeline, but it needs to use the start LSN of the backup. This would
cause failures with pg_verifybackup, or any tools making use of the
backup manifests.
This commit adds a test based on a logic using a self-promoted node,
making it rather cheap.
Author: Kyotaro Horiguchi
Discussion: https://postgr.es/m/20210818.143031.1867083699202617521.horikyota.ntt@gmail.com
Backpatch-through: 13
M src/backend/replication/backup_manifest.c
M src/bin/pg_verifybackup/t/007_wal.pl
Fix performance bug in regexp's citerdissect/creviterdissect.
commit : b30f7f399ea87f0621dfe900547cda401f8da762
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 20 Aug 2021 14:19:04 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 20 Aug 2021 14:19:04 -0400
After detecting a sub-match "dissect" failure (i.e., a backref match
failure) in the i'th sub-match of an iteration node, we should proceed
by adjusting the attempted length of the i'th submatch. As coded,
though, these functions changed the attempted length of the *last*
sub-match, and only after exhausting all possibilities for that would
they back up to adjust the next-to-last sub-match, and then the
second-from-last, etc; all of which is wasted effort, since only
changing the start or length of the i'th sub-match can possibly make
it succeed. This oversight creates the possibility for exponentially
bad performance. Fortunately the problem is masked in most cases by
optimizations or constraints applied elsewhere; which explains why
we'd not noticed it before. But it is possible to reach the problem
with fairly simple, if contrived, regexps.
Oversight in my commit 173e29aa5. That's pretty ancient now,
so back-patch to all supported branches.
Discussion: https://postgr.es/m/1808998.1629412269@sss.pgh.pa.us
M src/backend/regex/regexec.c
Avoid trying to lock OLD/NEW in a rule with FOR UPDATE.
commit : 7fa367d96bb1ae25f422d6c2a78424f0f9227b5a
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 19 Aug 2021 12:12:35 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Thu, 19 Aug 2021 12:12:35 -0400
transformLockingClause neglected to exclude the pseudo-RTEs for
OLD/NEW when processing a rule's query. This led to odd errors
or even crashes later on. This bug is very ancient, but it's
not terribly surprising that nobody noticed, since the use-case
for SELECT FOR UPDATE in a non-view rule is somewhere between
thin and non-existent. Still, crashing is not OK.
Per bug #17151 from Zhiyong Wu. Thanks to Masahiko Sawada
for analysis of the problem.
Discussion: https://postgr.es/m/17151-c03a3e6e4ec9aadb@postgresql.org
M src/backend/parser/analyze.c
M src/include/nodes/parsenodes.h
M src/test/regress/expected/rules.out
M src/test/regress/sql/rules.sql
Fix check_agg_arguments' examination of aggregate FILTER clauses.
commit : ecd4dd9f1df00ef9e872a4d13bbbe3f3d8d3f966
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 18 Aug 2021 18:12:51 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Wed, 18 Aug 2021 18:12:51 -0400
Recursion into the FILTER clause was mis-implemented, such that a
relevant Var or Aggref at the very top of the FILTER clause would
be ignored. (Of course, that'd have to be a plain boolean Var or
boolean-returning aggregate.) The consequence would be
mis-identification of the correct semantic level of the aggregate,
which could lead to not-per-spec query behavior. If the FILTER
expression is an aggregate, this could also lead to failure to issue
an expected "aggregate function calls cannot be nested" error, which
would likely result in a core dump later on, since the planner and
executor aren't expecting such cases to appear.
The root cause is that commit b560ec1b0 blindly copied some code
that assumed it's recursing into a List, and thus didn't examine the
top-level node. To forestall questions about why this call doesn't
look like the others, as well as possible future copy-and-paste
mistakes, let's change all three check_agg_arguments_walker calls in
check_agg_arguments, even though only the one for the filter clause
is really broken.
Per bug #17152 from Zhiyong Wu. This has been wrong since we
implemented FILTER, so back-patch to all supported versions.
(Testing suggests that pre-v11 branches manage to avoid crashing
in the bad-Aggref case, thanks to "redundant" checks in ExecInitAgg.
But I'm not sure how thorough that protection is, and anyway the
wrong-behavior issue remains, so fix 9.6 and 10 too.)
Discussion: https://postgr.es/m/17152-c7f906cc1a88e61b@postgresql.org
M src/backend/parser/parse_agg.c
M src/test/regress/expected/aggregates.out
M src/test/regress/sql/aggregates.sql
Prevent ALTER TYPE/DOMAIN/OPERATOR from changing extension membership.
commit : 7b01246e1de94a436c64f1d396674b2dc44dab70
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 17 Aug 2021 14:29:22 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 17 Aug 2021 14:29:22 -0400
If recordDependencyOnCurrentExtension is invoked on a pre-existing,
free-standing object during an extension update script, that object
will become owned by the extension. In our current code this is
possible in three cases:
* Replacing a "shell" type or operator.
* CREATE OR REPLACE overwriting an existing object.
* ALTER TYPE SET, ALTER DOMAIN SET, and ALTER OPERATOR SET.
The first of these cases is intentional behavior, as noted by the
existing comments for GenerateTypeDependencies. It seems like
appropriate behavior for CREATE OR REPLACE too; at least, the obvious
alternatives are not better. However, the fact that it happens during
ALTER is an artifact of trying to share code (GenerateTypeDependencies
and makeOperatorDependencies) between the CREATE and ALTER cases.
Since an extension script would be unlikely to ALTER an object that
didn't already belong to the extension, this behavior is not very
troubling for the direct target object ... but ALTER TYPE SET will
recurse to dependent domains, and it is very uncool for those to
become owned by the extension if they were not already.
Let's fix this by redefining the ALTER cases to never change extension
membership, full stop. We could minimize the behavioral change by
only changing the behavior when ALTER TYPE SET is recursing to a
domain, but that would complicate the code and it does not seem like
a better definition.
Per bug #17144 from Alex Kozhemyakin. Back-patch to v13 where ALTER
TYPE SET was added. (The other cases are older, but since they only
affect the directly-named object, there's not enough of a problem to
justify changing the behavior further back.)
Discussion: https://postgr.es/m/17144-e67d7a8f049de9af@postgresql.org
M src/backend/catalog/pg_depend.c
M src/backend/catalog/pg_operator.c
M src/backend/catalog/pg_type.c
M src/backend/commands/operatorcmds.c
M src/backend/commands/typecmds.c
M src/include/catalog/pg_operator.h
M src/include/catalog/pg_type.h
Set type identifier on BIO
commit : e15f32f0edf6e22669e309bd58fbd72b8aa43fd3
author : Daniel Gustafsson <dgustafsson@postgresql.org>
date : Tue, 17 Aug 2021 14:27:37 +0200
committer: Daniel Gustafsson <dgustafsson@postgresql.org>
date : Tue, 17 Aug 2021 14:27:37 +0200
In OpenSSL there are two types of BIO's (I/O abstractions):
source/sink and filters. A source/sink BIO is a source and/or
sink of data, ie one acting on a socket or a file. A filter
BIO takes a stream of input from another BIO and transforms it.
In order for BIO_find_type() to be able to traverse the chain
of BIO's and correctly find all BIO's of a certain type they
shall have the type bit set accordingly, source/sink BIO's
(what PostgreSQL implements) use BIO_TYPE_SOURCE_SINK and
filter BIO's use BIO_TYPE_FILTER. In addition to these, file
descriptor based BIO's should have the descriptor bit set,
BIO_TYPE_DESCRIPTOR.
The PostgreSQL implementation didn't set the type bits, which
went unnoticed for a long time as it's only really relevant
for code auditing the OpenSSL installation, or doing similar
tasks. It is required by the API though, so this fixes it.
Backpatch through 9.6 as this has been wrong for a long time.
Author: Itamar Gafni
Discussion: https://postgr.es/m/SN6PR06MB39665EC10C34BB20956AE4578AF39@SN6PR06MB3966.namprd06.prod.outlook.com
Backpatch-through: 9.6
M src/backend/libpq/be-secure-openssl.c
M src/interfaces/libpq/fe-secure-openssl.c
doc: \123 and \x12 escapes in COPY are in database encoding.
commit : c9e75c21d8c96c9385fdb3693d35628a2a625630
author : Heikki Linnakangas <heikki.linnakangas@iki.fi>
date : Tue, 17 Aug 2021 10:00:06 +0300
committer: Heikki Linnakangas <heikki.linnakangas@iki.fi>
date : Tue, 17 Aug 2021 10:00:06 +0300
The backslash sequences, including \123 and \x12 escapes, are interpreted
after encoding conversion. The docs failed to mention that.
Backpatch to all supported versions.
Reported-by: Andreas Grob
Discussion: https://www.postgresql.org/message-id/17142-9181542ca1df75ab%40postgresql.org
M doc/src/sgml/ref/copy.sgml
Refresh apply delay on reload of recovery_min_apply_delay at recovery
commit : 7f0873f328efe057090071805be3f1ade48ecc8a
author : Michael Paquier <michael@paquier.xyz>
date : Mon, 16 Aug 2021 12:11:53 +0900
committer: Michael Paquier <michael@paquier.xyz>
date : Mon, 16 Aug 2021 12:11:53 +0900
This commit ensures that the wait interval in the replay delay loop
waiting for an amount of time defined by recovery_min_apply_delay is
correctly handled on reload, recalculating the delay if this GUC value
is updated, based on the timestamp of the commit record being replayed.
The previous behavior would be problematic for example with replay
still waiting even if the delay got reduced or just cancelled. If the
apply delay was increased to a larger value, the wait would have just
respected the old value set, finishing earlier.
Author: Soumyadeep Chakraborty, Ashwin Agrawal
Reviewed-by: Kyotaro Horiguchi, Michael Paquier
Discussion: https://postgr.es/m/CAE-ML+93zfr-HLN8OuxF0BjpWJ17O5dv1eMvSE5jsj9jpnAXZA@mail.gmail.com
Backpatch-through: 9.6
M src/backend/access/transam/xlog.c
Add RISC-V spinlock support in s_lock.h.
commit : 48695decc27159275a10cf9e1f31bae51f311305
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 13 Aug 2021 13:58:47 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Fri, 13 Aug 2021 13:58:47 -0400
Like the ARM case, just use gcc's __sync_lock_test_and_set();
that will compile into AMOSWAP.W.AQ which does what we need.
At some point it might be worth doing some work on atomic ops
for RISC-V, but this should be enough for a creditable port.
Back-patch to all supported branches, just in case somebody
wants to try them on RISC-V.
Marek Szuba
Discussion: https://postgr.es/m/dea97b6d-f55f-1f6d-9109-504aa7dfa421@gentoo.org
M src/include/storage/s_lock.h
Fix incorrect hash table resizing code in simplehash.h
commit : 4873da79da4bbd54f580ca6e5b2f5c46ae6e4bc6
author : David Rowley <drowley@postgresql.org>
date : Fri, 13 Aug 2021 16:42:35 +1200
committer: David Rowley <drowley@postgresql.org>
date : Fri, 13 Aug 2021 16:42:35 +1200
This fixes a bug in simplehash.h which caused an incorrect size mask to be
used when the hash table grew to SH_MAX_SIZE (2^32). The code was
incorrectly setting the size mask to 0 when the hash tables reached the
maximum possible number of buckets. This would result always trying to
use the 0th bucket causing an infinite loop of trying to grow the hash
table due to there being too many collisions.
Seemingly it's not that common for simplehash tables to ever grow this big
as this bug dates back to v10 and nobody seems to have noticed it before.
However, probably the most likely place that people would notice it would
be doing a large in-memory Hash Aggregate with something close to at least
2^31 groups.
After this fix, the code now works correctly with up to within 98% of 2^32
groups and will fail with the following error when trying to insert any
more items into the hash table:
ERROR: hash table size exceeded
However, the work_mem (or hash_mem_multiplier in newer versions) settings
will generally cause Hash Aggregates to spill to disk long before reaching
that many groups. The minimal test case I did took a work_mem setting of
over 192GB to hit the bug.
simplehash hash tables are used in a few other places such as Bitmap Index
Scans, however, again the size that the hash table can become there is
also limited to work_mem and it would take a relation of around 16TB
(2^31) pages and a very large work_mem setting to hit this. With smaller
work_mem values the table would become lossy and never grow large enough
to hit the problem.
Author: Yura Sokolov
Reviewed-by: David Rowley, Ranier Vilela
Discussion: https://postgr.es/m/b1f7f32737c3438136f64b26f4852b96@postgrespro.ru
Backpatch-through: 10, where simplehash.h was added
M src/include/lib/simplehash.h
Make EXEC_BACKEND more convenient on macOS.
commit : 2c6275423535f86d0d4bc2a4651852d05b3d60c1
author : Thomas Munro <tmunro@postgresql.org>
date : Fri, 13 Aug 2021 10:38:22 +1200
committer: Thomas Munro <tmunro@postgresql.org>
date : Fri, 13 Aug 2021 10:38:22 +1200
It's hard to disable ASLR on current macOS releases, for testing with
-DEXEC_BACKEND. You could already set the environment variable
PG_SHMEM_ADDR to something not likely to collide with mappings created
earlier in process startup. Let's also provide a default value that
works on current releases and architectures, for developer convenience.
As noted in the pre-existing comment, this is a horrible hack, but
-DEXEC_BACKEND is only used by Unix-based PostgreSQL developers for
testing some otherwise Windows-only code paths, so it seems excusable.
Back-patch to all supported branches.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/20210806032944.m4tz7j2w47mant26%40alap3.anarazel.de
M src/backend/port/sysv_shmem.c
Fix failure of btree_gin indexscans with "char" type and </<= operators.
commit : 7ba487cf90071717c9c352dc074a9140860eba55
author : Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 10 Aug 2021 18:10:30 -0400
committer: Tom Lane <tgl@sss.pgh.pa.us>
date : Tue, 10 Aug 2021 18:10:30 -0400
As a result of confusion about whether the "char" type is signed or
unsigned, scans for index searches like "col < 'x'" or "col <= 'x'"
would start at the middle of the index not the left end, thus missing
many or all of the entries they should find. Fortunately, this
is not a symptom of index corruption. It's only the search logic
that is broken, and we can fix it without unpleasant side-effects.
Per report from Jason Kim. This has been wrong since btree_gin's
beginning, so back-patch to all supported branches.
Discussion: https://postgr.es/m/20210810001649.htnltbh7c63re42p@jasonk.me
M contrib/btree_gin/btree_gin.c
M contrib/btree_gin/expected/char.out