PostgreSQL 17.1 commit log

Stamp 17.1.

commit   : 91f20bc2f7e4fcf5de5c65a6cb1190e0afa91c0b    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 17:42:37 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 17:42:37 -0500    

Click here for diff

M configure
M configure.ac
M meson.build

Last-minute updates for release notes.

commit   : 052aa02971f2dd476c14689033727a0b8563d5d2    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 17:40:13 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 17:40:13 -0500    

Click here for diff

Security: CVE-2024-10976, CVE-2024-10977, CVE-2024-10978, CVE-2024-10979  

M doc/src/sgml/release-17.sgml

Parallel workers use AuthenticatedUserId for connection privilege checks.

commit   : f4f5d27d87247da1ec7e5a6e7990a22ffba9f63a    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 17:05:53 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 17:05:53 -0500    

Click here for diff

Commit 5a2fed911 had an unexpected side-effect: the parallel worker  
launched for the new test case would fail if it couldn't use a  
superuser-reserved connection slot.  The reason that test failed  
while all our pre-existing ones worked is that the connection  
privilege tests in InitPostgres had been based on the superuserness  
of the leader's AuthenticatedUserId, but after the rearrangements  
of 5a2fed911 we were testing the superuserness of CurrentUserId,  
which the new test case deliberately made to be a non-superuser.  
  
This all seems very accidental and probably not the behavior we really  
want, but a security patch is no time to be redesigning things.  
Pending some discussion about desirable semantics, hack it so that  
InitPostgres continues to pay attention to the superuserness of  
AuthenticatedUserId when starting a parallel worker.  
  
Nathan Bossart and Tom Lane, per buildfarm member sawshark.  
  
Security: CVE-2024-10978  

M src/backend/utils/init/postinit.c

Fix cross-version upgrade tests.

commit   : 8d19f3fea003b1f744516b84cbdb0097ae7b2912    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 13:57:21 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 13:57:21 -0500    

Click here for diff

TestUpgradeXversion knows how to make the main regression database's  
references to pg_regress.so be version-independent.  But it doesn't  
do that for plperl's database, so that the C function added by  
commit b7e3a52a8 is causing cross-version upgrade test failures.  
Path of least resistance is to just drop the function at the end  
of the new test.  
  
In <= v14, also take the opportunity to clean up the generated  
test files.  
  
Security: CVE-2024-10979  

M src/pl/plperl/expected/plperl_env.out
M src/pl/plperl/sql/plperl_env.sql

Avoid bizarre meson behavior with backslashes in command arguments.

commit   : 4cd4f3b97492c1b38115d0563a2e55b136eb542a    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 12:20:08 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 12:20:08 -0500    

Click here for diff

meson makes the backslashes in text2macro.pl's --strip argument  
into forward slashes, effectively disabling comment stripping.  
That hasn't caused us issues before, but it breaks the test case  
for b7e3a52a8.  We don't really need the pattern to be adjustable,  
so just hard-wire it into the script instead.  
  
Context: https://github.com/mesonbuild/meson/issues/1564  
Security: CVE-2024-10979  

M src/pl/plperl/GNUmakefile
M src/pl/plperl/meson.build
M src/pl/plperl/text2macro.pl

Fix improper interactions between session_authorization and role.

commit   : cd82afdda5e9d3269706a142e9093ba83f484185    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 10:29:54 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 11 Nov 2024 10:29:54 -0500    

Click here for diff

The SQL spec mandates that SET SESSION AUTHORIZATION implies  
SET ROLE NONE.  We tried to implement that within the lowest-level  
functions that manipulate these settings, but that was a bad idea.  
In particular, guc.c assumes that it doesn't matter in what order  
it applies GUC variable updates, but that was not the case for these  
two variables.  This problem, compounded by some hackish attempts to  
work around it, led to some security-grade issues:  
  
* Rolling back a transaction that had done SET SESSION AUTHORIZATION  
would revert to SET ROLE NONE, even if that had not been the previous  
state, so that the effective user ID might now be different from what  
it had been.  
  
* The same for SET SESSION AUTHORIZATION in a function SET clause.  
  
* If a parallel worker inspected current_setting('role'), it saw  
"none" even when it should see something else.  
  
Also, although the parallel worker startup code intended to cope  
with the current role's pg_authid row having disappeared, its  
implementation of that was incomplete so it would still fail.  
  
Fix by fully separating the miscinit.c functions that assign  
session_authorization from those that assign role.  To implement the  
spec's requirement, teach set_config_option itself to perform "SET  
ROLE NONE" when it sets session_authorization.  (This is undoubtedly  
ugly, but the alternatives seem worse.  In particular, there's no way  
to do it within assign_session_authorization without incompatible  
changes in the API for GUC assign hooks.)  Also, improve  
ParallelWorkerMain to directly set all the relevant user-ID variables  
instead of relying on some of them to get set indirectly.  That  
allows us to survive not finding the pg_authid row during worker  
startup.  
  
In v16 and earlier, this includes back-patching 9987a7bf3 which  
fixed a violation of GUC coding rules: SetSessionAuthorization  
is not an appropriate place to be throwing errors from.  
  
Security: CVE-2024-10978  

M src/backend/access/transam/parallel.c
M src/backend/commands/variable.c
M src/backend/utils/init/miscinit.c
M src/backend/utils/misc/guc.c
M src/include/miscadmin.h
M src/test/regress/expected/privileges.out
M src/test/regress/sql/privileges.sql

Ensure cached plans are correctly marked as dependent on role.

commit   : edcda9bb4c4500b75bb4a16c7c59834398ca2906    
  
author   : Nathan Bossart <[email protected]>    
date     : Mon, 11 Nov 2024 09:00:00 -0600    
  
committer: Nathan Bossart <[email protected]>    
date     : Mon, 11 Nov 2024 09:00:00 -0600    

Click here for diff

If a CTE, subquery, sublink, security invoker view, or coercion  
projection references a table with row-level security policies, we  
neglected to mark the plan as potentially dependent on which role  
is executing it.  This could lead to later executions in the same  
session returning or hiding rows that should have been hidden or  
returned instead.  
  
Reported-by: Wolfgang Walther  
Reviewed-by: Noah Misch  
Security: CVE-2024-10976  
Backpatch-through: 12  

M src/backend/executor/functions.c
M src/backend/rewrite/rewriteHandler.c
M src/test/regress/expected/rowsecurity.out
M src/test/regress/sql/rowsecurity.sql
M src/tools/pgindent/typedefs.list

Block environment variable mutations from trusted PL/Perl.

commit   : 3ebcfa54db3309651d8f1d3be6451a8449f6c6ec    
  
author   : Noah Misch <[email protected]>    
date     : Mon, 11 Nov 2024 06:23:43 -0800    
  
committer: Noah Misch <[email protected]>    
date     : Mon, 11 Nov 2024 06:23:43 -0800    

Click here for diff

Many process environment variables (e.g. PATH), bypass the containment  
expected of a trusted PL.  Hence, trusted PLs must not offer features  
that achieve setenv().  Otherwise, an attacker having USAGE privilege on  
the language often can achieve arbitrary code execution, even if the  
attacker lacks a database server operating system user.  
  
To fix PL/Perl, replace trusted PL/Perl %ENV with a tied hash that just  
replaces each modification attempt with a warning.  Sites that reach  
these warnings should evaluate the application-specific implications of  
proceeding without the environment modification:  
  
  Can the application reasonably proceed without the modification?  
  
    If no, switch to plperlu or another approach.  
  
    If yes, the application should change the code to stop attempting  
    environment modifications.  If that's too difficult, add "untie  
    %main::ENV" in any code executed before the warning.  For example,  
    one might add it to the start of the affected function or even to  
    the plperl.on_plperl_init setting.  
  
In passing, link to Perl's guidance about the Perl features behind the  
security posture of PL/Perl.  
  
Back-patch to v12 (all supported versions).  
  
Andrew Dunstan and Noah Misch  
  
Security: CVE-2024-10979  

M doc/src/sgml/plperl.sgml
M src/pl/plperl/GNUmakefile
A src/pl/plperl/expected/plperl_env.out
M src/pl/plperl/meson.build
M src/pl/plperl/plc_trusted.pl
A src/pl/plperl/sql/plperl_env.sql
M src/test/regress/regress.c

Translation updates

commit   : 6bf5bf11c3944886225778790bf35aaff3b12a34    
  
author   : Peter Eisentraut <[email protected]>    
date     : Mon, 11 Nov 2024 13:52:24 +0100    
  
committer: Peter Eisentraut <[email protected]>    
date     : Mon, 11 Nov 2024 13:52:24 +0100    

Click here for diff

Source-Git-URL: https://git.postgresql.org/git/pgtranslation/messages.git  
Source-Git-Hash: 2592030f456910263c8972668576f954fce10595  

M src/backend/po/de.po
M src/backend/po/ja.po
M src/backend/po/ru.po
M src/backend/po/uk.po
M src/bin/initdb/po/de.po
M src/bin/initdb/po/fr.po
M src/bin/initdb/po/ja.po
M src/bin/initdb/po/ka.po
M src/bin/initdb/po/ru.po
M src/bin/initdb/po/uk.po
M src/bin/pg_amcheck/po/ja.po
M src/bin/pg_amcheck/po/uk.po
M src/bin/pg_archivecleanup/po/ja.po
M src/bin/pg_archivecleanup/po/uk.po
M src/bin/pg_basebackup/po/ja.po
M src/bin/pg_basebackup/po/ru.po
M src/bin/pg_basebackup/po/uk.po
M src/bin/pg_checksums/po/uk.po
M src/bin/pg_combinebackup/po/LINGUAS
M src/bin/pg_combinebackup/po/de.po
M src/bin/pg_combinebackup/po/fr.po
M src/bin/pg_combinebackup/po/ja.po
M src/bin/pg_combinebackup/po/ru.po
A src/bin/pg_combinebackup/po/uk.po
M src/bin/pg_config/po/uk.po
M src/bin/pg_controldata/po/uk.po
M src/bin/pg_ctl/po/uk.po
M src/bin/pg_dump/po/ja.po
M src/bin/pg_dump/po/uk.po
M src/bin/pg_resetwal/po/uk.po
M src/bin/pg_rewind/po/ja.po
M src/bin/pg_rewind/po/uk.po
M src/bin/pg_test_fsync/po/uk.po
M src/bin/pg_test_timing/po/uk.po
M src/bin/pg_upgrade/po/de.po
M src/bin/pg_upgrade/po/fr.po
M src/bin/pg_upgrade/po/ja.po
M src/bin/pg_upgrade/po/ka.po
M src/bin/pg_upgrade/po/ru.po
M src/bin/pg_upgrade/po/uk.po
M src/bin/pg_verifybackup/po/ja.po
M src/bin/pg_verifybackup/po/uk.po
M src/bin/pg_waldump/po/uk.po
M src/bin/pg_walsummary/po/LINGUAS
A src/bin/pg_walsummary/po/uk.po
M src/bin/psql/po/ja.po
M src/bin/psql/po/ru.po
M src/bin/psql/po/uk.po
M src/bin/scripts/po/ru.po
M src/bin/scripts/po/uk.po
M src/interfaces/ecpg/ecpglib/po/uk.po
M src/interfaces/ecpg/preproc/po/ja.po
M src/interfaces/ecpg/preproc/po/uk.po
M src/interfaces/libpq/po/fr.po
M src/interfaces/libpq/po/ja.po
M src/interfaces/libpq/po/ru.po
M src/interfaces/libpq/po/uk.po
M src/pl/plperl/po/uk.po
M src/pl/plpgsql/src/po/uk.po
M src/pl/plpython/po/uk.po
M src/pl/tcl/po/uk.po

libpq: Bail out during SSL/GSS negotiation errors

commit   : a5cc4c66719be2ae1eebe92ad97727dc905bbc6d    
  
author   : Michael Paquier <[email protected]>    
date     : Mon, 11 Nov 2024 10:19:56 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Mon, 11 Nov 2024 10:19:56 +0900    

Click here for diff

This commit changes libpq so that errors reported by the backend during  
the protocol negotiation for SSL and GSS are discarded by the client, as  
these may include bytes that could be consumed by the client and write  
arbitrary bytes to a client's terminal.  
  
A failure with the SSL negotiation now leads to an error immediately  
reported, without a retry on any other methods allowed, like a fallback  
to a plaintext connection.  
  
A failure with GSS discards the error message received, and we allow a  
fallback as it may be possible that the error is caused by a connection  
attempt with a pre-11 server, GSS encryption having been introduced in  
v12.  This was a problem only with v17 and newer versions; older  
versions discard the error message already in this case, assuming a  
failure caused by a lack of support for GSS encryption.  
  
Author: Jacob Champion  
Reviewed-by: Peter Eisentraut, Heikki Linnakangas, Michael Paquier  
Security: CVE-2024-10977  
Backpatch-through: 12  

M doc/src/sgml/protocol.sgml
M src/interfaces/libpq/fe-connect.c

Release notes for 17.1, 16.5, 15.9, 14.14, 13.17, 12.21.

commit   : ca19f881b071fa57ee469fbc27b520fbb0c67280    
  
author   : Tom Lane <[email protected]>    
date     : Sun, 10 Nov 2024 13:40:41 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Sun, 10 Nov 2024 13:40:41 -0500    

Click here for diff

M doc/src/sgml/release-17.sgml

Fix sign-compare warnings in pg_iovec.h.

commit   : 0a883a067bd78f0ff0607afb18c4f783ac764504    
  
author   : Nathan Bossart <[email protected]>    
date     : Fri, 8 Nov 2024 16:11:08 -0600    
  
committer: Nathan Bossart <[email protected]>    
date     : Fri, 8 Nov 2024 16:11:08 -0600    

Click here for diff

The code in question (pg_preadv() and pg_pwritev()) has been around  
for a while, but commit 15c9ac3629 moved it to a header file.  If  
third-party code that includes this header file is built with  
-Wsign-compare on a system without preadv() or pwritev(), warnings  
ensue.  This commit fixes said warnings by casting the result of  
pg_pread()/pg_pwrite() to size_t, which should be safe because we  
will have already checked for a negative value.  
  
Author: Wolfgang Walther  
Discussion: https://postgr.es/m/16989737-1aa8-48fd-8dfe-b7ada06509ab%40technowledgy.de  
Backpatch-through: 17  

M src/include/port/pg_iovec.h

First-draft release notes for 17.1.

commit   : 4145ea0910d7bdbf131aa6514ffce8eb92230a5f    
  
author   : Tom Lane <[email protected]>    
date     : Fri, 8 Nov 2024 16:39:34 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Fri, 8 Nov 2024 16:39:34 -0500    

Click here for diff

(We lack a query for identifying broken foreign keys in the first  
changelog item, but the rest of this is in reviewable shape.)  
  
As usual, the release notes for other branches will be made by cutting  
these down, but put them up for community review first.  
  
Also as usual for a .1 release, there are some entries here that  
are not really relevant for v17 because they already appeared in 17.0.  
Those'll be removed later.  

M doc/src/sgml/release-17.sgml

Improve fix for not entering parallel mode when holding interrupts.

commit   : 943b65358e880222a66eeb13b40ad2ef8c172c6a    
  
author   : Tom Lane <[email protected]>    
date     : Fri, 8 Nov 2024 13:42:01 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Fri, 8 Nov 2024 13:42:01 -0500    

Click here for diff

Commit ac04aa84a put the shutoff for this into the planner, which is  
not ideal because it doesn't prevent us from re-using a previously  
made parallel plan.  Revert the planner change and instead put the  
shutoff into InitializeParallelDSM, modeling it on the existing code  
there for recovering from failure to allocate a DSM segment.  
  
However, that code path is mostly untested, and testing a bit harder  
showed there's at least one bug: ExecHashJoinReInitializeDSM is not  
prepared for us to have skipped doing parallel DSM setup.  I also  
thought the Assert in ReinitializeParallelWorkers is pretty  
ill-advised, and replaced it with a silent Min() operation.  
  
The existing test case added by ac04aa84a serves fine to test this  
version of the fix, so no change needed there.  
  
Patch by me, but thanks to Noah Misch for the core idea that we  
could shut off worker creation when !INTERRUPTS_CAN_BE_PROCESSED.  
Back-patch to v12, as ac04aa84a was.  
  
Discussion: https://postgr.es/m/CAC-SaSzHUKT=vZJ8MPxYdC_URPfax+yoA1hKTcF4ROz_Q6z0_Q@mail.gmail.com  

M src/backend/access/transam/parallel.c
M src/backend/executor/nodeHashjoin.c
M src/backend/optimizer/plan/planner.c

Disallow partitionwise join when collations don't match

commit   : a0cdfc8893673d1325f077b79f551aa105f360b8    
  
author   : Amit Langote <[email protected]>    
date     : Fri, 8 Nov 2024 16:29:38 +0900    
  
committer: Amit Langote <[email protected]>    
date     : Fri, 8 Nov 2024 16:29:38 +0900    

Click here for diff

If the collation of any join key column doesn’t match the collation of  
the corresponding partition key, partitionwise joins can yield incorrect  
results. For example, rows that would match under the join key collation  
might be located in different partitions due to the partitioning  
collation. In such cases, a partitionwise join would yield different  
results from a non-partitionwise join, so disallow it in such cases.  
  
Reported-by: Tender Wang <[email protected]>  
Author: Jian He <[email protected]>  
Reviewed-by: Tender Wang <[email protected]>  
Reviewed-by: Junwang Zhao <[email protected]>  
Discussion: https://postgr.es/m/CAHewXNno_HKiQ6PqyLYfuqDtwp7KKHZiH1J7Pqyz0nr+PS2Dwg@mail.gmail.com  
Backpatch-through: 12  

M src/backend/optimizer/util/relnode.c
M src/test/regress/expected/collate.icu.utf8.out
M src/test/regress/sql/collate.icu.utf8.sql

Disallow partitionwise grouping when collations don't match

commit   : b6484ca9535e947a068da01015620dfde0b9b488    
  
author   : Amit Langote <[email protected]>    
date     : Fri, 8 Nov 2024 16:07:13 +0900    
  
committer: Amit Langote <[email protected]>    
date     : Fri, 8 Nov 2024 16:07:13 +0900    

Click here for diff

If the collation of any grouping column doesn’t match the collation of  
the corresponding partition key, partitionwise grouping can yield  
incorrect results. For example, rows that would be grouped under the  
grouping collation may end up in different partitions under the  
partitioning collation. In such cases, full partitionwise grouping  
would produce results that differ from those without partitionwise  
grouping, so disallowed that.  
  
Partial partitionwise aggregation is still allowed, as the Finalize  
step reconciles partition-level aggregates with grouping requirements  
across all partitions, ensuring that the final output remains  
consistent.  
  
This commit also fixes group_by_has_partkey() by ensuring the  
RelabelType node is stripped from grouping expressions when matching  
them to partition key expressions to avoid false mismatches.  
  
Bug: #18568  
Reported-by: Webbo Han <[email protected]>  
Author: Webbo Han <[email protected]>  
Reviewed-by: Tender Wang <[email protected]>  
Reviewed-by: Aleksander Alekseev <[email protected]>  
Reviewed-by: Jian He <[email protected]>  
Discussion: https://postgr.es/m/[email protected]  
Discussion: https://postgr.es/m/[email protected]  
Discussion: https://postgr.es/m/CAHewXNno_HKiQ6PqyLYfuqDtwp7KKHZiH1J7Pqyz0nr+PS2Dwg@mail.gmail.com  
Backpatch-through: 12  

M src/backend/optimizer/plan/planner.c
M src/test/regress/expected/collate.icu.utf8.out
M src/test/regress/sql/collate.icu.utf8.sql

Fix inconsistent RestrictInfo serial numbers

commit   : 78b1c553bb1c1f334d0b7f3365070ba27a2fcc41    
  
author   : Richard Guo <[email protected]>    
date     : Fri, 8 Nov 2024 11:24:26 +0900    
  
committer: Richard Guo <[email protected]>    
date     : Fri, 8 Nov 2024 11:24:26 +0900    

Click here for diff

When we generate multiple clones of the same qual condition to cope  
with outer join identity 3, we need to ensure that all the clones get  
the same serial number.  To achieve this, we reset the  
root->last_rinfo_serial counter each time we produce RestrictInfo(s)  
from the qual list (see deconstruct_distribute_oj_quals).  This  
approach works only if we ensure that we are not changing the qual  
list in any way that'd affect the number of RestrictInfos built from  
it.  
  
However, with b262ad440, an IS NULL qual on a NOT NULL column might  
result in an additional constant-FALSE RestrictInfo.  And different  
versions of the same qual clause can lead to different conclusions  
about whether it can be reduced to constant-FALSE.  This would affect  
the number of RestrictInfos built from the qual list for different  
versions, causing inconsistent RestrictInfo serial numbers across  
multiple clones of the same qual.  This inconsistency can confuse  
users of these serial numbers, such as rebuild_joinclause_attr_needed,  
and lead to planner errors such as "ERROR:  variable not found in  
subplan target lists".  
  
To fix, reset the root->last_rinfo_serial counter after generating the  
additional constant-FALSE RestrictInfo.  
  
Back-patch to v17 where the issue crept in.  In v17, I failed to make  
a test case that would expose this bug, so no test case for v17.  
  
Author: Richard Guo  
Discussion: https://postgr.es/m/CAMbWs4-B6kafn+LmPuh-TYFwFyEm-vVj3Qqv7Yo-69CEv14rRg@mail.gmail.com  

M src/backend/optimizer/plan/initsplan.c
M src/backend/optimizer/util/joininfo.c

doc: Reword ALTER TABLE ATTACH restriction on NO INHERIT constraints

commit   : e2b5693c6cf204d6cd7ea15d52735959c8510ffe    
  
author   : Álvaro Herrera <[email protected]>    
date     : Thu, 7 Nov 2024 14:06:24 +0100    
  
committer: Álvaro Herrera <[email protected]>    
date     : Thu, 7 Nov 2024 14:06:24 +0100    

Click here for diff

The previous wording is easy to read incorrectly; this change makes it  
simpler, less ambiguous, and less prominent.  
  
Backpatch to all live branches.  
  
Reviewed-by: Amit Langote <[email protected]>  
Discussion: https://postgr.es/m/[email protected]  

M doc/src/sgml/ref/alter_table.sgml

Fix lc_collate_is_c() when LC_COLLATE != LC_CTYPE.

commit   : 8148e7124d550770e3a9064b9b45b354d783b949    
  
author   : Jeff Davis <[email protected]>    
date     : Wed, 6 Nov 2024 14:44:35 -0800    
  
committer: Jeff Davis <[email protected]>    
date     : Wed, 6 Nov 2024 14:44:35 -0800    

Click here for diff

An unfortunate typo in commit 2d819a08a1 can cause wrong results when  
the default collation provider is libc, LC_CTYPE=C, and LC_COLLATE is  
a real locale. Users with this combination of settings must REINDEX  
all affected indexes.  
  
The same typo can also cause performance degradation when LC_COLLATE=C  
and LC_CTYPE is a real locale.  
  
Problem does not exist in master (due to refactoring), so fix only in  
version 17.  
  
Reported-by: Drew Callahan  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/utils/adt/pg_locale.c

Monkey-patch LLVM code to fix ARM relocation bug.

commit   : b7467ab71c1982c120bb53811be4341730061212    
  
author   : Thomas Munro <[email protected]>    
date     : Wed, 6 Nov 2024 22:04:44 +1300    
  
committer: Thomas Munro <[email protected]>    
date     : Wed, 6 Nov 2024 22:04:44 +1300    

Click here for diff

Supply a new memory manager for RuntimeDyld, to avoid crashes in  
generated code caused by memory placement that can overflow a 32 bit  
data type.  This is a drop-in replacement for the  
llvm::SectionMemoryManager class in the LLVM library, with Michael  
Smith's proposed fix from  
https://www.github.com/llvm/llvm-project/pull/71968.  
  
We hereby slurp it into our own source tree, after moving into a new  
namespace llvm::backport and making some minor adjustments so that it  
can be compiled with older LLVM versions as far back as 12.  It's harder  
to make it work on even older LLVM versions, but it doesn't seem likely  
that people are really using them so that is not investigated for now.  
  
The problem could also be addressed by switching to JITLink instead of  
RuntimeDyld, and that is the LLVM project's recommended solution as  
the latter is about to be deprecated.  We'll have to do that soon enough  
anyway, and then when the LLVM version support window advances far  
enough in a few years we'll be able to delete this code.  Unfortunately  
that wouldn't be enough for PostgreSQL today: in most relevant versions  
of LLVM, JITLink is missing or incomplete.  
  
Several other projects have already back-ported this fix into their fork  
of LLVM, which is a vote of confidence despite the lack of commit into  
LLVM as of today.  We don't have our own copy of LLVM so we can't do  
exactly what they've done; instead we have a copy of the whole patched  
class so we can pass an instance of it to RuntimeDyld.  
  
The LLVM project hasn't chosen to commit the fix yet, and even if it  
did, it wouldn't be back-ported into the releases of LLVM that most of  
our users care about, so there is not much point in waiting any longer  
for that.  If they make further changes and commit it to LLVM 19 or 20,  
we'll still need this for older versions, but we may want to  
resynchronize our copy and update some comments.  
  
The changes that we've had to make to our copy can be seen by diffing  
our SectionMemoryManager.{h,cpp} files against the ones in the tree of  
the pull request.  Per the LLVM project's license requirements, a copy  
is in SectionMemoryManager.LICENSE.  
  
This should fix the spate of crash reports we've been receiving lately  
from users on large memory ARM systems.  
  
Back-patch to all supported releases.  
  
Co-authored-by: Thomas Munro <[email protected]>  
Co-authored-by: Anthonin Bonnefoy <[email protected]>  
Reviewed-by: Anthonin Bonnefoy <[email protected]>  
Reviewed-by: Daniel Gustafsson <[email protected]> (license aspects)  
Reported-by: Anthonin Bonnefoy <[email protected]>  
Discussion: https://postgr.es/m/CAO6_Xqr63qj%3DSx7HY6ZiiQ6R_JbX%2B-p6sTPwDYwTWZjUmjsYBg%40mail.gmail.com  

M src/backend/jit/llvm/Makefile
A src/backend/jit/llvm/SectionMemoryManager.LICENSE
A src/backend/jit/llvm/SectionMemoryManager.cpp
M src/backend/jit/llvm/llvmjit.c
M src/backend/jit/llvm/llvmjit_wrap.cpp
M src/backend/jit/llvm/meson.build
A src/include/jit/SectionMemoryManager.h
M src/include/jit/llvmjit.h
A src/include/jit/llvmjit_backport.h
M src/tools/pginclude/headerscheck
M src/tools/pgindent/exclude_file_patterns

Clear padding of PgStat_HashKey when handling pgstats entries

commit   : 7f3b41ce48a58f090da94dbcc737483c217ce9c3    
  
author   : Michael Paquier <[email protected]>    
date     : Tue, 5 Nov 2024 09:40:55 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Tue, 5 Nov 2024 09:40:55 +0900    

Click here for diff

PgStat_HashKey is currently initialized in a way that could result in  
random data if the structure has any padding bytes.  The structure  
has no padding bytes currently, fortunately, but it could become a  
problem should the structure change at some point in the future.  
  
The code is changed to use some memset(0) so as any padding would be  
handled properly, as it would be surprising to see random failures in  
the pgstats entry lookups.  PgStat_HashKey is a structure internal to  
pgstats, and an ABI change could be possible in the scope of a bug fix,  
so backpatch down to 15 where this has been introduced.  
  
Author: Bertrand Drouvot  
Reviewed-by: Jelte Fennema-Nio, Michael Paquier  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 15  

M src/backend/utils/activity/pgstat.c
M src/backend/utils/activity/pgstat_shmem.c

Use portable diff options in pg_bsd_indent's regression test.

commit   : 811f8d3f10a448644004954ad784bed4d51b88c7    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 4 Nov 2024 18:08:48 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 4 Nov 2024 18:08:48 -0500    

Click here for diff

We had been using "diff -upd", which evidently works for most people,  
but Solaris's diff doesn't like it.  (We'd not noticed because the  
Solaris buildfarm animals weren't running this test until they were  
upgraded to the latest buildfarm client script.)  Change to "diff -U3"  
which is what pg_regress has used for ages.  
  
Per buildfarm (and off-list discussion with Noah Misch).  
  
Back-patch to v16 where this test was added.  In v16,  
also back-patch the relevant part of 628c1d1f2 so that  
the test script looks about the same in all branches.  

M src/tools/pg_bsd_indent/t/001_pg_bsd_indent.pl

pg_basebackup, pg_receivewal: fix failure to find password in ~/.pgpass.

commit   : e2a912909308041d629f1e62ea84c76d179674b1    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 4 Nov 2024 14:36:04 -0500    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 4 Nov 2024 14:36:04 -0500    

Click here for diff

Sloppy refactoring in commit cca97ce6a caused these programs  
to pass dbname = NULL to libpq if there was no "--dbname" switch  
on the command line, where before "replication" would be passed.  
This didn't break things completely, because the source server doesn't  
care about the dbname specified for a physical replication connection.  
However, it did cause libpq to fail to match a ~/.pgpass entry that  
has "replication" in the dbname field.  Restore the previous behavior  
of passing "replication".  
  
Also, closer inspection shows that if you do specify a dbname  
in the connection string, that is what will be matched to ~/.pgpass,  
not "replication".  This was the pre-existing behavior so we should  
not change it, but the SGML docs were pretty misleading about it.  
Improve that.  
  
Per bug #18685 from Toshi Harada.  Back-patch to v17 where the  
error crept in.  
  
Discussion: https://postgr.es/m/[email protected]  
Discussion: https://postgr.es/m/[email protected]  

M doc/src/sgml/ref/pg_basebackup.sgml
M doc/src/sgml/ref/pg_receivewal.sgml
M src/bin/pg_basebackup/streamutil.c

pg_combinebackup: Error if incremental file exists in full backup.

commit   : e367114429e09696fce01dd6f256e58a2c858734    
  
author   : Robert Haas <[email protected]>    
date     : Mon, 4 Nov 2024 10:11:05 -0500    
  
committer: Robert Haas <[email protected]>    
date     : Mon, 4 Nov 2024 10:11:05 -0500    

Click here for diff

Suppose that you run a command like "pg_combinebackup b1 b2 -o output",  
but both b1 and b2 contain an INCREMENTAL.$something file in a directory  
that is expected to contain relation files. This is an error, but the  
previous code would not detect the problem and instead write a garbage  
full file named $something to the output directory. This commit adds  
code to detect the error and a test case to verify the behavior.  
  
It's difficult to imagine that this will ever happen unless someone  
is intentionally trying to break incremental backup, but per discussion,  
let's consider that the lack of adequate sanity checking in this area is  
a bug and back-patch to v17, where incremental backup was introduced.  
  
Patch by me, reviewed by Bertrand Drouvot and Amul Sul.  
  
Discussion: http://postgr.es/m/CA+TgmoaD7dBYPqe7kMtO0dyto7rd0rUh7joh=JPUSaFszKY6Pg@mail.gmail.com  

M src/bin/pg_combinebackup/meson.build
M src/bin/pg_combinebackup/reconstruct.c
A src/bin/pg_combinebackup/t/009_no_full_file.pl

pg_combinebackup: When reconstructing, avoid double slash in filename.

commit   : 0d635b615d3fc406b0b3df0637831a799561fa77    
  
author   : Robert Haas <[email protected]>    
date     : Mon, 4 Nov 2024 09:55:02 -0500    
  
committer: Robert Haas <[email protected]>    
date     : Mon, 4 Nov 2024 09:55:02 -0500    

Click here for diff

This function is always called with a relative_path that ends in a  
slash, so there's no need to insert a second one. So, don't. Instead,  
add an assertion to verify that nothing gets broken in the future, and  
adjust the comments.  
  
While this is not a critical bug, the duplicate slash is visible in  
error messages, which could create confusion, so back-patch to v17.  
This is also better in that it keeps the code consistent across  
branches.  
  
Patch by me, reviewed by Bertrand Drouvot and Amul Sul.  
  
Discussion: http://postgr.es/m/CA+TgmoaD7dBYPqe7kMtO0dyto7rd0rUh7joh=JPUSaFszKY6Pg@mail.gmail.com  

M src/bin/pg_combinebackup/reconstruct.c

Suppress new "may be used uninitialized" warning.

commit   : 54bc22fbf6ede8aac765acee5ac3a9d50604fb5f    
  
author   : Noah Misch <[email protected]>    
date     : Sat, 2 Nov 2024 19:42:52 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Sat, 2 Nov 2024 19:42:52 -0700    

Click here for diff

Buildfarm member mamba fails to deduce that the function never uses this  
variable without initializing it.  Back-patch to v12, like commit  
b412f402d1e020c5dac94f3bf4a005db69519b99.  

M src/backend/catalog/index.c

Move I/O before the index_update_stats() buffer lock region.

commit   : 0bcb9d07940c790912b0aff781ef605ec22e1a2d    
  
author   : Noah Misch <[email protected]>    
date     : Sat, 2 Nov 2024 09:04:55 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Sat, 2 Nov 2024 09:04:55 -0700    

Click here for diff

Commit a07e03fd8fa7daf4d1356f7cb501ffe784ea6257 enlarged the work done  
here under the pg_class heap buffer lock.  Two preexisting actions are  
best done before holding that lock.  Both RelationGetNumberOfBlocks()  
and visibilitymap_count() do I/O, and the latter might exclusive-lock a  
visibility map buffer.  Moving these reduces contention and risk of  
undetected LWLock deadlock.  Back-patch to v12, like that commit.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/catalog/index.c

Revert "For inplace update, send nontransactional invalidations."

commit   : c1099dd745b0135960895caa8892a1873ac6cbe5    
  
author   : Noah Misch <[email protected]>    
date     : Sat, 2 Nov 2024 09:05:00 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Sat, 2 Nov 2024 09:05:00 -0700    

Click here for diff

This reverts commit 95c5acb3fc261067ab65ddc0b2dca8e162f09442 (v17) and  
counterparts in each other non-master branch.  If released, that commit  
would have caused a worst-in-years minor release regression, via  
undetected LWLock self-deadlock.  This commit and its self-deadlock fix  
warrant more bake time in the master branch.  
  
Reported by Alexander Lakhin.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/access/heap/heapam.c
M src/backend/access/transam/xact.c
M src/backend/catalog/index.c
M src/backend/commands/event_trigger.c
M src/backend/replication/logical/decode.c
M src/backend/utils/cache/catcache.c
M src/backend/utils/cache/inval.c
M src/backend/utils/cache/syscache.c
M src/include/utils/catcache.h
M src/include/utils/inval.h
M src/test/isolation/expected/inplace-inval.out
M src/test/isolation/specs/inplace-inval.spec
M src/tools/pgindent/typedefs.list

Revert "WAL-log inplace update before revealing it to other sessions."

commit   : bc6bad88572501aecaa2ac5d4bc900ac0fd457d5    
  
author   : Noah Misch <[email protected]>    
date     : Sat, 2 Nov 2024 09:04:59 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Sat, 2 Nov 2024 09:04:59 -0700    

Click here for diff

This reverts commit bfd5c6e279c8e1702eea882439dc7ebdf4d4b3a5 (v17) and  
counterparts in each other non-master branch.  This unblocks reverting a  
commit on which it depends.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/access/heap/README.tuplock
M src/backend/access/heap/heapam.c

doc: fix ALTER DOMAIN domain_constraint to spell out options

commit   : 0a0a0f2c594db57f6e77a753315284778c0f5c97    
  
author   : Bruce Momjian <[email protected]>    
date     : Fri, 1 Nov 2024 13:54:28 -0400    
  
committer: Bruce Momjian <[email protected]>    
date     : Fri, 1 Nov 2024 13:54:28 -0400    

Click here for diff

It used to refer to CREATE DOMAIN, but CREATE DOMAIN allows NULL, while  
ALTER DOMAIN does not.  
  
Reported-by: [email protected]  
  
Discussion: https://postgr.es/m/[email protected]  
  
Backpatch-through: 12  

M doc/src/sgml/ref/alter_domain.sgml

doc: remove mention of ActiveState for Perl and Tcl on Windows

commit   : 787bd3dde20c4f651210f47974859fa00221ac5e    
  
author   : Bruce Momjian <[email protected]>    
date     : Fri, 1 Nov 2024 11:30:54 -0400    
  
committer: Bruce Momjian <[email protected]>    
date     : Fri, 1 Nov 2024 11:30:54 -0400    

Click here for diff

Replace with Strawberry Perl and Magicsplat Tcl.  
  
Reported-by: Yasir Hussain  
  
Discussion: https://postgr.es/m/CAA9OW9fAAM_WDYYpAquqF6j1hmfRMzHPsFkRfP5E6oSfkF=dMA@mail.gmail.com  
  
Backpatch-through: 12  

M doc/src/sgml/installation.sgml

Stabilize jsonb_path_query test case.

commit   : a358019159de68d4f045cbb5d89c8c8c2e96e483    
  
author   : Tom Lane <[email protected]>    
date     : Wed, 30 Oct 2024 11:42:28 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Wed, 30 Oct 2024 11:42:28 -0400    

Click here for diff

An operation like '12:34:56'::time_tz takes the UTC offset from  
the prevailing time zone, which means that the results change  
across DST transitions.  One of the test cases added in ed055d249  
failed to consider this.  
  
Per report from Bernhard Wiedemann.  Back-patch to v17, as the  
test case was.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/test/regress/expected/jsonb_jsonpath.out
M src/test/regress/sql/jsonb_jsonpath.sql

Fix bug in nbtree array primitive scan scheduling.

commit   : c177726ae41a4c13c575f2adf8942687fb6ec954    
  
author   : Peter Geoghegan <[email protected]>    
date     : Wed, 30 Oct 2024 10:57:17 -0400    
  
committer: Peter Geoghegan <[email protected]>    
date     : Wed, 30 Oct 2024 10:57:17 -0400    

Click here for diff

A bug in nbtree's handling of primitive index scan scheduling could lead  
to wrong answers when a scrollable cursor was used with an index scan  
that had a SAOP index qual.  Wrong answers were only possible when the  
scan direction changed after a primitive scan was scheduled, but before  
_bt_next was asked to fetch the next tuple in line (i.e. for things to  
break, _bt_next had to be denied the opportunity to step off the page in  
the same direction as the one used when the primscan was scheduled).  
Furthermore, the issue only occurred when the page in question happened  
to be the first page to be visited by the entire top-level scan; the  
issue hinged upon the cursor backing up to the absolute beginning of the  
key space that it returns tuples from (fetching in the opposite scan  
direction across a "primitive scan boundary" always worked correctly).  
  
To fix, make _bt_next unset the "needs primitive index scan" flag when  
it detects that the current scan direction is not the one that was used  
by _bt_readpage back when the primitive scan in question was scheduled.  
This fixes the cases that are known to be faulty, and also seems like a  
good idea on general robustness grounds.  
  
Affected scrollable cursor cases now avoid a spurious primitive index  
scan when they fetch backwards to the absolute start of the key space to  
be visited by their cursor.  Fetching backwards now only returns those  
tuples at the start of the scan, as expected.  It'll also be okay to  
once again fetch forwards from the start at that point, since the scan  
will be left in a state that's exactly consistent with the state it was  
in before any tuples were ever fetched, as expected.  
  
Oversight in commit 5bf748b8, which enhanced nbtree ScalarArrayOp  
execution.  
  
Author: Peter Geoghegan <[email protected]>  
Discussion: https://postgr.es/m/CAH2-Wznv49bFsE2jkt4GuZ0tU2C91dEST=50egzjY2FeOcHL4Q@mail.gmail.com  
Backpatch: 17-, where commit 5bf748b8 first appears.  

M src/backend/access/nbtree/nbtsearch.c
M src/backend/access/nbtree/nbtutils.c

Fix some more bugs in foreign keys connecting partitioned tables

commit   : 936ab6de95957e1c407ce9dfdc38c54ad146eb5d    
  
author   : Álvaro Herrera <[email protected]>    
date     : Wed, 30 Oct 2024 10:54:03 +0100    
  
committer: Álvaro Herrera <[email protected]>    
date     : Wed, 30 Oct 2024 10:54:03 +0100    

Click here for diff

* In DetachPartitionFinalize() we were applying a tuple conversion map  
  to tuples that didn't need one, which can lead to erratic behavior if  
  a partitioned table has a partition with a different column order, as  
  reported by Alexander Lakhin. This was introduced by 53af9491a043.  
  Don't do that.  Also, modify a recently added test case to exercise  
  this.  
  
* The same function as well as CloneFkReferenced() were acquiring  
  AccessShareLock on a partition, only to have CreateTrigger() later  
  acquire ShareRowExclusiveLock on it.  This can lead to deadlock by  
  lock escalation, unnecessarily.  Avoid that by acquiring the stronger  
  lock to begin with.  This probably dates back to branch 12, but I have  
  never seen a report of this being a problem in the field.  
  
* Innocuous but wasteful: also introduced by 53af9491a043, we were  
  reading a pg_constraint tuple from syscache that we don't need, as  
  reported by Tender Wang.  Don't.  
  
Backpatch to 15.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/commands/tablecmds.c
M src/test/regress/expected/foreign_key.out
M src/test/regress/sql/foreign_key.sql

Unpin buffer before inplace update waits for an XID to end.

commit   : 9aef6f19ac27a607f0c557d19934d5ce96ed5f1a    
  
author   : Noah Misch <[email protected]>    
date     : Tue, 29 Oct 2024 09:39:55 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Tue, 29 Oct 2024 09:39:55 -0700    

Click here for diff

Commit a07e03fd8fa7daf4d1356f7cb501ffe784ea6257 changed inplace updates  
to wait for heap_update() commands like GRANT TABLE and GRANT DATABASE.  
By keeping the pin during that wait, a sequence of autovacuum workers  
and an uncommitted GRANT starved one foreground LockBufferForCleanup()  
for six minutes, on buildfarm member sarus.  Prevent, at the cost of a  
bit of complexity.  Back-patch to v12, like the earlier commit.  That  
commit and heap_inplace_lock() have not yet appeared in any release.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/access/heap/heapam.c
M src/backend/access/index/genam.c
M src/include/access/heapam.h

Update time zone data files to tzdata release 2024b.

commit   : cad65907ee42be80bea475b9fe8c02c22cb6ce0c    
  
author   : Tom Lane <[email protected]>    
date     : Tue, 29 Oct 2024 11:49:38 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Tue, 29 Oct 2024 11:49:38 -0400    

Click here for diff

Historical corrections for Mexico, Mongolia, and Portugal.  
Notably, Asia/Choibalsan is now an alias for Asia/Ulaanbaatar  
rather than being a separate zone, mainly because the differences  
between those zones were found to be based on untrustworthy data.  

M src/timezone/data/tzdata.zi
M src/timezone/known_abbrevs.txt
M src/timezone/tznames/Default
M src/timezone/tznames/Europe.txt

doc: Add better description for rewrite functions in event triggers

commit   : 709ce29b16569de7ed7d013399a6249849eaae40    
  
author   : Michael Paquier <[email protected]>    
date     : Tue, 29 Oct 2024 15:35:14 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Tue, 29 Oct 2024 15:35:14 +0900    

Click here for diff

There are two functions that can be used in event triggers to get more  
details about a rewrite happening on a relation.  Both had a limited  
documentation:  
- pg_event_trigger_table_rewrite_reason() and  
pg_event_trigger_table_rewrite_oid() were not mentioned in the main  
event trigger section in the paragraph dedicated to the event  
table_rewrite.  
- pg_event_trigger_table_rewrite_reason() returns an integer which is a  
bitmap of the reasons why a rewrite happens.  There was no explanation  
about the meaning of these values, forcing the reader to look at the  
code to find out that these are defined in event_trigger.h.  
  
While on it, let's add a comment in event_trigger.h where the  
AT_REWRITE_* are defined, telling to update the documentation when  
these values are changed.  
  
Backpatch down to 13 as a consequence of 1ad23335f36b, where this area  
of the documentation has been heavily reworked.  
  
Author: Greg Sabino Mullane  
Discussion: https://postgr.es/m/CAKAnmmL+Z6j-C8dAx1tVrnBmZJu+BSoc68WSg3sR+CVNjBCqbw@mail.gmail.com  
Backpatch-through: 13  

M doc/src/sgml/event-trigger.sgml
M doc/src/sgml/func.sgml
M src/include/commands/event_trigger.h

Doc: clarify enable_indexscan=off also disabled Index Only Scans

commit   : e5086b3ff4413fdd163c85a2ab5bfa26c7668cd3    
  
author   : David Rowley <[email protected]>    
date     : Tue, 29 Oct 2024 16:24:47 +1300    
  
committer: David Rowley <[email protected]>    
date     : Tue, 29 Oct 2024 16:24:47 +1300    

Click here for diff

Disabling enable_indexscan has always also disabled Index Only Scans.  
Here we make that more clear in the documentation in an attempt to  
prevent future complaints complaining about this expected behavior.  
  
Reported-by: Melanie Plageman  
Author: David G. Johnston, David Rowley  
Backpatch-through: 12, oldest supported version  
Discussion: https://postgr.es/m/CAAKRu_atV=kovgpaLREyG68PB5+ncKvJ2UNoeRetEgyC3Yb5Sw@mail.gmail.com  

M doc/src/sgml/config.sgml

Fix dependency of partitioned table and table AM with CREATE TABLE .. USING

commit   : bb584e831288042d9aee368a1c5daa3dafbb95c3    
  
author   : Michael Paquier <[email protected]>    
date     : Tue, 29 Oct 2024 08:41:53 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Tue, 29 Oct 2024 08:41:53 +0900    

Click here for diff

A pg_depend entry between a partitioned table and its table access  
method was missing when using CREATE TABLE .. USING with an unpinned  
access method.  DROP ACCESS METHOD could be used, while it should be  
blocked if CASCADE is not specified, even if there was a partitioned  
table that depends on the table access method.  pg_class.relam would  
then hold an orphaned OID value still pointing to the AM dropped.  
  
The problem is fixed by adding a dependency between the partitioned  
table and its table access method if set when the relation is created.  
A test checking the contents of pg_depend in this case is added.  
  
Issue introduced in 374c7a229042, that has added support for CREATE  
TABLE .. USING for partitioned tables.  
  
Reviewed-by: Alexander Lakhin  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 17  

M src/backend/catalog/heap.c
M src/test/regress/expected/create_am.out
M src/test/regress/sql/create_am.sql

Guard against enormously long input in pg_saslprep().

commit   : e631ed89c2e58464d3969badf2d5d065411b93f0    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 28 Oct 2024 14:33:55 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 28 Oct 2024 14:33:55 -0400    

Click here for diff

Coverity complained that pg_saslprep() could suffer integer overflow,  
leading to under-allocation of the output buffer, if the input string  
exceeds SIZE_MAX/4.  This hazard seems largely hypothetical, but it's  
easy enough to defend against, so let's do so.  
  
This patch creates a third place in src/common/ where we are locally  
defining MaxAllocSize so that we can test against that in the same way  
in backend and frontend compiles.  That seems like about two places  
too many, so the next patch will move that into common/fe_memutils.h.  
I'm hesitant to do that in back branches however.  
  
Back-patch to v14.  The code looks similar in older branches, but  
before commit 67a472d71 there was a separate test on the input string  
length that prevented this hazard.  
  
Per Coverity report.  

M src/common/saslprep.c

Fix overflow in bsearch_arg() with more than INT_MAX elements

commit   : 22b914121ad898cd2f5f935240f9e1334479e25f    
  
author   : Heikki Linnakangas <[email protected]>    
date     : Mon, 28 Oct 2024 14:07:38 +0200    
  
committer: Heikki Linnakangas <[email protected]>    
date     : Mon, 28 Oct 2024 14:07:38 +0200    

Click here for diff

This was introduced in commit bfa2cee784, which replaced the old  
bsearch_cmp() function we had in extended_stats.c with the current  
implementation. The original discussion or commit message of  
bfa2cee784 didn't mention where the new implementation came from, but  
based on some googling, I'm guessing *BSD or libiberty, all of which  
share this same code, with or without this fix.  
  
Author: Ranier Vilela  
Reviewed-by: Nathan Bossart  
Backpatch-through: 14  
Discussion: https://www.postgresql.org/message-id/CAEudQAp34o_8u6sGSVraLwuMv9F7T9hyHpePXHmRaxR2Aboi%2Bw%40mail.gmail.com  

M src/port/bsearch_arg.c

WAL-log inplace update before revealing it to other sessions.

commit   : bfd5c6e279c8e1702eea882439dc7ebdf4d4b3a5    
  
author   : Noah Misch <[email protected]>    
date     : Fri, 25 Oct 2024 06:51:03 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Fri, 25 Oct 2024 06:51:03 -0700    

Click here for diff

A buffer lock won't stop a reader having already checked tuple  
visibility.  If a vac_update_datfrozenid() and then a crash happened  
during inplace update of a relfrozenxid value, datfrozenxid could  
overtake relfrozenxid.  That could lead to "could not access status of  
transaction" errors.  Back-patch to v12 (all supported versions).  In  
v14 and earlier, this also back-patches the assertion removal from  
commit 7fcf2faf9c7dd473208fd6d5565f88d7f733782b.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/access/heap/README.tuplock
M src/backend/access/heap/heapam.c

For inplace update, send nontransactional invalidations.

commit   : 95c5acb3fc261067ab65ddc0b2dca8e162f09442    
  
author   : Noah Misch <[email protected]>    
date     : Fri, 25 Oct 2024 06:51:02 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Fri, 25 Oct 2024 06:51:02 -0700    

Click here for diff

The inplace update survives ROLLBACK.  The inval didn't, so another  
backend's DDL could then update the row without incorporating the  
inplace update.  In the test this fixes, a mix of CREATE INDEX and ALTER  
TABLE resulted in a table with an index, yet relhasindex=f.  That is a  
source of index corruption.  Back-patch to v12 (all supported versions).  
The back branch versions don't change WAL, because those branches just  
added end-of-recovery SIResetAll().  All branches change the ABI of  
extern function PrepareToInvalidateCacheTuple().  No PGXN extension  
calls that, and there's no apparent use case in extensions.  
  
Reviewed by Nitin Motiani and (in earlier versions) Andres Freund.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/access/heap/heapam.c
M src/backend/access/transam/xact.c
M src/backend/catalog/index.c
M src/backend/commands/event_trigger.c
M src/backend/replication/logical/decode.c
M src/backend/utils/cache/catcache.c
M src/backend/utils/cache/inval.c
M src/backend/utils/cache/syscache.c
M src/include/utils/catcache.h
M src/include/utils/inval.h
M src/test/isolation/expected/inplace-inval.out
M src/test/isolation/specs/inplace-inval.spec
M src/tools/pgindent/typedefs.list

At end of recovery, reset all sinval-managed caches.

commit   : a4668c99f0f85ba1b4ab120b5b6e168a6a5faef2    
  
author   : Noah Misch <[email protected]>    
date     : Fri, 25 Oct 2024 06:51:06 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Fri, 25 Oct 2024 06:51:06 -0700    

Click here for diff

An inplace update's invalidation messages are part of its transaction's  
commit record.  However, the update survives even if its transaction  
aborts or we stop recovery before replaying its transaction commit.  
After recovery, a backend that started in recovery could update the row  
without incorporating the inplace update.  That could result in a table  
with an index, yet relhasindex=f.  That is a source of index corruption.  
  
This bulk invalidation avoids the functional consequences.  A future  
change can fix the !RecoveryInProgress() scenario without changing the  
WAL format.  Back-patch to v17 - v12 (all supported versions).  v18 will  
instead add invalidations to WAL.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/access/transam/xlog.c
M src/backend/storage/ipc/sinvaladt.c
M src/include/storage/sinvaladt.h

Stop reading uninitialized memory in heap_inplace_lock().

commit   : e11907682846a38068cf0960cd05006b31a9335f    
  
author   : Noah Misch <[email protected]>    
date     : Thu, 24 Oct 2024 09:16:14 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Thu, 24 Oct 2024 09:16:14 -0700    

Click here for diff

Stop computing a never-used value.  This removes the read; the read had  
no functional implications.  Back-patch to v12, like commit  
a07e03fd8fa7daf4d1356f7cb501ffe784ea6257.  
  
Reported by Alexander Lakhin.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/access/heap/heapam.c
M src/test/isolation/expected/intra-grant-inplace.out
M src/test/isolation/specs/intra-grant-inplace.spec

doc: Fix INSERT statement syntax for identity columns

commit   : 0a059206fc5904820c838a6ae7478084ffdcf650    
  
author   : Daniel Gustafsson <[email protected]>    
date     : Wed, 23 Oct 2024 14:58:17 +0200    
  
committer: Daniel Gustafsson <[email protected]>    
date     : Wed, 23 Oct 2024 14:58:17 +0200    

Click here for diff

The INSERT statements in the examples were erroneously using  
VALUE instead of VALUES. Backpatch to v17 where the examples  
were added through a37bb7c1399.  
  
Reported-by: [email protected]  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 17  

M doc/src/sgml/ddl.sgml

Remove unnecessary word in a comment

commit   : f92f6b3db4fe40059845a9ea1f71666881deedd1    
  
author   : Amit Langote <[email protected]>    
date     : Wed, 23 Oct 2024 17:54:48 +0900    
  
committer: Amit Langote <[email protected]>    
date     : Wed, 23 Oct 2024 17:54:48 +0900    

Click here for diff

Relations opened by the executor are only closed once in  
ExecCloseRangeTableRelations(), so the word "again" in the comment  
for ExecGetRangeTableRelation() is misleading and unnecessary.  
  
Discussion: https://postgr.es/m/CA+HiwqHnw-zR+u060i3jp4ky5UR0CjByRFQz50oZ05de7wUg=Q@mail.gmail.com  
Backpatch-through: 12  

M src/backend/executor/execUtils.c

ecpg: Fix out-of-bound read in DecodeDateTime()

commit   : 2c37cb26f8af0aecb1b534585f3a0234f70e69ee    
  
author   : Michael Paquier <[email protected]>    
date     : Wed, 23 Oct 2024 08:35:00 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Wed, 23 Oct 2024 08:35:00 +0900    

Click here for diff

It was possible for the code to read out-of-bound data from the  
"day_tab" table with some crafted input data.  Let's treat these as  
invalid input as the month number is incorrect.  
  
A test is added to test this case with a check on the errno returned by  
the decoding routine.  A test close to the new one added in this commit  
was testing for a failure, but did not look at the errno generated, so  
let's use this commit to also change it, adding a check on the errno  
returned by DecodeDateTime().  
  
Like the other test scripts, dt_test should likely be expanded to  
include more checks based on the errnos generated in these code paths.  
This is left as future work.  
  
This issue exists since 2e6f97560a83, so backpatch all the way down.  
  
Reported-by: Pavel Nekrasov  
Author: Bruce Momjian, Pavel Nekrasov  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 12  

M src/interfaces/ecpg/pgtypeslib/dt_common.c
M src/interfaces/ecpg/test/expected/pgtypeslib-dt_test.c
M src/interfaces/ecpg/test/expected/pgtypeslib-dt_test.stderr
M src/interfaces/ecpg/test/expected/pgtypeslib-dt_test.stdout
M src/interfaces/ecpg/test/pgtypeslib/dt_test.pgc

Restructure foreign key handling code for ATTACH/DETACH

commit   : 5914a22f6ea5a3a8485cf626e76a63f3f6463de4    
  
author   : Álvaro Herrera <[email protected]>    
date     : Tue, 22 Oct 2024 16:01:18 +0200    
  
committer: Álvaro Herrera <[email protected]>    
date     : Tue, 22 Oct 2024 16:01:18 +0200    

Click here for diff

... to fix bugs when the referenced table is partitioned.  
  
The catalog representation we chose for foreign keys connecting  
partitioned tables (in commit f56f8f8da6af) is inconvenient, in the  
sense that a standalone table has a different way to represent the  
constraint when referencing a partitioned table, than when the same  
table becomes a partition (and vice versa).  Because of this, we need to  
create additional catalog rows on detach (pg_constraint and pg_trigger),  
and remove them on attach.  We were doing some of those things, but not  
all of them, leading to missing catalog rows in certain cases.  
  
The worst problem seems to be that we are missing action triggers after  
detaching a partition, which means that you could update/delete rows  
from the referenced partitioned table that still had referencing rows on  
that table, the server failing to throw the required errors.  
  
!!!  
Note that this means existing databases with FKs that reference  
partitioned tables might have rows that break relational integrity, on  
tables that were once partitions on the referencing side of the FK.  
  
Another possible problem is that trying to reattach a table  
that had been detached would fail indicating that internal triggers  
cannot be found, which from the user's point of view is nonsensical.  
  
In branches 15 and above, we fix this by creating a new helper function  
addFkConstraint() which is in charge of creating a standalone  
pg_constraint row, and repurposing addFkRecurseReferencing() and  
addFkRecurseReferenced() so that they're only the recursive routine for  
each side of the FK, and they call addFkConstraint() to create  
pg_constraint at each partitioning level and add the necessary triggers.  
These new routines can be used during partition creation, partition  
attach and detach, and foreign key creation.  This reduces redundant  
code and simplifies the flow.  
  
In branches 14 and 13, we have a much simpler fix that consists on  
simply removing the constraint on detach.  The reason is that those  
branches are missing commit f4566345cf40, which reworked the way this  
works in a way that we didn't consider back-patchable at the time.  
  
We opted to leave branch 12 alone, because it's different from branch 13  
enough that the fix doesn't apply; and because it is going in EOL mode  
very soon, patching it now might be worse since there's no way to undo  
the damage if it goes wrong.  
  
Existing databases might need to be repaired.  
  
In the future we might want to rethink the catalog representation to  
avoid this problem, but for now the code seems to do what's required to  
make the constraints operate correctly.  
  
Co-authored-by: Jehan-Guillaume de Rorthais <[email protected]>  
Co-authored-by: Tender Wang <[email protected]>  
Co-authored-by: Alvaro Herrera <[email protected]>  
Reported-by: Guillaume Lelarge <[email protected]>  
Reported-by: Jehan-Guillaume de Rorthais <[email protected]>  
Reported-by: Thomas Baehler (SBB CFF FFS) <[email protected]>  
Discussion: https://postgr.es/m/20230420144344.40744130@karst  
Discussion: https://postgr.es/m/20230705233028.2f554f73@karst  
Discussion: https://postgr.es/m/GVAP278MB02787E7134FD691861635A8BC9032@GVAP278MB0278.CHEP278.PROD.OUTLOOK.COM  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/commands/tablecmds.c
M src/test/regress/expected/foreign_key.out
M src/test/regress/sql/foreign_key.sql
M src/tools/pgindent/typedefs.list

Fix wrong assertion and poor error messages in "COPY (query) TO".

commit   : 3685ad6188c1697c8a685f8fabe79e508b0dcc6e    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 21 Oct 2024 15:08:22 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 21 Oct 2024 15:08:22 -0400    

Click here for diff

If the query is rewritten into a NOTIFY command by a DO INSTEAD  
rule, we'd get an assertion failure, or in non-assert builds  
issue a rather confusing error message.  Improve that.  
  
Also fix a longstanding grammar mistake in a nearby error message.  
  
Per bug #18664 from Alexander Lakhin.  Back-patch to all supported  
branches.  
  
Tender Wang and Tom Lane  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/commands/copyto.c
M src/test/regress/expected/copydml.out
M src/test/regress/sql/copydml.sql

Fix race condition in committing a serializable transaction

commit   : 234f6d09e531b3b5b021ce9574737219a6ac844b    
  
author   : Heikki Linnakangas <[email protected]>    
date     : Mon, 21 Oct 2024 09:49:21 +0300    
  
committer: Heikki Linnakangas <[email protected]>    
date     : Mon, 21 Oct 2024 09:49:21 +0300    

Click here for diff

The finished transaction list can contain XIDs that are older than the  
serializable global xmin. It's a short-lived state;  
ClearOldPredicateLocks() removes any such transactions from the list,  
and it's called whenever the global xmin advances. But if another  
backend calls SummarizeOldestCommittedSxact() in that window, it will  
call SerialAdd() on an XID that's older than the global xmin, or if  
there are no more transactions running, when global xmin is  
invalid. That trips the assertion in SerialAdd().  
  
Fixes bug #18658 reported by Andrew Bille. Thanks to Alexander Lakhin  
for analysis. Backpatch to all versions.  
  
Discussion: https://www.postgresql.org/message-id/18658-7dab125ec688c70b%40postgresql.org  

M src/backend/storage/lmgr/predicate.c

Note that index_name in ALTER INDEX ATTACH PARTITION can be schema-qualified

commit   : e9959ff71c4065b2e2912e4982bba9d5e2eef279    
  
author   : Álvaro Herrera <[email protected]>    
date     : Sun, 20 Oct 2024 15:36:20 +0200    
  
committer: Álvaro Herrera <[email protected]>    
date     : Sun, 20 Oct 2024 15:36:20 +0200    

Click here for diff

Missed in 8b08f7d4820f; backpatch to all supported branches.  
  
Reported-by: [email protected]  
Reviewed-by: Tom Lane <[email protected]>  
Discussion: https://postgr.es/m/[email protected]  

M doc/src/sgml/ref/alter_index.sgml

SQL/JSON: Fix some oversights in commit b6e1157e7

commit   : 7148cb3e3067a21f98031ff4d01be8895b5d164e    
  
author   : Amit Langote <[email protected]>    
date     : Sun, 20 Oct 2024 12:21:12 +0900    
  
committer: Amit Langote <[email protected]>    
date     : Sun, 20 Oct 2024 12:21:12 +0900    

Click here for diff

The decision in b6e1157e7 to ignore raw_expr when evaluating a  
JsonValueExpr was incorrect.  While its value is not ultimately  
used (since formatted_expr's value is), failing to initialize it  
can lead to problems, for instance,  when the expression tree in  
raw_expr contains Aggref nodes, which must be initialized to  
ensure the parent Agg node works correctly.  
  
Also, optimize eval_const_expressions_mutator()'s handling of  
JsonValueExpr a bit.  Currently, when formatted_expr cannot be folded  
into a constant, we end up processing it twice -- once directly in  
eval_const_expressions_mutator() and again recursively via  
ece_generic_processing().  This recursive processing is required to  
handle raw_expr. To avoid the redundant processing of formatted_expr,  
we now  process raw_expr directly in eval_const_expressions_mutator().  
  
Finally, update the comment of JsonValueExpr to describe the roles of  
raw_expr and formatted_expr more clearly.  
  
Bug: #18657  
Reported-by: Alexander Lakhin <[email protected]>  
Diagnosed-by: Fabio R. Sluzala <[email protected]>  
Diagnosed-by: Tender Wang <[email protected]>  
Reviewed-by: Tom Lane <[email protected]>  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 16  

M src/backend/executor/execExpr.c
M src/backend/optimizer/util/clauses.c
M src/include/nodes/primnodes.h
M src/test/regress/expected/sqljson.out
M src/test/regress/sql/sqljson.sql

Adjust documentation for configuring Linux huge pages.

commit   : 053b6daeb9c5ec26289479ab63601f0cd39f51a7    
  
author   : Nathan Bossart <[email protected]>    
date     : Fri, 18 Oct 2024 10:20:15 -0500    
  
committer: Nathan Bossart <[email protected]>    
date     : Fri, 18 Oct 2024 10:20:15 -0500    

Click here for diff

The present wording about viewing shared_memory_size_in_huge_pages  
seems to suggest that the parameter cannot be viewed after startup  
at all, whereas the intent is to make it clear that you can't use  
"postgres -C" to view this parameter while the server is running.  
This commit rephrases this section to remove the ambiguity.  
  
Author: Seino Yuki  
Reviewed-by: Michael Paquier, David G. Johnston, Fujii Masao  
Discussion: https://postgr.es/m/420584fd274f9ec4f337da55ffb3b790%40oss.nttdata.com  
Backpatch-through: 15  

M doc/src/sgml/runtime.sgml

Fix description of PostgreSQL::Test::Cluster::wait_for_event()

commit   : b8d08aafc0ef8b5f7bf15dcf4bc6af736ee81f15    
  
author   : Michael Paquier <[email protected]>    
date     : Fri, 18 Oct 2024 13:50:07 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Fri, 18 Oct 2024 13:50:07 +0900    

Click here for diff

The arguments of the function were listed in an incorrect order in the  
description of the routine.  This information can be seen with perldoc.  
  
Issue spotted while working on this area of the code.  
  
Backpatch-through: 17  

M src/test/perl/PostgreSQL/Test/Cluster.pm

Fix extreme skew detection in Parallel Hash Join.

commit   : 4ac5d33a8b08eac6e1df235eb38b48e18195c7a4    
  
author   : Thomas Munro <[email protected]>    
date     : Thu, 17 Oct 2024 15:52:24 +1300    
  
committer: Thomas Munro <[email protected]>    
date     : Thu, 17 Oct 2024 15:52:24 +1300    

Click here for diff

After repartitioning the inner side of a hash join that would have  
exceeded the allowed size, we check if all the tuples from a parent  
partition moved to one child partition.  That is evidence that it  
contains duplicate keys and later attempts to repartition will also  
fail, so we should give up trying to limit memory (for lack of a better  
fallback strategy).  
  
A thinko prevented the check from working correctly in partition 0 (the  
one that is partially loaded into memory already).  After  
repartitioning, we should check for extreme skew if the *parent*  
partition's space_exhausted flag was set, not the child partition's.  
The consequence was repeated futile repartitioning until per-partition  
data exceeded various limits including "ERROR: invalid DSA memory alloc  
request size 1811939328", OS allocation failure, or temporary disk space  
errors.  (We could also do something about some of those symptoms, but  
that's material for separate patches.)  
  
This problem only became likely when PostgreSQL 16 introduced support  
for Parallel Hash Right/Full Join, allowing NULL keys into the hash  
table.  Repartitioning always leaves NULL in partition 0, no matter how  
many times you do it, because the hash value is all zero bits.  That's  
unlikely for other hashed values, but they might still have caused  
wasted extra effort before giving up.  
  
Back-patch to all supported releases.  
  
Reported-by: Craig Milhiser <[email protected]>  
Reviewed-by: Andrei Lepikhov <[email protected]>  
Discussion: https://postgr.es/m/CA%2BwnhO1OfgXbmXgC4fv_uu%3DOxcDQuHvfoQ4k0DFeB0Qqd-X-rQ%40mail.gmail.com  

M src/backend/executor/nodeHash.c

Fix whitespace

commit   : e90d108823ad647bc67327ee14ebfd097f262c8f    
  
author   : Peter Eisentraut <[email protected]>    
date     : Thu, 17 Oct 2024 08:42:58 +0200    
  
committer: Peter Eisentraut <[email protected]>    
date     : Thu, 17 Oct 2024 08:42:58 +0200    

Click here for diff

M src/test/regress/expected/merge.out
M src/test/regress/sql/merge.sql

Fix validation of COPY FORCE_NOT_NULL/FORCE_NULL for the all-column cases

commit   : c06a4746b1be841dee998a75f9a52b2d06348ca7    
  
author   : Michael Paquier <[email protected]>    
date     : Thu, 17 Oct 2024 08:45:56 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Thu, 17 Oct 2024 08:45:56 +0900    

Click here for diff

This commit adds missing checks for COPY FORCE_NOT_NULL and FORCE_NULL  
when applied to all columns via "*".  These options now correctly  
require CSV mode and are disallowed in COPY TO, making their behavior  
consistent with FORCE_QUOTE.  
  
Some regression tests are added to verify the correct behavior for the  
all-columns case, including FORCE_QUOTE, which was not tested.  
  
Backpatch down to 17, where support for the all-column grammar with  
FORCE_NOT_NULL and FORCE_NULL has been added.  
  
Author: Joel Jacobson  
Reviewed-by: Zhang Mingli  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 17  

M src/backend/commands/copy.c
M src/test/regress/expected/copy2.out
M src/test/regress/sql/copy2.sql

Rewrite some regression queries for option checks with COPY

commit   : a30c1ca21c584cb0917a5e6fbe086164cd8bec6a    
  
author   : Michael Paquier <[email protected]>    
date     : Thu, 17 Oct 2024 07:21:40 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Thu, 17 Oct 2024 07:21:40 +0900    

Click here for diff

Some queries in copy2 are there to check various option combinations,  
and used "stdin" or "stdout" incompatible with the COPY TO or FROM  
clauses combined with them, which was confusing.  This commit rewrites  
these queries to use a compatible grammar.  
  
The coverage of the tests is unchanged.  Like the original commit  
451d1164b9d0, backpatch down to 16 where these have been introduced.  A  
follow-up commit will rely on this area of the tests for a bug fix.  
  
Author: Joel Jacobson  
Reviewed-by: Zhang Mingli  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 16  

M src/test/regress/expected/copy2.out
M src/test/regress/sql/copy2.sql

Further refine _SPI_execute_plan's rule for atomic execution.

commit   : b5eef75391c8e9dd770ba3de9625bb260b086295    
  
author   : Tom Lane <[email protected]>    
date     : Wed, 16 Oct 2024 17:36:29 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Wed, 16 Oct 2024 17:36:29 -0400    

Click here for diff

Commit 2dc1deaea turns out to have been still a brick shy of a load,  
because CALL statements executing within a plpgsql exception block  
could still pass the wrong snapshot to stable functions within the  
CALL's argument list.  That happened because standard_ProcessUtility  
forces isAtomicContext to true if IsTransactionBlock is true, which  
it always will be inside a subtransaction.  Then ExecuteCallStmt  
would think it does not need to push a new snapshot --- but  
_SPI_execute_plan didn't do so either, since it thought it was in  
nonatomic mode.  
  
The best fix for this seems to be for _SPI_execute_plan to operate  
in atomic execution mode if IsSubTransaction() is true, even when the  
SPI context as a whole is non-atomic.  This makes _SPI_execute_plan  
have the same rules about when non-atomic execution is allowed as  
_SPI_commit/_SPI_rollback have about when COMMIT/ROLLBACK are allowed,  
which seems appropriately symmetric.  (If anyone ever tries to allow  
COMMIT/ROLLBACK inside a subtransaction, this would all need to be  
rethought ... but I'm unconvinced that such a thing could be logically  
consistent at all.)  
  
For further consistency, also check IsSubTransaction() in  
SPI_inside_nonatomic_context.  That does not matter for its  
one present-day caller StartTransaction, which can't be reached  
inside a subtransaction.  But if any other callers ever arise,  
they'd presumably want this definition.  
  
Per bug #18656 from Alexander Alehin.  Back-patch to all  
supported branches, like previous fixes in this area.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/executor/spi.c
M src/pl/plpgsql/src/expected/plpgsql_call.out
M src/pl/plpgsql/src/sql/plpgsql_call.sql

Reduce memory block size for decoded tuple storage to 8kB.

commit   : eef9cc4dc28fbe17d621eeb773c56e9ac3f065ff    
  
author   : Masahiko Sawada <[email protected]>    
date     : Wed, 16 Oct 2024 12:08:02 -0700    
  
committer: Masahiko Sawada <[email protected]>    
date     : Wed, 16 Oct 2024 12:08:02 -0700    

Click here for diff

Commit a4ccc1cef introduced the Generation Context and modified the  
logical decoding process to use a Generation Context with a fixed  
block size of 8MB for storing tuple data decoded during logical  
decoding (i.e., rb->tup_context). Several reports have indicated that  
the logical decoding process can be terminated due to  
out-of-memory (OOM) situations caused by excessive memory usage in  
rb->tup_context.  
  
This issue can occur when decoding a workload involving several  
concurrent transactions, including a long-running transaction that  
modifies tuples. By design, the Generation Context does not free a  
memory block until all chunks within that block are  
released. Consequently, if tuples modified by the long-running  
transaction are stored across multiple memory blocks, these blocks  
remain allocated until the long-running transaction completes, leading  
to substantial memory fragmentation. The memory usage during logical  
decoding, tracked by rb->size, does not account for memory  
fragmentation, resulting in potentially much higher memory consumption  
than the value of the logical_decoding_work_mem parameter.  
  
Various improvement strategies were discussed in the relevant  
thread. This change reduces the block size of the Generation Context  
used in rb->tup_context from 8MB to 8kB. This modification  
significantly decreases the likelihood of substantial memory  
fragmentation occurring and is relatively straightforward to  
backport. Performance testing across multiple platforms has confirmed  
that this change will not introduce any performance degradation that  
would impact actual operation.  
  
Backport to all supported branches.  
  
Reported-by: Alex Richman, Michael Guissine, Avi Weinberg  
Reviewed-by: Amit Kapila, Fujii Masao, David Rowley  
Tested-by: Hayato Kuroda, Shlok Kyal  
Discussion: https://postgr.es/m/CAD21AoBTY1LATZUmvSXEssvq07qDZufV4AF-OHh9VD2pC0VY2A%40mail.gmail.com  
Backpatch-through: 12  

M src/backend/replication/logical/reorderbuffer.c

Fix typo in comment of transformJsonAggConstructor()

commit   : 064e04008533b2b8a82b5dbff7da10abd6e41565    
  
author   : Amit Langote <[email protected]>    
date     : Wed, 16 Oct 2024 18:11:53 +0900    
  
committer: Amit Langote <[email protected]>    
date     : Wed, 16 Oct 2024 18:11:53 +0900    

Click here for diff

An oversight of 3a8a1f3254b.  
  
Reported-by: Tender Wang <[email protected]>  
Author: Tender Wang <[email protected]>  
Backpatch-through: 16  

M src/backend/parser/parse_expr.c

Add type cast to foreach_internal's loop variable.

commit   : 8aaca078510436213be5416a7227115ec83b68a3    
  
author   : Nathan Bossart <[email protected]>    
date     : Tue, 15 Oct 2024 16:20:49 -0500    
  
committer: Nathan Bossart <[email protected]>    
date     : Tue, 15 Oct 2024 16:20:49 -0500    

Click here for diff

C++ requires explicitly casting void pointers to the appropriate  
pointer type, which means the foreach_ptr macro cannot be used in  
C++ code without this change.  
  
Author: Jelte Fennema-Nio  
Reviewed-by: Bruce Momjian  
Discussion: https://postgr.es/m/CAGECzQSYG3QfHrc-rOk2KbnB9iJOd7Qu-Xii1s-GTA%3D3JFt49Q%40mail.gmail.com  
Backpatch-through: 17  

M src/include/nodes/pg_list.h

psql: Fix \watch when using interval values less than 1ms

commit   : 8a6170860c83d3f44b7d97193a2c23539c24f76d    
  
author   : Michael Paquier <[email protected]>    
date     : Mon, 14 Oct 2024 12:27:57 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Mon, 14 Oct 2024 12:27:57 +0900    

Click here for diff

Attempting to use an interval of time less than 1ms would cause \watch  
to hang.  This was confusing, so let's change the logic so as an  
interval lower than 1ms behaves the same as 0.  
  
Comments are added to mention that the internals of do_watch() had  
better rely on "sleep_ms", the interval value in milliseconds.  While on  
it, this commit adds a test to check the behavior of interval values  
less than 1ms.  
  
\watch hanging for interval values less than 1ms existed before  
6f9ee74d45aa, that has changed the code to support an interval value of  
0.  
  
Reported-by: Heikki Linnakangas  
Author: Andrey M. Borodin, Michael Paquier  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 16  

M src/bin/psql/command.c
M src/bin/psql/t/001_basic.pl

Correctly identify which EC members are computable at a plan node.

commit   : 54889ea64baaa5970330a0097fb6689ee1af7591    
  
author   : Tom Lane <[email protected]>    
date     : Sat, 12 Oct 2024 14:56:08 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Sat, 12 Oct 2024 14:56:08 -0400    

Click here for diff

find_computable_ec_member() had the wrong mental model of what  
its primary caller prepare_sort_from_pathkeys() would do with  
the selected EquivalenceClass member expression.  We will not  
compute the EC expression in a plan node atop the one returning  
the passed-in targetlist; rather, the EC expression will be  
computed as an additional column of that targetlist.  So any  
Var or quasi-Var used in the given tlist is also available to the  
EC expression.  In simple cases this makes no difference because  
the given tlist is just a list of Vars or quasi-Vars --- but if  
we are considering an appendrel member produced by flattening  
a UNION ALL, the tlist may contain expressions, resulting in  
failure to match and a "could not find pathkey item to sort"  
error.  
  
To fix, we can flatten both the tlist and the EC members with  
pull_var_clause(), and then just check for subset-ness, so  
that the code is actually shorter than before.  
  
While this bug is quite old, the present patch only works back to  
v13.  We could possibly make it work in v12 by back-patching parts  
of 375398244.  On the whole though I don't like the risk/reward  
ratio of that idea.  v12's final release is next month, meaning  
there would be no chance to correct matters if the patch causes a  
regression.  Since this failure has escaped notice for 14 years,  
it's likely nobody will hit it in the field with v12.  
  
Per bug #18652 from Alexander Lakhin.  
  
Andrei Lepikhov and Tom Lane  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/optimizer/path/equivclass.c
M src/test/regress/expected/inherit.out
M src/test/regress/sql/inherit.sql

Fix missed case for builtin collation provider.

commit   : ff33df26c2446107a20e3432b4c73850536678b9    
  
author   : Jeff Davis <[email protected]>    
date     : Fri, 11 Oct 2024 16:58:22 -0700    
  
committer: Jeff Davis <[email protected]>    
date     : Fri, 11 Oct 2024 16:58:22 -0700    

Click here for diff

A missed check for the builtin collation provider could result in  
falling through to call isalpha().  
  
This does not appear to have practical consequences because it only  
happens for characters in the ASCII range. Regardless, the builtin  
provider should not be calling libc functions, so backpatch.  
  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 17  

M src/backend/utils/adt/like_support.c

doc PG 17 relnotes: clarify pg_upgrade and logical slot preserv.

commit   : 912d15cba50c257f2195f79d3f80bad26996c018    
  
author   : Bruce Momjian <[email protected]>    
date     : Wed, 9 Oct 2024 23:05:50 -0400    
  
committer: Bruce Momjian <[email protected]>    
date     : Wed, 9 Oct 2024 23:05:50 -0400    

Click here for diff

Reported-by: Amit Kapila  
  
Discussion: https://postgr.es/m/CAA4eK1+bChgccySmcm0LbvkmBDJ3ufsLneF4iNa_aZ7t2P6=8w@mail.gmail.com  
  
Backpatch-through: 17 only  

M doc/src/sgml/release-17.sgml

doc PG 17 relnotes: add missing commands for safe search path

commit   : 7e059fb6c04e76b712a5a92de8c62e56f42e1bbf    
  
author   : Bruce Momjian <[email protected]>    
date     : Wed, 9 Oct 2024 22:58:03 -0400    
  
committer: Bruce Momjian <[email protected]>    
date     : Wed, 9 Oct 2024 22:58:03 -0400    

Click here for diff

Reported-by: Yugo NAGATA  
  
Discussion: https://postgr.es/m/[email protected]  
  
Backpatch-through: 17 only  

M doc/src/sgml/release-17.sgml

Avoid crash in estimate_array_length with null root pointer.

commit   : a3c4a91f1e283cc4b79f0b0482d2c490a599d880    
  
author   : Tom Lane <[email protected]>    
date     : Wed, 9 Oct 2024 17:07:53 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Wed, 9 Oct 2024 17:07:53 -0400    

Click here for diff

Commit 9391f7152 added a "PlannerInfo *root" parameter to  
estimate_array_length, but failed to consider the possibility that  
NULL would be passed for that, leading to a null pointer dereference.  
  
We could rectify the particular case shown in the bug report by fixing  
simplify_function/inline_function to pass through the root pointer.  
However, as long as eval_const_expressions is documented to accept  
NULL for root, similar hazards would remain.  For now, let's just do  
the narrow fix of hardening estimate_array_length to not crash.  
Its behavior with NULL root will be the same as it was before  
9391f7152, so this is not too awful.  
  
Per report from Fredrik Widlert (via Paul Ramsey).  Back-patch to v17  
where 9391f7152 came in.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/optimizer/path/costsize.c
M src/backend/utils/adt/selfuncs.c

doc: Fix mention of AT LOCAL in release notes

commit   : 647e76c0ff4ad81a457b66ef1581e84b5edf3f84    
  
author   : Daniel Gustafsson <[email protected]>    
date     : Wed, 9 Oct 2024 10:13:20 +0200    
  
committer: Daniel Gustafsson <[email protected]>    
date     : Wed, 9 Oct 2024 10:13:20 +0200    

Click here for diff

The release notes accidentally spelled AT LOCAL as AS LOCAL.  
  
Reported-by: Takatsuka Haruka <[email protected]>  
Discussion: https://postgr.es/m/[email protected]  

M doc/src/sgml/release-17.sgml

Remove incorrect function import from pgindent

commit   : c5b9097255ba70eed9ffe3d804453c80dc1a79f0    
  
author   : Daniel Gustafsson <[email protected]>    
date     : Wed, 9 Oct 2024 09:34:34 +0200    
  
committer: Daniel Gustafsson <[email protected]>    
date     : Wed, 9 Oct 2024 09:34:34 +0200    

Click here for diff

Commit 149ac7d4559 which re-implemented pgindent in Perl explicitly  
imported the devnull function from File::Spec, but the module does  
not export anything.  In recent versions of Perl calling a missing  
import function cause a warning, which combined with warnings being  
fatal cause pgindent to error out.  
  
Backpatch to all supported versions.  
  
Author: Erik Wienhold <[email protected]>  
Reviewed-by: Andrew Dunstan <[email protected]>  
Reviewed-by: Daniel Gustafsson <[email protected]>  
Discusson: https://postgr.es/m/[email protected]  
Backpatch-through: v12  

M src/tools/pgindent/pgindent

Stabilize the test added by commit 022564f60c.

commit   : c4b8a916f8a53379621825028532b038e004f891    
  
author   : Amit Kapila <[email protected]>    
date     : Tue, 8 Oct 2024 12:13:28 +0530    
  
committer: Amit Kapila <[email protected]>    
date     : Tue, 8 Oct 2024 12:13:28 +0530    

Click here for diff

The test was unstable in branches 14 and 15 as we were relying on the  
number of changes in the table having a toast column to start streaming.  
On branches >= 16, we have a GUC debug_logical_replication_streaming which  
can stream each change, so the test was stable in those branches.  
  
Change the test to use PREPARE TRANSACTION as that should make the result  
consistent and test the code changed in 022564f60c.  
  
Reported-by: Daniel Gustafsson as per buildfarm  
Author: Hou Zhijie, Amit Kapila  
Backpatch-through: 14  
Discussion: https://postgr.es/m/[email protected]  

M contrib/test_decoding/expected/stream.out
M contrib/test_decoding/expected/twophase.out
M contrib/test_decoding/sql/stream.sql
M contrib/test_decoding/sql/twophase.sql

Fix search_path cache initialization.

commit   : 2fe4167bc6f2a446eb3112bab14ae98958e490f4    
  
author   : Jeff Davis <[email protected]>    
date     : Mon, 7 Oct 2024 17:54:19 -0700    
  
committer: Jeff Davis <[email protected]>    
date     : Mon, 7 Oct 2024 17:54:19 -0700    

Click here for diff

The cache needs to be available very early, so don't rely on  
InitializeSearchPath() to initialize the it.  
  
Reported-by: Murat Efendioğlu  
Discussion: https://postgr.es/m/CACbCzujQ4zS8MM1bx-==+tr+D3Hk5G1cjN4XkUQ+Q=cEpwhzqg@mail.gmail.com  
Backpatch-through: 17  

M src/backend/catalog/namespace.c

doc PG 17 relnotes: move adminpack item to incompatibilities

commit   : a8b2402041c79a16a0c5e0c5498f9f0e242f87f3    
  
author   : Bruce Momjian <[email protected]>    
date     : Mon, 7 Oct 2024 20:11:25 -0400    
  
committer: Bruce Momjian <[email protected]>    
date     : Mon, 7 Oct 2024 20:11:25 -0400    

Click here for diff

Reported-by: Laurenz Albe  
  
Discussion: https://postgr.es/m/[email protected]  
  
Backpatch-through: 17 only  

M doc/src/sgml/release-17.sgml

vacuumdb: Schema-qualify operator in catalog query's WHERE clause.

commit   : 5bd26e6527800749641dcb6a4409c0eb0ec1ba8d    
  
author   : Nathan Bossart <[email protected]>    
date     : Mon, 7 Oct 2024 16:49:20 -0500    
  
committer: Nathan Bossart <[email protected]>    
date     : Mon, 7 Oct 2024 16:49:20 -0500    

Click here for diff

Commit 1ab67c9dfa, which modified this catalog query so that it  
doesn't return temporary relations, forgot to schema-qualify the  
operator.  A comment earlier in the function implores us to fully  
qualify everything in the query:  
  
	 * Since we execute the constructed query with the default search_path  
	 * (which could be unsafe), everything in this query MUST be fully  
	 * qualified.  
  
This commit fixes that.  While at it, add a newline for consistency  
with surrounding code.  
  
Reviewed-by: Noah Misch  
Discussion: https://postgr.es/m/ZwQJYcuPPUsF0reU%40nathan  
Backpatch-through: 12  

M src/bin/scripts/vacuumdb.c

Fix Y2038 issues with MyStartTime.

commit   : a356d23fd3ad7884895a68d4f9c9fdd643d686ae    
  
author   : Nathan Bossart <[email protected]>    
date     : Mon, 7 Oct 2024 13:51:03 -0500    
  
committer: Nathan Bossart <[email protected]>    
date     : Mon, 7 Oct 2024 13:51:03 -0500    

Click here for diff

Several places treat MyStartTime as a "long", which is only 32 bits  
wide on some platforms.  In reality, MyStartTime is a pg_time_t,  
i.e., a signed 64-bit integer.  This will lead to interesting bugs  
on the aforementioned systems in 2038 when signed 32-bit integers  
are no longer sufficient to store Unix time (e.g., "pg_ctl start"  
hanging).  To fix, ensure that MyStartTime is handled as a 64-bit  
value everywhere.  (Of course, users will need to ensure that  
time_t is 64 bits wide on their system, too.)  
  
Co-authored-by: Max Johnson  
Discussion: https://postgr.es/m/CO1PR07MB905262E8AC270FAAACED66008D682%40CO1PR07MB9052.namprd07.prod.outlook.com  
Backpatch-through: 12  

M contrib/postgres_fdw/option.c
M src/backend/utils/error/csvlog.c
M src/backend/utils/error/elog.c
M src/backend/utils/error/jsonlog.c
M src/backend/utils/init/miscinit.c
M src/bin/pg_ctl/pg_ctl.c

Fix fetching default toast value during decoding of in-progress transactions.

commit   : 91810775904f48d8bceeb6c3197105f71da1e314    
  
author   : Amit Kapila <[email protected]>    
date     : Mon, 7 Oct 2024 15:15:05 +0530    
  
committer: Amit Kapila <[email protected]>    
date     : Mon, 7 Oct 2024 15:15:05 +0530    

Click here for diff

During logical decoding of in-progress transactions, we perform the toast  
table scan while fetching the default toast value for an attribute. We  
forgot to initialize the flag during this scan to indicate that the system  
table scan is in progress. We need this flag to ensure that during logical  
decoding we never directly access the tableam or heap APIs because we check  
for concurrent aborts only in systable_* APIs.  
  
Reported-by: Alexander Lakhin  
Author: Takeshi Ideriha, Hou Zhijie  
Reviewed-by: Amit Kapila, Hou Zhijie  
Backpatch-through: 14  
Discussion: https://postgr.es/m/[email protected]  

M contrib/test_decoding/expected/stream.out
M contrib/test_decoding/sql/stream.sql
M src/backend/access/index/genam.c

Ignore not-yet-defined Portals in pg_cursors view.

commit   : 3daeb539a66acce5d8870b2b0d99d4ec9a2c0314    
  
author   : Tom Lane <[email protected]>    
date     : Sun, 6 Oct 2024 16:03:48 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Sun, 6 Oct 2024 16:03:48 -0400    

Click here for diff

pg_cursor() supposed that any Portal it finds in the hash table must  
have sourceText set up, but there's an edge case where that is not so.  
A newly-created Portal has sourceText = NULL, and that doesn't change  
until PortalDefineQuery is called.  In SPI_cursor_open_internal,  
we perform GetCachedPlan between CreatePortal and PortalDefineQuery,  
and it's possible for user-defined code to execute during that  
planning and cause a fetch from the pg_cursors view, resulting in a  
null-pointer-dereference crash.  (It looks like the same could happen  
in exec_bind_message, but I've not tried to provoke a failure there.)  
  
I considered trying to fix this by setting sourceText sooner, but  
there may be instances of this same calling pattern in extensions,  
and we couldn't be sure they'd get the memo promptly.  It seems  
better to redefine pg_cursor as not showing Portals that have  
not yet had PortalDefineQuery called on them, which we can do by  
just skipping them if sourceText is still NULL.  
  
(Before a1c692358, pg_cursor would instead return a row with NULL  
in the statement column.  We could revert to that behavior but it  
doesn't really seem like a better definition, especially since our  
documentation doesn't suggest that the column could be NULL.)  
  
Per report from PetSerAl.  Back-patch to all supported branches.  
  
Discussion: https://postgr.es/m/CAKygsHTBXLXjwV43kpZa+Cs+XTiaeeJiZdL4cPBm9f4MTdw7wg@mail.gmail.com  

M src/backend/utils/mmgr/portalmem.c

Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.

commit   : fee8cb9473462e023dcf0b41212ca3890ddc28d6    
  
author   : Tom Lane <[email protected]>    
date     : Sat, 5 Oct 2024 14:46:44 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Sat, 5 Oct 2024 14:46:44 -0400    

Click here for diff

When instantiating an existing partitioned index for a new child  
partition, we use generateClonedIndexStmt to build a suitable  
IndexStmt to pass to DefineIndex.  However, when DefineIndex needs  
to recurse to instantiate a newly created partitioned index on an  
existing child partition, it was doing copyObject on the given  
IndexStmt and then applying a bunch of ad-hoc fixups.  This has  
a number of problems, primarily that it implies fresh lookups of  
referenced objects such as opclasses and collations.  Since commit  
2af07e2f7 caused DefineIndex to restrict search_path internally, those  
lookups could fail or deliver different results than the original one.  
We can avoid those problems and save a few dozen lines of code by  
using generateClonedIndexStmt in this code path too.  
  
Another thing this fixes is incorrect propagation of parent-index  
comments to child indexes (because the copyObject approach copies  
the idxcomment field while generateClonedIndexStmt doesn't).  I had  
noticed this in connection with commit c01eb619a, but not run the  
problem to ground.  
  
I'm tempted to back-patch this further than v17, but the only thing  
it's known to fix in older branches is the comment issue, which is  
pretty minor and doesn't seem worth the risk of introducing new  
issues in stable branches.  (If anyone does care about that,  
clearing idxcomment in the copied IndexStmt would be a safer fix.)  
  
Per bug #18637 from usamoi.  Back-patch to v17 where the search_path  
change came in.  
  
Discussion: https://postgr.es/m/[email protected]  

M contrib/seg/Makefile
A contrib/seg/expected/partition.out
M contrib/seg/meson.build
A contrib/seg/sql/partition.sql
M src/backend/commands/indexcmds.c
M src/test/regress/expected/alter_table.out
M src/test/regress/sql/alter_table.sql

Reject non-ASCII locale names.

commit   : 9c7acc33307bf971ff5fe4b7f7431b7aa93052ac    
  
author   : Thomas Munro <[email protected]>    
date     : Sat, 5 Oct 2024 13:48:33 +1300    
  
committer: Thomas Munro <[email protected]>    
date     : Sat, 5 Oct 2024 13:48:33 +1300    

Click here for diff

Commit bf03cfd1 started scanning all available BCP 47 locale names on  
Windows.  This caused an abort/crash in the Windows runtime library if  
the default locale name contained non-ASCII characters, because of our  
use of the setlocale() save/restore pattern with "char" strings.  After  
switching to another locale with a different encoding, the saved name  
could no longer be understood, and setlocale() would abort.  
  
"Turkish_Türkiye.1254" is the example from recent reports, but there are  
other examples of countries and languages with non-ASCII characters in  
their names, and they appear in Windows' (old style) locale names.  
  
To defend against this:  
  
1.  In initdb, reject non-ASCII locale names given explicity on the  
command line, or returned by the operating system environment with  
setlocale(..., ""), or "canonicalized" by the operating system when we  
set it.  
  
2.  In initdb only, perform the save-and-restore with Windows'  
non-standard wchar_t variant of setlocale(), so that it is not subject  
to round trip failures stemming from char string encoding confusion.  
  
3.  In the backend, we don't have to worry about the save-and-restore  
problem because we have already vetted the defaults, so we just have to  
make sure that CREATE DATABASE also rejects non-ASCII names in any new  
databases.  SET lc_XXX doesn't suffer from the problem, but the ban  
applies to it too because it uses check_locale().  CREATE COLLATION  
doesn't suffer from the problem either, but it doesn't use  
check_locale() so it is not included in the new ban for now, to minimize  
the change.  
  
Anyone who encounters the new error message should either create a new  
duplicated locale with an ASCII-only name using Windows Locale Builder,  
or consider using BCP 47 names like "tr-TR".  Users already couldn't  
initialize a cluster with "Turkish_Türkiye.1254" on PostgreSQL 16+, but  
the new failure mode is an error message that explains why, instead of a  
crash.  
  
Back-patch to 16, where bf03cfd1 landed.  Older versions are affected  
in theory too, but only 16 and later are causing crash reports.  
  
Reviewed-by: Andrew Dunstan <[email protected]> (the idea, not the patch)  
Reported-by: Haifang Wang (Centific Technologies Inc) <[email protected]>  
Discussion: https://postgr.es/m/PH8PR21MB3902F334A3174C54058F792CE5182%40PH8PR21MB3902.namprd21.prod.outlook.com  

M src/backend/utils/adt/pg_locale.c
M src/bin/initdb/initdb.c

Fix wrong varnullingrels error for MERGE WHEN NOT MATCHED BY SOURCE.

commit   : 34ae54af92e006e601f808e58765cc979ca9135f    
  
author   : Dean Rasheed <[email protected]>    
date     : Thu, 3 Oct 2024 13:45:37 +0100    
  
committer: Dean Rasheed <[email protected]>    
date     : Thu, 3 Oct 2024 13:45:37 +0100    

Click here for diff

If a MERGE command contains WHEN NOT MATCHED BY SOURCE actions, the  
source relation appears on the outer side of the join. Thus, any Vars  
referring to the source in the merge join condition, actions, and  
RETURNING list should be marked as nullable by the join, since they  
are used in the ModifyTable node above the join. Note that this only  
applies to the copy of join condition used in the executor to  
distinguish MATCHED from NOT MATCHED BY SOURCE cases. Vars in the  
original join condition, inside the join node itself, should not be  
marked.  
  
Failure to correctly mark these Vars led to a "wrong varnullingrels"  
error in the final stage of query planning, in some circumstances. We  
happened to get away without this in all previous tests, since they  
all involved a ModifyTable node directly on top of the join node, so  
that the top plan targetlist coincided with the output of the join,  
and the varnullingrels check was more lax. However, if another plan  
node, such as a one-time filter Result node, gets inserted between the  
ModifyTable node and the join node, then a stricter check is applied,  
which fails.  
  
Per bug #18634 from Alexander Lakhin. Thanks to Tom Lane and Richard  
Guo for review and analysis.  
  
Back-patch to v17, where WHEN NOT MATCHED BY SOURCE support was added  
to MERGE.  
  
Discussion: https://postgr.es/m/18634-db5299c937877f2b%40postgresql.org  

M src/backend/optimizer/prep/prepjointree.c
M src/test/regress/expected/merge.out
M src/test/regress/sql/merge.sql

Fix incorrect non-strict join recheck in MERGE WHEN NOT MATCHED BY SOURCE.

commit   : d7d297f8449641bfd71750d04c302572a350052c    
  
author   : Dean Rasheed <[email protected]>    
date     : Thu, 3 Oct 2024 12:50:38 +0100    
  
committer: Dean Rasheed <[email protected]>    
date     : Thu, 3 Oct 2024 12:50:38 +0100    

Click here for diff

If a MERGE command contains WHEN NOT MATCHED BY SOURCE actions, the  
merge join condition is used by the executor to distinguish MATCHED  
from NOT MATCHED BY SOURCE cases. However, this qual is executed using  
the output from the join subplan node, which nulls the output from the  
source relation in the not matched case, and so the result may be  
incorrect if the join condition is "non-strict" -- for example,  
something like "src.col IS NOT DISTINCT FROM tgt.col".  
  
Fix this by enhancing the join recheck condition with an additional  
"src IS NOT NULL" check, so that it does the right thing when  
evaluated using the output from the join subplan.  
  
Noted by Tom Lane while investigating bug #18634 from Alexander  
Lakhin.  
  
Back-patch to v17, where WHEN NOT MATCHED BY SOURCE support was added  
to MERGE.  
  
Discussion: https://postgr.es/m/18634-db5299c937877f2b%40postgresql.org  

M src/backend/optimizer/prep/prepjointree.c
M src/test/regress/expected/merge.out
M src/test/regress/sql/merge.sql

Parse libpq's "keepalives" option more like other integer options.

commit   : c7a201053e3f7d322782e57879b43d1fd8b7bf32    
  
author   : Tom Lane <[email protected]>    
date     : Wed, 2 Oct 2024 17:30:36 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Wed, 2 Oct 2024 17:30:36 -0400    

Click here for diff

Use pqParseIntParam (nee parse_int_param) instead of using strtol  
directly.  This allows trailing whitespace, which the previous coding  
didn't, and makes the spelling of the error message consistent with  
other similar cases.  
  
This seems to be an oversight in commit e7a221797, which introduced  
parse_int_param.  That fixed places that were using atoi(), but missed  
this place which was randomly using strtol() instead.  
  
Ordinarily I'd consider this minor cleanup not worth back-patching.  
However, it seems that ecpg assumes it can add trailing whitespace  
to URL parameters, so that use of the keepalives option fails in  
that context.  Perhaps that's worth improving as a separate matter.  
In the meantime, back-patch this to all supported branches.  
  
Yuto Sasaki (some further cleanup by me)  
  
Discussion: https://postgr.es/m/TY2PR01MB36286A7B97B9A15793335D18C1772@TY2PR01MB3628.jpnprd01.prod.outlook.com  

M src/interfaces/libpq/fe-connect.c

doc: Clarify name of files generated by pg_waldump --save-fullpage

commit   : 97dccefc3651902f3dfad504a39be9a896279bac    
  
author   : Michael Paquier <[email protected]>    
date     : Wed, 2 Oct 2024 11:12:45 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Wed, 2 Oct 2024 11:12:45 +0900    

Click here for diff

The fork name is always separated with the block number by an underscore  
in the names of the files generated, but the docs stuck them together  
without a separator, which was confusing.  
  
Author: Christoph Berg  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 16  

M doc/src/sgml/ref/pg_waldump.sgml

Doc: replace unnecessary non-breaking space with ordinal space.

commit   : a94d5b3728dd3e5fd4adb25350712eb785bb9ab6    
  
author   : Tatsuo Ishii <[email protected]>    
date     : Tue, 1 Oct 2024 11:35:42 +0900    
  
committer: Tatsuo Ishii <[email protected]>    
date     : Tue, 1 Oct 2024 11:35:42 +0900    

Click here for diff

There were unnecessary non-breaking spaces (nbsp, U+00A0, 0xc2a0 in  
UTF-8) in the docs.  This commit replaces them with ASCII spaces  
(0x20).  
  
config.sgml is backpatched through 17.  
ref/drop_extension.sgml is backpatched through 13.  
  
Discussion: https://postgr.es/m/20240930.153404.202479334310259810.ishii%40postgresql.org  
Reviewed-by: Yugo Nagata, Daniel Gustafsson  
Backpatch-through: 17, 13  

M doc/src/sgml/config.sgml
M doc/src/sgml/ref/drop_extension.sgml

Fix race condition in COMMIT PREPARED causing orphaned 2PC files

commit   : f250cb29d908343254689c1b1721c287efffc10b    
  
author   : Michael Paquier <[email protected]>    
date     : Tue, 1 Oct 2024 15:44:07 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Tue, 1 Oct 2024 15:44:07 +0900    

Click here for diff

COMMIT PREPARED removes on-disk 2PC files near its end, but the state  
checked if a file is on-disk or not gets read from shared memory while  
not holding the two-phase state lock.  
  
Because of that, there was a small window where a second backend doing a  
PREPARE TRANSACTION could reuse the GlobalTransaction put back into the  
2PC free list by the COMMIT PREPARED, overwriting the "ondisk" flag read  
afterwards by the COMMIT PREPARED to decide if its on-disk two-phase  
state file should be removed, preventing the file deletion.  
  
This commit fixes this issue so as the "ondisk" flag in the  
GlobalTransaction is read while holding the two-phase state lock, not  
from shared memory after its entry has been added to the free list.  
  
Orphaned two-phase state files flushed to disk after a checkpoint are  
discarded at the beginning of recovery.  However, a truncation of  
pg_xact/ would make the startup process issue a FATAL when it cannot  
read the SLRU page holding the state of the transaction whose 2PC file  
was orphaned, which is a necessary step to decide if the 2PC file should  
be removed or not.  Removing manually the file would be necessary in  
this case.  
  
Issue introduced by effe7d9552dd, so backpatch all the way down.  
  
Mea culpa.  
  
Author: wuchengwen  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 12  

M src/backend/access/transam/twophase.c

Remove incorrect entries in pg_walsummary's getopt_long call.

commit   : 6596a8c8603d31a21d9e21bc342221e0e698fe5e    
  
author   : Tom Lane <[email protected]>    
date     : Mon, 30 Sep 2024 12:06:54 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Mon, 30 Sep 2024 12:06:54 -0400    

Click here for diff

For some reason this listed "-f" and "-w" as valid switches, though  
the code doesn't implement any such thing nor do the docs mention  
them.  The effect of this was that if you tried to use one of these  
switches, you'd get an unhelpful error message.  
  
Yusuke Sugie  
  
Discussion: https://postgr.es/m/[email protected]  

M src/bin/pg_walsummary/pg_walsummary.c

reindexdb: Skip reindexing temporary tables and indexes.

commit   : 77f1546819818aa740fd8e54427ccb33f507f2cf    
  
author   : Fujii Masao <[email protected]>    
date     : Mon, 30 Sep 2024 11:13:55 +0900    
  
committer: Fujii Masao <[email protected]>    
date     : Mon, 30 Sep 2024 11:13:55 +0900    

Click here for diff

Reindexing temp tables or indexes of other sessions is not allowed.  
However, reindexdb in parallel mode previously listed them as  
the objects to process, leading to failures.  
  
This commit ensures reindexdb in parallel mode skips temporary tables  
and indexes by adding a condition based on the relpersistence column  
in pg_class to the object listing queries, preventing these issues.  
  
Note that this commit does not affect reindexdb when temporary tables  
or indexes are explicitly specified using the -t or -j options;  
reindexdb in that case still does not skip them and can cause an error.  
  
Back-patch to v13 where parallel mode was introduced in reindexdb.  
  
Author: Fujii Masao  
Reviewed-by: Michael Paquier  
Discussion: https://postgr.es/m/[email protected]  

M src/bin/scripts/reindexdb.c

Remove NULL dereference from RenameRelationInternal().

commit   : da99df15c25d9c769f24a721f34ed062d0b5d4d2    
  
author   : Noah Misch <[email protected]>    
date     : Sun, 29 Sep 2024 15:54:25 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Sun, 29 Sep 2024 15:54:25 -0700    

Click here for diff

Defect in last week's commit aac2c9b4fde889d13f859c233c2523345e72d32b,  
per Coverity.  Reaching this would need catalog corruption.  Back-patch  
to v12, like that commit.  

M src/backend/commands/tablecmds.c

Avoid 037_invalid_database.pl hang under debug_discard_caches.

commit   : 4aad471688993d3f6cfb8afa8b3af52b2fbdc7a3    
  
author   : Noah Misch <[email protected]>    
date     : Fri, 27 Sep 2024 15:28:56 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Fri, 27 Sep 2024 15:28:56 -0700    

Click here for diff

Back-patch to v12 (all supported versions).  

M src/test/recovery/t/037_invalid_database.pl

doc: Note that CREATE MATERIALIZED VIEW restricts search_path.

commit   : 18cea252ac74c185f754b3f0007921e7955ed1d6    
  
author   : Nathan Bossart <[email protected]>    
date     : Fri, 27 Sep 2024 16:21:21 -0500    
  
committer: Nathan Bossart <[email protected]>    
date     : Fri, 27 Sep 2024 16:21:21 -0500    

Click here for diff

Since v17, CREATE MATERIALIZED VIEW has set search_path to  
"pg_catalog, pg_temp" while running the query.  The docs for the  
other commands that restrict search_path mention it, but the page  
for CREATE MATERIALIZED VIEW does not.  Fix that.  
  
Oversight in commit 4b74ebf726.  
  
Author: Yugo Nagata  
Reviewed-by: Jeff Davis  
Discussion: https://postgr.es/m/20240805160502.d2a4975802a832b1e04afb80%40sraoss.co.jp  
Backpatch-through: 17  

M doc/src/sgml/ref/create_materialized_view.sgml

Fix incorrect memory access in VACUUM FULL with invalid toast indexes

commit   : 1532599a854a7bf0e102d708a288ff96d2722218    
  
author   : Michael Paquier <[email protected]>    
date     : Fri, 27 Sep 2024 09:40:14 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Fri, 27 Sep 2024 09:40:14 +0900    

Click here for diff

An invalid toast index is skipped in reindex_relation().  These would be  
remnants of a failed REINDEX CONCURRENTLY and they should never been  
rebuilt as there can only be one valid toast index at a time.  
  
REINDEX_REL_SUPPRESS_INDEX_USE, used by CLUSTER and VACUUM FULL, needs  
to maintain a list of the indexes being processed.  The list of indexes  
is retrieved from the relation cache, and includes invalid indexes.  The  
code has missed that invalid toast indexes are ignored in  
reindex_relation() as this leads to a hard failure in reindex_index(),  
and they were left in the reindex pending list, making the list  
inconsistent when rechecked.  The incorrect memory access was happening  
when scanning pg_class for the refresh of pg_database.datfrozenxid, when  
doing a scan of pg_class.  
  
This issue exists since REINDEX CONCURRENTLY exists, where invalid toast  
indexes can exist, so backpatch all the way down.  
  
Reported-by: Alexander Lakhin  
Author: Tender Wang  
Discussion: https://postgr.es/m/[email protected]  
Backpatch-through: 12  

M src/backend/catalog/index.c

Doc: InitPlans aren't parallel-restricted any more.

commit   : 3e8c92c956a1f273f7cb324aa3ad07f0fca44ce9    
  
author   : Tom Lane <[email protected]>    
date     : Thu, 26 Sep 2024 10:37:51 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Thu, 26 Sep 2024 10:37:51 -0400    

Click here for diff

Commit e08d74ca1 removed that restriction, but missed updating  
the documentation about it.  Noted by Egor Rogov.  
  
Discussion: https://postgr.es/m/[email protected]  

M doc/src/sgml/parallel.sgml

Remove extra whitespace in pg_upgrade status message.

commit   : 4e0864af16b5ada4b251fe2dee1060bae81628bf    
  
author   : Nathan Bossart <[email protected]>    
date     : Wed, 25 Sep 2024 11:18:56 -0500    
  
committer: Nathan Bossart <[email protected]>    
date     : Wed, 25 Sep 2024 11:18:56 -0500    

Click here for diff

There's no need to add another level of indentation to this status  
message.  pg_log() will put it in the right place.  
  
Oversight in commit 347758b120.  
  
Reviewed-by: Daniel Gustafsson  
Discussion: https://postgr.es/m/ZunW7XHLd2uTts4f%40nathan  
Backpatch-through: 17  

M src/bin/pg_upgrade/check.c

vacuumdb: Skip temporary tables in query to build list of relations

commit   : 85cb21df673f3a0545b21eadcced7cb038c9fe9c    
  
author   : Michael Paquier <[email protected]>    
date     : Wed, 25 Sep 2024 14:44:50 +0900    
  
committer: Michael Paquier <[email protected]>    
date     : Wed, 25 Sep 2024 14:44:50 +0900    

Click here for diff

Running vacuumdb with a non-superuser while another user has created a  
temporary table would lead to a mid-flight permission failure,  
interrupting the operation.  vacuum_rel() skips temporary relations of  
other backends, and it makes no sense for vacuumdb to know about these  
relations, so let's switch it to ignore temporary relations entirely.  
  
Adding a qual in the query based on relpersistence simplifies the  
generation of its WHERE clause in vacuum_one_database(), per se the  
removal of "has_where".  
  
Author: VaibhaveS, Michael Paquier  
Reviewed-by: Fujii Masao  
Discussion: https://postgr.es/m/CAM_eQjwfAR=y3G1fGyS1U9FTmc+FyJm9amNfY2QCZBnDDbNPZg@mail.gmail.com  
Backpatch-through: 12  

M src/bin/scripts/vacuumdb.c

For inplace update durability, make heap_update() callers wait.

commit   : 3b7a689e1a805c4dac2f35ff14fd5c9fdbddf150    
  
author   : Noah Misch <[email protected]>    
date     : Tue, 24 Sep 2024 15:25:18 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Tue, 24 Sep 2024 15:25:18 -0700    

Click here for diff

The previous commit fixed some ways of losing an inplace update.  It  
remained possible to lose one when a backend working toward a  
heap_update() copied a tuple into memory just before inplace update of  
that tuple.  In catalogs eligible for inplace update, use LOCKTAG_TUPLE  
to govern admission to the steps of copying an old tuple, modifying it,  
and issuing heap_update().  This includes MERGE commands.  To avoid  
changing most of the pg_class DDL, don't require LOCKTAG_TUPLE when  
holding a relation lock sufficient to exclude inplace updaters.  
Back-patch to v12 (all supported versions).  In v13 and v12, "UPDATE  
pg_class" or "UPDATE pg_database" can still lose an inplace update.  The  
v14+ UPDATE fix needs commit 86dc90056dfdbd9d1b891718d2e5614e3e432f35,  
and it wasn't worth reimplementing that fix without such infrastructure.  
  
Reviewed by Nitin Motiani and (in earlier versions) Heikki Linnakangas.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/access/heap/README.tuplock
M src/backend/access/heap/heapam.c
M src/backend/access/index/genam.c
M src/backend/catalog/aclchk.c
M src/backend/catalog/catalog.c
M src/backend/commands/dbcommands.c
M src/backend/commands/event_trigger.c
M src/backend/commands/indexcmds.c
M src/backend/commands/tablecmds.c
M src/backend/executor/execMain.c
M src/backend/executor/execReplication.c
M src/backend/executor/nodeModifyTable.c
M src/backend/utils/cache/relcache.c
M src/backend/utils/cache/syscache.c
M src/include/nodes/execnodes.h
M src/include/storage/lockdefs.h
M src/include/utils/syscache.h
M src/test/isolation/expected/intra-grant-inplace.out
M src/test/isolation/specs/eval-plan-qual.spec
M src/test/isolation/specs/intra-grant-inplace.spec

Fix data loss at inplace update after heap_update().

commit   : fd27b878c2ea54b6132758084270cb13e5e66f2e    
  
author   : Noah Misch <[email protected]>    
date     : Tue, 24 Sep 2024 15:25:18 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Tue, 24 Sep 2024 15:25:18 -0700    

Click here for diff

As previously-added tests demonstrated, heap_inplace_update() could  
instead update an unrelated tuple of the same catalog.  It could lose  
the update.  Losing relhasindex=t was a source of index corruption.  
Inplace-updating commands like VACUUM will now wait for heap_update()  
commands like GRANT TABLE and GRANT DATABASE.  That isn't ideal, but a  
long-running GRANT already hurts VACUUM progress more just by keeping an  
XID running.  The VACUUM will behave like a DELETE or UPDATE waiting for  
the uncommitted change.  
  
For implementation details, start at the systable_inplace_update_begin()  
header comment and README.tuplock.  Back-patch to v12 (all supported  
versions).  In back branches, retain a deprecated heap_inplace_update(),  
for extensions.  
  
Reported by Smolkin Grigory.  Reviewed by Nitin Motiani, (in earlier  
versions) Heikki Linnakangas, and (in earlier versions) Alexander  
Lakhin.  
  
Discussion: https://postgr.es/m/CAMp+ueZQz3yDk7qg42hk6-9gxniYbp-=bG2mgqecErqR5gGGOA@mail.gmail.com  

M src/backend/access/heap/README.tuplock
M src/backend/access/heap/heapam.c
M src/backend/access/index/genam.c
M src/backend/catalog/index.c
M src/backend/catalog/toasting.c
M src/backend/commands/dbcommands.c
M src/backend/commands/event_trigger.c
M src/backend/commands/vacuum.c
M src/include/access/genam.h
M src/include/access/heapam.h
M src/test/isolation/expected/intra-grant-inplace-db.out
M src/test/isolation/expected/intra-grant-inplace.out
M src/test/isolation/specs/intra-grant-inplace-db.spec
M src/test/isolation/specs/intra-grant-inplace.spec
M src/test/modules/injection_points/expected/inplace.out
M src/test/modules/injection_points/specs/inplace.spec

Warn if LOCKTAG_TUPLE is held at commit, under debug_assertions.

commit   : f33bf1c7d501af56b257256349990ad9000dcffa    
  
author   : Noah Misch <[email protected]>    
date     : Tue, 24 Sep 2024 15:25:18 -0700    
  
committer: Noah Misch <[email protected]>    
date     : Tue, 24 Sep 2024 15:25:18 -0700    

Click here for diff

The current use always releases this locktag.  A planned use will  
continue that intent.  It will involve more areas of code, making unlock  
omissions easier.  Warn under debug_assertions, like we do for various  
resource leaks.  Back-patch to v12 (all supported versions), the plan  
for the commit of the new use.  
  
Reviewed by Heikki Linnakangas.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/backend/storage/lmgr/lock.c

Fix psql describe commands' handling of ACL columns for old servers.

commit   : 923a71584fd7efb5302cb8bf5a5bd417b531123f    
  
author   : Tom Lane <[email protected]>    
date     : Tue, 24 Sep 2024 17:21:38 -0400    
  
committer: Tom Lane <[email protected]>    
date     : Tue, 24 Sep 2024 17:21:38 -0400    

Click here for diff

Commit d1379ebf4 carelessly broke printACLColumn for pre-9.4 servers,  
by using the cardinality() function which we introduced in 9.4.  
We expect psql's describe-related commands to work back to 9.2, so  
this is bad.  Use the longstanding array_length() function instead.  
  
Per report from Christoph Berg.  Back-patch to v17.  
  
Discussion: https://postgr.es/m/[email protected]  

M src/bin/psql/describe.c