Skip to content

gh-148286: Fix undefined behaviour in per-thread refcount merging - #154915

Open
matthiasgoergens wants to merge 1 commit into
python:mainfrom
matthiasgoergens:freethreaded-refcount-ub
Open

gh-148286: Fix undefined behaviour in per-thread refcount merging#154915
matthiasgoergens wants to merge 1 commit into
python:mainfrom
matthiasgoergens:freethreaded-refcount-ub

Conversation

@matthiasgoergens

@matthiasgoergens matthiasgoergens commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

_PyObject_MergePerThreadRefcounts() merges each thread's local reference count deltas into the shared count:

_Py_atomic_add_ssize(&obj->ob_ref_shared, refcnt << _Py_REF_SHARED_SHIFT);

refcnt is a delta, so it is routinely negative — in practice almost always -1 — and shifting a negative value left is undefined behaviour. This does the shift in the unsigned domain and converts back.

This is not reachable from the UBSan CI job today. The sanitizer matrix in .github/workflows/build.yml pairs UBSan only with free-threading: false, so Python/uniqueid.c is never built under UBSan at all. Building --disable-gil together with --with-undefined-behavior-sanitizer shows how load-bearing it is: the first line of output from ./python -c pass is the UBSan report, raised during interpreter startup via _PyImport_InitExternal, and it recurs from gc_collect_main on every collection.

Measured over the full test suite on that configuration, with every entry in Tools/ubsan/suppressions.txt disabled:

UB reports test failures failing files
before 1762 529 61
after 0 0 0

Most of those failures are tests asserting that a subprocess produced no stderr, which the UBSan diagnostic breaks.

I have not touched the CI matrix here, since adding a job is a maintainer's call and this fix stands on its own. But the reason this went unnoticed is structural rather than accidental, and I would like to follow up separately with a proposal to add free-threading: true to the UBSan matrix — with this fix in, that configuration is green, so it would be a regression guard rather than a source of new work. The same appears to be true of --enable-experimental-jit, which CI builds but never sanitizes: I ran the full suite against a machine-code JIT build under UBSan and it is clean today, while gh-139269 (an unaligned uint64_t store in Python/jit.c, which segfaulted release builds) was originally found by exactly that combination. I will write that up with the measurements rather than expand the scope of this PR.

…ng (free-threaded)

_PyObject_MergePerThreadRefcounts() merges each thread's local reference
count deltas into the shared count:

    _Py_atomic_add_ssize(&obj->ob_ref_shared,
                         refcnt << _Py_REF_SHARED_SHIFT);

`refcnt` is a delta, so it is routinely negative -- the observed value is
almost always -1 -- and shifting a negative value left is undefined
behaviour. Do the shift in the unsigned domain and convert back.

This is not reachable from the UBSan CI job today: the sanitizer matrix in
.github/workflows/build.yml pairs UBSan only with free-threading: false, so
this file is never built under UBSan. Building --disable-gil with
--with-undefined-behavior-sanitizer shows how load-bearing it is; the very
first line of output from `./python -c pass` is the UBSan report, raised
from interpreter startup via _PyImport_InitExternal, and it recurs from
gc_collect_main on every collection.

Measured over the full test suite on that configuration, with all entries
in Tools/ubsan/suppressions.txt disabled:

  before:  1762 UB reports, 529 test failures across 61 test files
  after:      0 UB reports, 0 failures, suite green (run=51,517)

Most of those failures are tests that assert a subprocess produced no
stderr, which the UBSan diagnostic breaks.

Adding free-threading: true to the UBSan matrix would keep this fixed; that
is left as a separate decision since it costs a CI job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant