Skip to content

Make dict traversal safe on free-threaded Python - #144

Open
divyanshu-iitian wants to merge 1 commit into
google-deepmind:masterfrom
divyanshu-iitian:fix/free-threaded-dict-keys
Open

divyanshu-iitian wants to merge 1 commit into
google-deepmind:masterfrom
divyanshu-iitian:fix/free-threaded-dict-keys

Conversation

@divyanshu-iitian

Copy link
Copy Markdown

Summary

  • replace borrowed dictionary values with strong references, using PyDict_GetItemRef on Python 3.13+ and a compatible fallback on older versions
  • snapshot dictionary keys before lookups that may invoke Python, avoiding PyDict_Next across concurrent mutation
  • add a free-threaded regression stress test matching the reported comparer/mutator workload

Fixes #143

Testing

  • CPython 3.11: 70 passed, 1 skipped
  • CPython 3.14t (free-threaded): 71 passed

@espressolee

Copy link
Copy Markdown

Thanks @divyanshu-iitian for tracking this down and fixing it. I reported #143, and independently verified this PR at its current head 5fe2725 against base 9330c95. I could no longer reproduce the three race failures I described there.

Built and run on macOS arm64 across seven environments: CPython 3.12.11, 3.13.5, 3.13.5t, 3.14.6 and 3.14.0rc1t for the candidate, plus 3.14.6 and 3.14.0rc1t for the base.

Concurrent arms, fresh subprocesses throughout:

arm processes clean faults
base, free-threaded, mutating 60 0 28 SIGSEGV, 32 SIGTRAP
candidate 3.14t, same mutation 60 60 —
candidate 3.13t, minimum branch 30 30 —
controls 180 180 —

Three arms drive the base row: the original value-replacement assertion race, key deletion and reinsertion, and the DictValueIterator flatten value-fetch. The 180 controls are the no-mutator and decoy-dict variants of each for both base and candidate, plus the same mutation with the GIL enabled. All were clean, supporting that the base failures depend on concurrent free-threaded mutation rather than on the harness alone.

Across those seven build/environment runs I observed 495 unittest cases and 350 doctest examples in total, with 0 failures and 3 expected skips.

After this lands, I'm happy to contribute additional concurrent regression coverage for the modified DictValueIterator/flatten path.

Scope: local macOS arm64 only. This does not establish Windows or Linux behaviour, other architectures, or concurrent semantics beyond the three named races.

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The borrowed-reference hazard is fixed, but there is still a size/snapshot race: the sizes can match, then o1 can lose a key before PyDict_Keys(o1); every key in that smaller snapshot may still exist in o2, so this can report equal structures while o2 has an extra key. Could the snapshot length be rechecked against o2, or both key sets be snapshotted, so concurrent mutation fails rather than yielding a false equality?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Free-threaded (no-GIL) use-after-free: borrowed dict key from PyDict_Next used across __hash__/__eq__ in assert_same_structure

3 participants