Skip to content

[Python][C++] 25.0.0 aarch64 wheel: deterministic dictionary-decode failure on Neoverse V1 / AWS Graviton3 (SIGSEGV via iter_batches; ArrowInvalid single-threaded); N1/V2 unaffected; 24.0.0 OK #50503

Closed
Closed
[Python][C++] 25.0.0 aarch64 wheel: deterministic dictionary-decode failure on Neoverse V1 / AWS Graviton3 (SIGSEGV via iter_batches; ArrowInvalid single-threaded); N1/V2 unaffected; 24.0.0 OK#50503
Assignees
AntoinePrv
Labels
Component: C++Component: Pythonbackport-candidate
Milestone
25.0.1

Description

@eaglenebulaplayer
eaglenebulaplayer
opened on Jul 14, 2026

Describe the bug, including details regarding any error messages, version, and platform.

Summary: the pyarrow 25.0.0 manylinux aarch64 wheel deterministically fails to decode dictionary-encoded Parquet on AWS Graviton3 (Neoverse V1, CPU part 0xd40) hosts. The same files, same container image, same code path work correctly on Graviton2 (Neoverse N1, 0xd0c) and Graviton4 (Neoverse V2, 0xd4f), and everything works on all three CPU generations with pyarrow 24.0.0.

Symptoms on Graviton3 + 25.0.0:

  • ParquetFile.iter_batches() with default use_threads=True → hard SIGSEGV (faulthandler shows the crash inside native decode).
  • use_threads=False → no crash, but raises ArrowInvalid: Index not in dictionary bounds — i.e. single-threaded decode is also wrong, so this is not a threading race and use_threads=False is not a mitigation.

Determinism evidence (from an internal investigation; counts are per 8 identical scan tasks on a fixed corpus per arm, tasks landing on random ARM host generations on AWS Fargate):

  • pyarrow 25.0.0, defaults: 3/8 tasks failed — exactly the tasks placed on 0xd40 hosts.
  • pyarrow 25.0.0, use_threads=False: 4/8 failed with ArrowInvalid — again exactly the 0xd40 placements.
  • Same base image with only pyarrow==24.0.0 swapped in: 8/8 green across all host generations.
  • Within a failing task: re-decoding the exact same bytes twice fails identically; re-downloading the file (sha256-verified identical) fails identically; the same file decoded on N1/V2 hosts or on x86-64 is byte-for-byte correct. 12 probe tasks reading /proc/cpuinfo: failure correlates 100% with CPU part 0xd40.
  • The failing wheel was installed from PyPI (pyarrow==25.0.0, cp312 manylinux aarch64) in a python:3.12-slim (Debian, glibc 2.41) container.

Files that trigger it: zstd-compressed Parquet written by earlier pyarrow (write_table, defaults → dictionary-encoded string columns), ~500k rows per file, ~27 columns (strings, int64, date32, bool, one timestamp). Unfortunately the data is proprietary financial content and we cannot attach a triggering file. We have not yet managed to isolate a synthetic reproducer (we currently have no pinned-Graviton3 test host — Fargate placement is random), but the failure is 100% deterministic per (file, 0xd40 host, 25.0.0).

Hypothesis: a Neoverse-V1-specific code path in the 25.0.0 aarch64 build — V1 is the only one of the three affected/unaffected cores with 256-bit SVE (N1 has no SVE, V2 is 128-bit SVE2), so an SVE(-width)-dispatched kernel in the dictionary decode path is our leading suspect. Related plumbing: #47769 (dynamic dispatch on ARM64) / xsimd bump #49921.

Repro shape (on a Graviton3 host, e.g. EC2 c7g/m7g/r7g, with any sufficiently large dictionary-encoded zstd Parquet):

import pyarrow.parquet as pq
pf = pq.ParquetFile(path)
for batch in pf.iter_batches(batch_size=5000):      # SIGSEGV on 0xd40 with 25.0.0
    pass
# use_threads=False instead → ArrowInvalid: Index not in dictionary bounds

We are happy to run candidate wheels/builds or additional diagnostics (faulthandler tracebacks, ARROW_USER_SIMD_LEVEL sweeps, etc.) on Graviton3 and report back — please tell us what would be most useful.

Component(s)

Python, C++

Activity

  1. github-actions
    added
    Component: C++
    Component: Python
    on Jul 14, 2026

  2. conzy commented on Jul 14, 2026

    @conzy
    conzy
    on Jul 14, 2026

    Also experiencing this on Fargate

  3. wudidapaopao
    added a commit that references this issue on Jul 15, 2026

    DEBUG: disable regular CI, add Graviton3 SVE-256 repro workflow

    bab4586

  4. vfilter commented on Jul 16, 2026

    @vfilter
    vfilter
    on Jul 16, 2026

    We had this happen to us on Gravitron 3 Lambda instances. I can share an affected Parquet file here. Symptom in this case were absolutely mysterious duplicate datetime indices when loading in Python.

    d72579f558c154379de39b75272dc612588fda4b.parquet.zip

  5. cyb70289 commented on Jul 17, 2026

    @cyb70289
    cyb70289
    on Jul 17, 2026
    Contributor

    Reproduced this issue on Graviton 3, with a parquet file generated by LLM based on @eaglenebulaplayer 's description.
    gen-test-parquet.py

    pyarrow 25.0.0 release (pip installed) crashed with "SIGSEGV on 0xd40".
    I built arrow from tip, it didn't crash (maybe due to different build args, not sure). Still debugging.

  6. raulcd commented on Jul 20, 2026

    @raulcd
    raulcd
    on Jul 20, 2026
    Member

    @cyb70289 where you able to find anything? Building from tip not being able to reproduce is interesting? Did you use the nightly wheels or did you build pyarrow yourself? You can use the latest nightly wheels from:

    pip install \
 -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
 pyarrow

  7. Klavinco commented on Jul 20, 2026

    @Klavinco
    Klavinco
    on Jul 20, 2026

    Additional data points + a public 6-line reproducer (independent hit of what looks like the same bug)

    Environment: AWS c7g.large (Graviton3 / Neoverse V1; sve svebf16 svei8mm in cpuinfo), Ubuntu 24.04 aarch64, kernel 6.17.0-1019-aws, CPython 3.13, manylinux aarch64 wheel pyarrow==25.0.0. The same versions on Apple Silicon macOS (NEON-only arm64) are not affected — consistent with the SVE-dispatch suspicion.

    1. Public minimal repro — no proprietary data needed. A single dictionary-encoded int64 column with 300 distinct values comes back with silently duplicated values (no crash, no error):

    import pyarrow as pa, pyarrow.parquet as pq
vals = list(range(100_000, 100_300))
pq.write_table(pa.table({"x": pa.array(vals, pa.int64())}), "/tmp/min.parquet")
back = pq.read_table("/tmp/min.parquet")["x"].to_pylist()
print("match:", back == vals, "| unique:", len(set(back)), "of", len(back))
# Graviton3 + pyarrow 25.0.0:            match: False | unique: 84 of 300
# same + ARROW_USER_SIMD_LEVEL=NONE:     match: True  | unique: 300 of 300
# same box + pyarrow 24.0.0:             match: True  | unique: 300 of 300

    Unlike the SIGSEGV / "Index not in dictionary bounds" reported above, on our data the flat-column failure mode is silent wrong data (values repeated from earlier in the page), fully deterministic across runs, identical with use_threads=False — so the manifestation seems data-shape-dependent, which makes the silent case the scarier one.

    2. ARROW_USER_SIMD_LEVEL=NONE fully fixes every read path on this box (the sweep offered above) — pointing at a runtime-dispatched SIMD kernel.

    3. Not codec-related. Re-encoding the same data as zstd, snappy, and uncompressed gives identical corruption; writing with use_dictionary=False avoids it for flat columns → the corrupt path is the RLE/bit-packed dictionary-index decode.

    4. Repetition/definition levels are also affected (same RLE/bit-packed decoder): reading tables that contain a list<struct<decimal128, decimal128>> column on this box raises e.g. OSError: Malformed levels. min: 0 max: 21827 out of range. Max Level: 1 or ArrowInvalid: Column 7 named asks expected length 218 but got length 239 (the message varies between layouts), even for files written with use_dictionary=False. So the blast radius is wider than dictionary decode.

    5. Write path appears unaffected: parquet written by 25.0.0 on the same box is read back correctly by polars 1.42.1 and by pyarrow 24.0.0 (update-id chains over millions of rows verify clean).

    Happy to run candidate wheels or further env sweeps on this box.

  8. cyb70289 commented on Jul 21, 2026

    @cyb70289
    cyb70289
    on Jul 21, 2026
    Contributor

    @cyb70289 where you able to find anything? Building from tip not being able to reproduce is interesting? Did you use the nightly wheels or did you build pyarrow yourself? You can use the latest nightly wheels from:

    pip install \
 -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
 pyarrow

    Nightly build also failed. @raulcd , do you know how to build the release binaries? My manual build probably have different build options, cannot reproduce the issue. Needs binary with debug info.

  9. cyb70289 commented on Jul 21, 2026

    @cyb70289
    cyb70289
    on Jul 21, 2026
    Contributor

    @AntoinePrv , do you think there might be some edge cases of SIMD dispatcher that might break on SVE-256 machine?

  10. raulcd commented on Jul 21, 2026

    @raulcd
    raulcd
    on Jul 21, 2026
    Member

    do you know how to build the release binaries?

    Yes, currently the release binaries are built using archery (docker). From this latest nightly workflow the command to reproduce wheels for 3.14:

    $ ARCHERY_DEBUG=1 ARCH=arm64/v8 PYTHON=3.14 PYTHON_ABI_TAG=cp314 PYTHON_IMAGE_TAG=3.14 archery docker run -e SETUPTOOLS_SCM_PRETEND_VERSION=26.0.0.dev16 python-wheel-manylinux-2-28

    This Dockerfile is the one that gets run which manages third party dependencies via vcpkg:
    https://github.com/apache/arrow/blob/0f54c056e6d3576344971121708fc93deeaa3918/ci/docker/python-wheel-manylinux.dockerfile

    And the build script for the wheel itself is here:
    https://github.com/apache/arrow/blob/0f54c056e6d3576344971121708fc93deeaa3918/ci/scripts/python_wheel_xlinux_build.sh

  11. AntoinePrv commented on Jul 21, 2026

    @AntoinePrv
    AntoinePrv
    on Jul 21, 2026
    Collaborator

    Gonna investigate!

  12. AntoinePrv commented on Jul 21, 2026

    @AntoinePrv
    AntoinePrv
    on Jul 21, 2026
    Collaborator

    I could not reproduce, even with the pip installed pyarrow 25.0.0 on with m7gd.xlarge (Graviton 3 / Neoverse V1: 6.12.94+deb13-cloud-arm64, glibc 2.41). I do find this surprising, I wonder if I messed up something.

  13. cyb70289 commented on Jul 22, 2026

    @cyb70289
    cyb70289
    on Jul 22, 2026
    Contributor

    This is strange. I can reproduce in on a m7gd.xlarge instance, pyarrow 25.0.0, ubuntu-24.04, 6.17.0-1017-aws, also on debian-13 with 6.12.94+deb13-cloud-arm64 kernel.

    Using @Klavinco 's test code:

    import pyarrow as pa, pyarrow.parquet as pq
vals = list(range(100_000, 100_300))
pq.write_table(pa.table({"x": pa.array(vals, pa.int64())}), "/tmp/min.parquet")
back = pq.read_table("/tmp/min.parquet")["x"].to_pylist()
print("match:", back == vals, "| unique:", len(set(back)), "of", len(back))
# Graviton3 + pyarrow 25.0.0:            match: False | unique: 84 of 300
# same + ARROW_USER_SIMD_LEVEL=NONE:     match: True  | unique: 300 of 300
# same box + pyarrow 24.0.0:             match: True  | unique: 300 of 300

    Surprisingly, ARROW_USER_SIMD_LEVEL=SVE128 passes the test.

    $ pip install pyarrow==25.0.0

# fail
$ python test.py
match: False | unique: 84 of 300

# fail
$ ARROW_USER_SIMD_LEVEL=SVE256 python test.py
match: False | unique: 84 of 300

# pass
$ ARROW_USER_SIMD_LEVEL=NONE python test.py
match: True | unique: 300 of 300

# pass
$ ARROW_USER_SIMD_LEVEL=SVE128 python test.py
match: True | unique: 300 of 300

  14. AntoinePrv commented on Jul 22, 2026

    @AntoinePrv
    AntoinePrv
    on Jul 22, 2026
    Collaborator

    This is strange. I can reproduce in on a m7gd.xlarge instance, pyarrow 25.0.0, ubuntu-24.04, 6.17.0-1017-aws, also on debian-13 with 6.12.94+deb13-cloud-arm64 kernel.

    Ha because I think I was focused on reproducing the segfaults, so I may have read the print statement too fast.

    Surprisingly, ARROW_USER_SIMD_LEVEL=SVE128 passes the test.

    I guess because there is no fallback SVE256 to SVE128 (on purpose) so this setting will actually run Neon.

  15. 5 remaining items

  16. AntoinePrv commented on Jul 23, 2026

    @AntoinePrv
    AntoinePrv
    on Jul 23, 2026
    Collaborator

    I think you may need LTO enabled to reproduce.

    In addition, would some of you mind double checking this wheel solves the issue? @cyb70289 @Klavinco @eaglenebulaplayer
    https://github.com/ursacomputing/crossbow/actions/runs/29994419195/artifacts/8558628657

  17. cyb70289 commented on Jul 23, 2026

    @cyb70289
    cyb70289
    on Jul 23, 2026
    Contributor

    I think you may need LTO enabled to reproduce.

    In addition, would some of you mind double checking this wheel solves the issue? @cyb70289 @Klavinco @eaglenebulaplayer https://github.com/ursacomputing/crossbow/actions/runs/29994419195/artifacts/8558628657

    Great! This wheel runs correctly per my test on graviton3. Both the segfault and wrong value cases pass.

  18. raulcd commented on Jul 23, 2026

    @raulcd
    raulcd
    on Jul 23, 2026
    Member

    Just thinking about next steps, once we have a proper, reproducible fix. This probably deserves a patch release and yanking the wheels for 25.0.0 on PyPI, @pitrou what are your thoughts on it?

  19. pitrou commented on Jul 23, 2026

    @pitrou
    pitrou
    on Jul 23, 2026
    Member

    @pitrou does it not get dictionary encoded? Then the indices get RleBitPacked.

    Hmm, perhaps. Still, would be nice to print the actual values.

  20. cyb70289 commented on Jul 23, 2026

    @cyb70289
    cyb70289
    on Jul 23, 2026
    Contributor

    Wrong back values

    [
100000, 100001, 100002, 100003, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100016, 100017, 100018, 100019, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100032, 100033, 100034, 100035, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100048, 100049, 100050, 100051, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100064, 100065, 100066, 100067, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100080, 100081, 100082, 100083, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100096, 100097, 100098, 100099, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100112, 100113, 100114, 100115, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100128, 100129, 100130, 100131, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100144, 100145, 100146, 100147, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100160, 100161, 100162, 100163, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100176, 100177, 100178, 100179, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100192, 100193, 100194, 100195, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100208, 100209, 100210, 100211, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100224, 100225, 100226, 100227, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100240, 100241, 100242, 100243, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100256, 100257, 100258, 100259, 100000, 100000, 100000, 100000, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100272, 100273, 100274, 100275, 
100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 
100288, 100289, 100290, 100291, 100292, 100293, 100294, 100295, 100296, 100297, 100298, 100299
]

  21. pitrou commented on Jul 23, 2026

    @pitrou
    pitrou
    on Jul 23, 2026
    Member

    @cyb70289 Can you try the wheel that was uploaded as an artifact in https://github.com/ursacomputing/crossbow/actions/runs/30025352042 ?

  22. cyb70289 commented on Jul 24, 2026

    @cyb70289
    cyb70289
    on Jul 24, 2026
    Contributor

    @cyb70289 Can you try the wheel that was uploaded as an artifact in https://github.com/ursacomputing/crossbow/actions/runs/30025352042 ?

    It works fine. pyarrow-26.0.0.dev45

  23. pitrou
    added a commit that references this issue on Jul 30, 2026

    GH-50503: [Parquet] Remove SVE128 unpack (#50611)

    51131e2
  24. pitrou
    added this to the 26.0.0 milestone on Jul 30, 2026

  25. pitrou commented on Jul 30, 2026

    @pitrou
    pitrou
    on Jul 30, 2026
    Member

    Issue resolved by pull request 50611
    #50611

  26. pitrou
    closed this as completedon Jul 30, 2026
  27. raulcd
    added
    backport-candidate
    on Jul 30, 2026
  28. raulcd
    modified the milestones: 26.0.0, 25.0.1 on Aug 4, 2026
  29. raulcd
    marked [Python][Parquet] Silent wrong values decoding a double column on aarch64 SVE (pyarrow 25.0.0) #50605 as a duplicate of this issue on Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: C++Component: Pythonbackport-candidate

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions