[C++][Python] SIGSEGV in bundled mimalloc mi_thread_init when libarrow is first loaded on a non-main thread that exits (mimalloc 3.3.x, pyarrow 25.0.0)

Describe the bug, including details regarding any error messages, version, and platform

pyarrow 25.0.0 segfaults deterministically when pyarrow is first imported on a non-main thread that later exits, and a different thread then performs its first Arrow allocation. The crash is inside the bundled mimalloc's thread initialization:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)  KERN_INVALID_ADDRESS at 0x0000000000000018

libarrow.2500.dylib   mi_heap_main
libarrow.2500.dylib   mi_thread_init
libarrow.2500.dylib   _mi_malloc_generic
libarrow.2500.dylib   mi_theap_malloc_zero_aligned_at_generic
libarrow.2500.dylib   arrow::(anonymous namespace)::MimallocAllocator::AllocateAligned(long long, long long, unsigned char**)
libarrow.2500.dylib   arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::MimallocAllocator>::Allocate(...)
libarrow.2500.dylib   arrow::PoolBuffer::Reserve(long long)
libarrow.2500.dylib   arrow::PoolBuffer::Resize(long long, bool)
libarrow_python.2500.dylib  arrow::py::ConvertPySequence / arrow::py::NdarrayToArrow

Minimal reproduction

import threading

def imp():
    import pyarrow  # dlopens libarrow -> bundled mimalloc initializes on this thread

def alloc():
    import pyarrow as pa
    pa.array(["x"] * 5000)

a = threading.Thread(target=imp); a.start(); a.join()
b = threading.Thread(target=alloc); b.start(); b.join()

Result: SIGSEGV (exit code 139). The same script:

does not crash if import pyarrow happens on the main thread first;
does not crash with ARROW_DEFAULT_MEMORY_POOL=system;
does not crash on pyarrow 24.0.0.

Root cause

This is upstream mimalloc bug microsoft/mimalloc#1287 ("mimalloc >= 3.3.0 causes segmentation faults when used from multiple threads", closed 2026-06-20): mimalloc treats the first thread that initializes it as the process main thread; when that thread exits, subsequent threads crash in mi_thread_init/_mi_malloc_generic. mimalloc 3.2.x is unaffected.

Arrow 25.0.0 is the first release affected because GH-49772 / PR #49801 bumped ARROW_MIMALLOC_BUILD_VERSION from v3.2.7 to v3.3.1 (confirmed in cpp/thirdparty/versions.txt at tag apache-arrow-25.0.0).

The upstream fix exists only on mimalloc's dev3 branch (commit microsoft/mimalloc@b92c116b67d0, "allow initial main thread to terminate before the process terminates (see issue microsoft/mimalloc#1287)"); the latest tagged mimalloc release (v3.3.2, 2026-04-29) predates it, so there is currently no tagged mimalloc version Arrow could bump to.

Real-world impact

This import pattern is not exotic — it is structurally guaranteed by Streamlit + pandas 3:

pandas 3.x imports pyarrow eagerly (default pyarrow-backed str dtype);
Streamlit executes user scripts (including their imports) on short-lived ScriptRunner threads, never on the main thread.

So in any Streamlit app using pandas 3 + pyarrow 25, libarrow first loads on script-run thread #1, and the next script run (a simple page reload) segfaults the whole server on its first string-column conversion (pandas maybe_convert_objects → arrow::py::NdarrayToArrow). We observed 7+ identical crashes per day on one such app before diagnosing.

Workarounds

ARROW_DEFAULT_MEMORY_POOL=system (verified);
import pyarrow on the main thread before any worker thread touches it (verified);
pin pyarrow<=24.0.0 (bundles unaffected mimalloc v3.2.7).

Suggested resolution

Consider reverting the bundled mimalloc to the v3.2.x line until a tagged mimalloc release contains the #1287 fix, or picking up that fix as a patch on top of v3.3.1.

Related but distinct: #50428 (interpreter-teardown crash when co-loading another mimalloc-v3-bundling extension; the ARROW_DEFAULT_MEMORY_POOL=system workaround reportedly does not help there, whereas it does here).

Environment

pyarrow 25.0.0 (PyPI wheel), pandas 3.0.3, streamlit 1.59.1
CPython 3.14.5 (Homebrew), macOS 26 (Darwin 25.5.0), Apple Silicon
Note: mimalloc#1287 was reproduced upstream on Linux and Windows as well, so this is unlikely to be macOS-specific.

Component(s)

C++, Python

@L3o-pold

Streamlit dev here 👋 @L3o-pold, thanks for investigating!

The PyArrow 25.0.0 update broke our CI and, very likely, also a significant number of apps that get updated to this version. Is there a way to get this patched soon? cc @raulcd

@kou

I haven't investigated yet so I don't fully understand the issue or the solution but from my initial understanding we could add to our ci/vcpkg/ports.patch the patch from the upstream commit microsoft/mimalloc@b92c116b67d0 and build the wheels with a patched mimalloc. There were some discussions about a possible 25.0.1 patch release but it's not entirely clear when those patches will be ready.

@kou @pitrou any thoughts on the issue?

[fix] Avoid PyArrow 25 thread initialization crash (#15947)

I don't object that we backport the patch by ourselves but I'm negative for the approach because it may introduce another problem.

How about using system malloc() on non-main thread something like the following?

diff --git a/cpp/src/arrow/memory_pool.cc b/cpp/src/arrow/memory_pool.cc
index 0b4843bec3..d00760291d 100644
--- a/cpp/src/arrow/memory_pool.cc
+++ b/cpp/src/arrow/memory_pool.cc
@@ -130,12 +130,40 @@ std::optional<MemoryPoolBackend> UserSelectedBackend() {
   return user_selected_backend;
 }
 
+bool IsMainThread() {
+#ifdef __linux__
+  // The main thread's ID must equal to process ID on Linux.
+  return gettid() == getpid();
+#elif defined(__APPLE__) || defined(BSD)
+  // We can use pthread_main_np() on BSD family.
+  return pthread_main_np() == 1;
+#elif defined(_WIN32)
+  // Windows doesn't provide an API for this. We can implement a
+  // heuristic approach.
+  return true;
+#else
+  // Assume the current thread is the main thread on others.
+  return true;
+#endif
+}
+
 MemoryPoolBackend DefaultBackend() {
   auto backend = UserSelectedBackend();
   if (backend.has_value()) {
     return backend.value();
   }
-  struct SupportedBackend default_backend = SupportedBackends().front();
+  static struct SupportedBackend default_backend;
+  static std::once_flag once;
+  std::call_once(once, [&]() {
+    if (IsMainThread()) {
+      default_backend = SupportedBackends().front();
+    } else {
+      // Fallback to the system backend.
+      // We can remove this once mimalloc()'s thread related problem is fixed.
+      // See also: https://github.com/microsoft/mimalloc/issues/1287
+      default_backend = SupportedBackends().back();
+    }
+  });
   return default_backend.backend;
 }

[fix] Cap server dependency major versions (#15948)

fix: Pinar pyarrow<25 para corrigir segfault no Streamlit Cloud

fix: pin pyarrow<25 to stop Streamlit Cloud segfault

We hit this same crash in production and can confirm it from a different
platform and a different trigger path than the original report.

Environment: Linux x86_64 (python:3.14-slim Debian container on
Kubernetes; also reproduced on a local Linux machine), Python 3.14,
pyarrow 25.0.0, polars 1.42.1, pandas 2.3.3, Streamlit 1.53.1.

Trigger: not the pandas 3 eager-import path described above — we are on
pandas 2.3.3. Our entry point is polars.DataFrame.to_pandas() inside a
Streamlit app: libarrow is first imported on a short-lived Streamlit
ScriptRunner thread, and the crash fires when the conversion
(pyarrow.Table.to_pandas → table_to_dataframe) performs its first
allocation on an Arrow C++ thread-pool worker. Same mechanism, different
entry point — so it is not specific to pandas 3 or NdarrayToArrow.

Faulthandler output (trimmed):

Fatal Python error: Segmentation fault

Current thread 0x00007fe72bfff6c0 [ScriptRunner.sc] (most recent call first):
  <no Python frame>    # Arrow C++ pool worker; thread name inherited from creator

Thread 0x00007fe6457ff6c0 [ScriptRunner.sc] (most recent call first):
  File "site-packages/pyarrow/pandas_compat.py", line 827 in table_to_dataframe
  File "site-packages/polars/dataframe/frame.py", line 2691 in _to_pandas_without_object_columns
  File "site-packages/polars/dataframe/frame.py", line 2629 in to_pandas
  ...

Current thread's C stack trace (most recent call first):
  libarrow.so.2500      +0x16f8b89 / +0x16feaa1 / +0x16ee5c7
  libarrow.so.2500      +0xb953bb / +0xb9bd9c / +0xb9be29
  libarrow.so.2500      arrow::AllocateBuffer(long long, long long, arrow::MemoryPool*)+0x35
  libarrow.so.2500      arrow::AllocateBuffer(long long, arrow::MemoryPool*)+0x11
  libarrow_python.so.2500  +0x5fc70 / +0x79cfb / +0x5d705 / +0x5f96d / +0x5eb5e / +0x6f2c4
  libarrow.so.2500      +0xbdb7f6 / +0xbdb1c4      # thread pool task loop
  libstdc++.so.6        +0xeb919                   # std::thread

Operational impact matches the report: the whole Streamlit server dies with
SIGSEGV and no Python traceback. On Kubernetes this presents as unexplained
pod crash-loops immediately after deployment (first page loads), which took
some effort to attribute.

Workaround confirmed: ARROW_DEFAULT_MEMORY_POOL=system — deployed to
production, no crashes since (previously two crashes within the first two
minutes of the pod's life).

Possibly useful for resolving this: mimalloc v3.4.1 (tagged
2026-07-15) now contains the fix commit
microsoft/mimalloc@b92c116b67d0 — I verified the commit is an ancestor of
the v3.4.1 tag. So a tagged mimalloc release containing the fix exists
now, which was not the case when this issue was opened; bumping
ARROW_MIMALLOC_BUILD_VERSION may be viable for a 25.0.1.

@raulcd

I don't object that we backport the patch by ourselves but I'm negative for the approach because it may introduce another problem.

We can simply bump mimalloc to the just-released 3.4.1. Also @raulcd

@L3o-pold

I've been able to reproduce this issue consistently with the minimal reproducer shared above. Thanks @L3o-pold for it.
Bumping mimalloc with the newly released version (3.4.1) and building a new wheel solves the issue. I haven't been able to reproduce it with the new wheel anymore. I've pushed a PR bumping mimalloc.

Epingler pyarrow<25 (24.0.0) pour coller au runtime du Cloud

Contraindre pyarrow < 25 : la plateforme refuse la version que je fig…

Modulariza o app, otimiza custodia e torna a busca no BCB resiliente …

Pin pyarrow below 25 to avoid a known segfault

Figer les versions des dependances (crash natif au demarrage sur Cloud)

Pin pyarrow<25 to avoid a known segfault (apache/arrow#50471)

Fix dataset merge crashes and preserve gripper metadata during conver…

test: drive st.dataframe through the UI suite with a feature-aware mock

Arreglar el OOM en Streamlit Cloud: dtypes category en el pipeline

Bump streamlit 1.64.0 + pyarrow 25.0.1; Dependabot groups (#166)

Infra/boot: fixa pyarrow<25 — elimina reinstalação redundante do pyar…

Pin pyarrow below the known-segfault 25.x release

Pin pyarrow 24.0.0 to match production and avoid a known segfault

Pin pyarrow 24.0.0, the version Streamlit Cloud actually runs

Navigation Menu

Repository navigation

[C++][Python] SIGSEGV in bundled mimalloc mi_thread_init when libarrow is first loaded on a non-main thread that exits (mimalloc 3.3.x, pyarrow 25.0.0) #50471

Describe the bug, including details regarding any error messages, version, and platform

Minimal reproduction

Root cause

Real-world impact

Workarounds

Suggested resolution

Environment

Component(s)

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[C++][Python] SIGSEGV in bundled mimalloc mi_thread_init when libarrow is first loaded on a non-main thread that exits (mimalloc 3.3.x, pyarrow 25.0.0) #50471

Description

Describe the bug, including details regarding any error messages, version, and platform

Minimal reproduction

Root cause

Real-world impact

Workarounds

Suggested resolution

Environment

Component(s)

Activity

lukasmasuch commented on Jul 12, 2026

raulcd commented on Jul 13, 2026

kou commented on Jul 14, 2026

gkotian commented on Jul 17, 2026

pitrou commented on Jul 20, 2026

raulcd commented on Jul 20, 2026

11 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions