Repository navigation
[Python] Convert maps to dicts without per-element Scalars in to_pylist(maps_as_pydicts=...) #50429
Description
Describe the enhancement requested
Follow-up of #50326. GH-50327 makes Array.to_pylist() convert without per-element Scalars, but the maps_as_pydicts option still routes to the Scalar-based path: every map row allocates a MapScalar, converts its keys via per-element as_py, and builds the dict in Python.
Converting maps to dicts is the natural consumption pattern for engines whose map values are Python dicts (e.g. Spark's Arrow-serialized Python UDFs currently receive association lists from to_pylist() and rebuild a dict per row in pure Python — one of the dominant remaining costs in that path).
Proposal: thread maps_as_pydicts through the scalar-free _getitem_py mechanism introduced in GH-50327 so MapArray builds the dict directly from the flattened keys/items children:
- default (
None) semantics unchanged (association lists); 'lossy'/'strict'build the dict in one pass; when the dict size shows duplicate keys, the row is redone with the careful per-key loop so the warning ('lossy') andKeyError('strict') semantics stay exactly identical toMapScalar.as_py;- invalid values still raise the same
ValueErrorwhen a map value is converted; - the option propagates through nested types (list/struct/map values) as before, and unspecialized types keep the exact Scalar fallback.
Component(s)
Python
Activity
apacheGH-50429: [Python] Convert maps to dicts without per-element Sc…
apacheGH-50429: [Python] Detect duplicate keys before converting values
apacheGH-50326: [Python] Add view-type test cases and error-message a…
apacheGH-50429: [Python] Convert maps to dicts without per-element Sc…
apacheGH-50429: [Python] Detect duplicate keys before converting values
apacheGH-50429: [Python] Convert maps to dicts without per-element Sc…
apacheGH-50429: [Python] Detect duplicate keys before converting values
apacheGH-50429: [Python] Handle unhashable map keys in the dict fast …
apacheGH-50429: [Python] Convert maps to dicts without per-element Sc…
apacheGH-50429: [Python] Detect duplicate keys before converting values
apacheGH-50429: [Python] Handle unhashable map keys in the dict fast …
apacheGH-50429: [Python] Convert maps to dicts without per-element Sc…
apacheGH-50429: [Python] Detect duplicate keys before converting values
apacheGH-50429: [Python] Handle unhashable map keys in the dict fast …
apacheGH-50429: [Python] Always use the per-key loop for dict conversion
apacheGH-50429: [Python] Preserve StructScalar.as_py's nested KeyErro…
apacheGH-50429: [Python] Validate maps_as_pydicts like MapScalar.as_py
apacheGH-50429: [Python] Use literal expected values in the maps_as_p…
apacheGH-50429: [Python] Cover large_list and fixed_size_list in the …
apacheGH-50429: [Python] Simplify the dict path per review