Repository navigation
bad_alloc error when loading NMX nexus file #259
Description
See https://git.esss.dk/dmsc-nightly/dmsc-nightly/-/jobs/53623
On scippnexus 24.11.1:
In [1]: import scippnexus as snx
In [2]: with snx.File('994160_00043336.hdf', "r", locking=False) as f: dg = f[()]
MemoryError: std::bad_alloc
...on scippnexus 24.3.1 this does not happen.
Metadata
Metadata
Assignees
Labels
No labels
Activity
jl-wynen commentedon Jan 30, 2025
This is an NMX file, not DREAM.
The offending commit is 3a97343
The problem seems to be that in the new code in the commit, we have
which gets propagated to
Field.__getitem__fordetector_numberwhich then raisesbad_allocbecause the data volume is too large.The
detector_numberin the file has shape(1280, 1280). And8097is the number of events, not a z-offset. Even though the shape matches the comment in the linked commit, this does not seem right to me. Why would we want to allocate such an array in the first place?[-]`bad_alloc` error when loading Dream nexus file[/-][+]`bad_alloc` error when loading NMX nexus file[/+]SimonHeybrock commentedon Jan 30, 2025
I presume this is related to the NMX now having (or still not having?) an
axisattribute for the pixel offset fields?jokasimr commentedon Jan 30, 2025
NMX files now have an
axisattribute on the pixel offset fieldsjokasimr commentedon Jan 31, 2025
Looks like 8097 is the number of pulses in detector panel 0, not the number of events.
The number of events is considerably larger, the file size is 740MB.
The other panels have no pulses or events.
SimonHeybrock commentedon Jan 31, 2025
Maybe
z_pixel_offsethas a bad value in it'saxis? Ifdetector_numberis 2-D but we haveaxis=2, ScippNexus might think it refers to some other data dimension?jokasimr commentedon Jan 31, 2025
Yes this is definitely part of the issue.
z_pixel_offsethasaxis=3(x=1, y=2).But the length of
z_pixel_offsetis1so I don't know why it picks up 8097 as the size of that dimension.SimonHeybrock commentedon Jan 31, 2025
I think it happens the other way round: This claims to be the third axis of the data/detector_number, therefore ScippNexus assigns it the dim label of the third axis.
jokasimr commentedon Jan 31, 2025
Is it just a typo in the commit @jl-wynen referenced?
Should
be
?