Skip to content

fix: use shape of grouping and not shape of grouping + shape of signal - #262

Merged
jokasimr merged 6 commits into
mainfrom
fix-detector-dim
Feb 28, 2025
Merged

jokasimr merged 6 commits into
mainfrom
fix-detector-dim

Conversation

@jokasimr

@jokasimr jokasimr commented Jan 31, 2025 •

Copy link
Copy Markdown
Contributor

Fixes #259

This change solves the issue and passes the tests, and it does seem reasonable that we don't care about the shape of the signal when assigning the size of the grouping.

However the removed comment seems to indicate that we do care.

When loading the offending file with this change the instrument/detector_0 entry gets the somewhat cryptic sizes: sizes={'x_pixel_offset': 1280, 'y_pixel_offset': 1280, 'face': 6, 'vertex': 8, 'winding_order': 24, 'time': 0, 'event_time_zero': 1}

So there's still something wrong here.

@jokasimr
jokasimr marked this pull request as draft January 31, 2025 10:13
@jokasimr

Copy link
Copy Markdown
Contributor Author

Here's the instrument/detector_panel_0:

DataGroup(sizes={'x_pixel_offset': 1280, 'y_pixel_offset': 1280, 'face': 6, 'vertex': 8, 'winding_order': 24, 'time': 0, 'event_time_zero': 1}, keys=[
    data: DataArray({'x_pixel_offset': 1280, 'y_pixel_offset': 1280}),
    depends_on: TransformationChain(parent='/entry/instrument/detector_panel_1', value='/entry/instrument/detector_panel_1/transformations/orientation', transformations=DataGroup(sizes={'time': 0}, keys=[
    /entry/instrument/detector_panel_1/transformations/orientation: Transform({}),
    /entry/instrument/detector_panel_1/transformations/axis6: Transform({'time': 0}),
    /entry/instrument/detector_panel_1/transformations/axis5: Transform({'time': 0}),
    /entry/instrument/detector_panel_1/transformations/axis4: Transform({'time': 0}),
    /entry/instrument/detector_panel_1/transformations/axis3: Transform({'time': 0}),
    /entry/instrument/detector_panel_1/transformations/axis2: Transform({'time': 0}),
    /entry/instrument/detector_panel_1/transformations/axis1: Transform({'time': 0}),
    /entry/instrument/detector_panel_1/transformations/stageZ: Transform({'time': 0}),
])),
    detector_number: Variable({'x_pixel_offset': 1280, 'y_pixel_offset': 1280}),
    efu_publication: DataGroup(4, {}),
    geometry: DataGroup(3, {'face': 6, 'vertex': 8, 'winding_order': 24}),
    transformations: DataGroup(8, {'time': 0}),
    x_pixel_offset: Variable({'x_pixel_offset': 1280}),
    y_pixel_offset: Variable({'y_pixel_offset': 1280}),
    z_pixel_offset: Variable({'event_time_zero': 1}),

The only issue I see there is: z_pixel_offset: Variable({'event_time_zero': 1}),

@jokasimr

Copy link
Copy Markdown
Contributor Author

Okay I think I got the gist of the problem now.

  • The axis attribute on z_pixel_offset tells us that this is a dimension of the NXdetector.
  • The signal in the NXdetector does not have this dimension, it has the dimensions of detector_numbers and its own intrinsic dimensions (for event data that is event_time_zero).
  • For the result to be a DataArray the data field needs to have all dimensions of all coordinates.
  • Then to fix this either we have to exclude the z_pixel_offset coordinate from the resulting DataArray or we need to return a DataGroup instead of a DataArray if the dimensions of the signal and the dimensions of the detector don't match.

@SimonHeybrock do you have an opinion about the best approach here?
I think dropping the z_pixel_offset coordinate makes most sense.

@SimonHeybrock

Copy link
Copy Markdown
Member

@SimonHeybrock do you have an opinion about the best approach here? I think dropping the z_pixel_offset coordinate makes most sense.

The coord can be there, if it is a scalar, i.e., the length-1 dimension is squeezed out. But isn't the issue arising even before the data array assembly is attempted?

@jokasimr

jokasimr commented Feb 26, 2025 •

Copy link
Copy Markdown
Contributor Author

But isn't the issue arising even before the data array assembly is attempted?

It does, because the NXdetector code assumes that the dimensions of the signal field contains all dimensions found in the NXdetector group. But to fix it we need to know how extra dimensions should be handled, that is, if they should be dropped or included.

How about this?

  • If the coordinate is length 1 it's treated as a scalar and the dimension is removed.
  • If the coordinate is non-scalar we raise.

@jokasimr

Copy link
Copy Markdown
Contributor Author

Just to clarify, the result of loading a NXdetector should be a DataArray right?

@SimonHeybrock

Copy link
Copy Markdown
Member

Just to clarify, the result of loading a NXdetector should be a DataArray right?

No. It is a DataGroup, containing a DataArray. The latter contains the signal dataset as well as all its coords and masks. NXdetector may have additional bits that are no coords, those are just places in the data group.

@jokasimr
jokasimr marked this pull request as ready for review February 26, 2025 12:57
Comment thread src/scippnexus/nxdata.py
Comment on lines -214 to +218
# EventField uses dims of detector_number (the grouping) plus dims of event
# data. The former may be defined by the group dims.
if group_dims is not None:
self._signal._grouping.sizes = dict(
zip(group_dims, self._signal.shape, strict=False)
zip(group_dims, self._signal._grouping.shape, strict=False)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change still needed, given the latest changes below?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes otherwise we still get the out of memory error. Do you think it might be incorrect to change this?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not 100% sure, but I think it is covered by existing tests. But can you preserve the comment, at least?

@jokasimr
jokasimr enabled auto-merge (squash) February 28, 2025 07:23
@jokasimr
jokasimr merged commit 647258b into main Feb 28, 2025
@jokasimr
jokasimr deleted the fix-detector-dim branch February 28, 2025 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bad_alloc error when loading NMX nexus file

2 participants