Indexing sometimes treats booleans as indices

When indexing by a boolean array, NumPy selects elements where the boolean array is True. Scipp also does that, but only in one specific situation:

>>> import numpy as np
>>> import scipp as sc
>>> a = sc.array(dims=['x'], values=[1,2,3,4,5])
>>> a[sc.array(dims=['x'], values=[False, True, True, True, False])].values
array([2, 3, 4])

This correctly selects some array elements.

But if we change the call slightly to any of the following, the booleans are interpreted as indices instead:

Add an explicit dimension label to the slice
Use a NumPy array
Use a list

>>> a['x', sc.array(dims=['x'], values=[False, True, True, True, False])].values
array([1, 2, 2, 2, 1])
>>> a[np.array([False, True, True, True, False])].values
array([1, 2, 2, 2, 1])
>>> a['x', np.array([False, True, True, True, False])].values
array([1, 2, 2, 2, 1])
>>> a[[False, True, True, True, False]].values
array([1, 2, 2, 2, 1])
>>> a['x', [False, True, True, True, False]].values
array([1, 2, 2, 2, 1])

All of these are surprising to me and I think they need to be changed. Booleans should always act as a filter, not an index.

Integer-array indexing is along a single dim (which we allow to be omitted, just like when indexing with a single int). Boolean-array indexing is different as one needs to have an exact match of the indexed array (dims and shape). We do not want to support deducing dims as that would be error prone, e.g., when you have a NxN array, going against a core concept of Scipp.

You argue for deducing the dim it in 1D, but I think discriminating from integer-array indexing via the dtype is a dangerous idea. You might similarly suggest to discriminate between positional and label-based indexing by looking if an index matches the dtype of a coord. For example, while in the code you wrote above it is obvious the the indices are booleans, as soon as they originate from a helper function it will be much less obvious.

As you have found, interpreting boolean as integer-indices is also surprising (and thus error prone). Given that there are likely few genuine cases where someone wants to index with boolean we can consider raising an exception when a bool dtype shows up in the integer-array indexing methods.

Boolean-array indexing is different as one needs to have an exact match of the indexed array (dims and shape).

Why, though? Numpy doesn't have that restriction:

>>> a = np.array([[1, 2, 3], [4, 5, 6]])
>>> a[[True, False]]
array([[1, 2, 3]])

If we change this to support lower-dim masks, we should also support deducing the dim for 1D arrays.

I think discriminating from integer-array indexing via the dtype is a dangerous idea.

Isn't this exactly what we do? The dtype (and shape) completely change the meaning of the operation. I think this this is just a point I raised a while back that overloading __getitem__ as heavily as we do is confusing.

We do not want to support deducing dims as that would be error prone, e.g., when you have a NxN array, going against a core concept of Scip

You might similarly suggest to discriminate between positional and label-based indexing by looking if an index matches the dtype of a coord.

Neither of those are what I suggested and they don't follow from this.

I think discriminating from integer-array indexing via the dtype is a dangerous idea.

Isn't this exactly what we do? The dtype (and shape) completely change the meaning of the operation. I think this this is just a point I raised a while back that overloading __getitem__ as heavily as we do is confusing.

I can't think of a case where we discriminate based on dtype in scipp? The type of the index is used, but the dtype?

I can't think of a case where we discriminate based on dtype in scipp? The type of the index is used, but the dtype?

Right, I thought a[sc.array(dims=['x'], values=[1, 0])] would be allowed. But it isn't.
But isn't it strange that a[var] and a['x', var] mean different things while a[int] and a['x', int] are the same (for 1D)?

But isn't it strange that a[var] and a['x', var] mean different things

You mean boolean indexing vs. label-based indexing? Yes, we have likely overloaded __getitem__ to much.

But isn't it strange that a[var] and a['x', var] mean different things

You mean boolean indexing vs. label-based indexing? Yes, we have likely overloaded __getitem__ to much.

But are we forced to live with that decision or can we still improve the interface? Why can't we support lower-dimensional bool arrays and simply broadcast to the shape of the array? Or support a['x', mask]?

But are we forced to live with that decision or can we still improve the interface?

Difficult to remove existing overloads without break many user's scripts or workflows.

Maybe we are broadcasting already, I can't remember?

Or support a['x', mask]

So you'd write some code that

Checks if mask has dtype=bool
Checks if there is an 'x' coord
If mask is 1D or higher, interpret it as a boolean mask.
If mask is 0D and there is a coord, do label-based indexing
If mask is 0D and there is no coord, broadcast

Or alternatively not support label-based indexing with a bool coord?

In either case this seems odd and highly error prone (many tests and docs needed?). I would still argue for simply raising an exception when attempting to do integer-array-indexing with dtype=bool index arrays or lists.

I just tested something. We treat array-indexing inconsistently:

>>> da = sc.DataArray(sc.arange('x', 4), coords={'x': sc.arange('x', 4, 8)})
>>> da['x', sc.array(dims=['x'], values=[2, 3])]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
scipp._scipp.core.DimensionError: Expected a scalar, got dimensions (x: 2).
>>> da.data['x', sc.array(dims=['x'], values=[2, 3])]
<scipp.Variable> (x: 2)      int64  [dimensionless]  [2, 3]

For data arrays, indexing with a sc.Variable is treated as label-based indexing regardless of dimensions. For variables, it is treated as indexing by an index list.

Seems like a bug. Is it some pybind11 conversion path or do we actually support variables for integer-array indexing on the C++ side?

It calls this overload:

[](T &self, std::tuple<std::string, std::vector<scipp::index>> indices) { ... }

So pybind11 seems to convert the index variable to a vector, including the dtype.

Navigation Menu

Repository navigation

Indexing sometimes treats booleans as indices #3909

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Indexing sometimes treats booleans as indices #3909

Description

Activity

SimonHeybrock commented on Jun 19, 2026

jl-wynen commented on Jun 22, 2026

SimonHeybrock commented on Jun 22, 2026

jl-wynen commented on Jun 22, 2026

SimonHeybrock commented on Jun 22, 2026

jl-wynen commented on Jun 23, 2026

SimonHeybrock commented on Jun 23, 2026

jl-wynen commented on Jun 23, 2026

SimonHeybrock commented on Jun 23, 2026

jl-wynen commented on Jun 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions