Skip to content

BUG: IndexError when using np.histogram with small values #23110

Description

@astrofrog

Describe the issue:

When using very small values with np.histogram, one can encounter an IndexError under some circumstances

Reproduce the code example:

np.histogram(np.array([-0.9e-308], dtype='>f8'), bins=2, range=(-1e-308, -2e-313))

Error message:

/home/tom/python/dev/lib/python3.11/site-packages/numpy/lib/histograms.py:810: RuntimeWarning: overflow encountered in scalar divide
  norm = n_equal_bins / _unsigned_subtract(last_edge, first_edge)
/home/tom/python/dev/lib/python3.11/site-packages/numpy/lib/histograms.py:839: RuntimeWarning: invalid value encountered in cast
  indices = f_indices.astype(np.intp)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[11], line 1
----> 1 np.histogram(np.array([-0.9e-308], dtype='>f8'), bins=2, range=(-1e-308, -2e-313))

File <__array_function__ internals>:200, in histogram(*args, **kwargs)

File ~/python/dev/lib/python3.11/site-packages/numpy/lib/histograms.py:844, in histogram(a, bins, range, density, weights)
    840 indices[indices == n_equal_bins] -= 1
    842 # The index computation is not guaranteed to give exactly
    843 # consistent results within ~1 ULP of the bin edges.
--> 844 decrement = tmp_a < bin_edges[indices]
    845 indices[decrement] -= 1
    846 # The last bin includes the right edge. The other bins do not.

IndexError: index -9223372036854775808 is out of bounds for axis 0 with size 3

Runtime information:

1.24.1

3.11.1 (main, Dec  7 2022, 01:11:34) [GCC 11.3.0]

[{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2'],
                      'not_found': ['AVX512F',
                                    'AVX512CD',
                                    'AVX512_KNL',
                                    'AVX512_KNM',
                                    'AVX512_SKX',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL']}},
 {'architecture': 'Haswell',
  'filepath': '/home/tom/python/dev/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so',
  'internal_api': 'openblas',
  'num_threads': 16,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]

Context for the issue:

No response

Activity

  1. ngoldbaum commented on Jan 27, 2023

    @ngoldbaum
    Member

    This is happening because norm is ending up inf after this line:

    https://github.com/numpy/numpy/blob/main/numpy/lib/histograms.py#L810

    ipdb> p _unsigned_subtract(last_edge, first_edge)
    9.9998e-309
    ipdb> p n_equal_bins
    2
    ipdb> p 2 / 9.9998e-309
    inf
    

    The large negative index comes from casting inf to np.intp.

  2. added 2 commits that reference this issue on Jul 10, 2023
    4d8a833
    0b83726
  3. added a commit that references this issue on Jul 14, 2023
    78137ba
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions