Repository navigation
Enable the reading of tile compressed FITS files with kerchunk #14575
Description
kerchunk is a project which parses data and generates a zarr array using fsspec. kerchunk currently supports most FITS files, but not any tile compressed data.
This issue is intended to be a tracking issue for all the things that need to be done in all the places they need to be done to make this work.
Changes in Astropy
The main functionality that astropy needs to provide are numcodecs Codec classes for the decompression and dequantization post processing steps that need to be applied to the data on disk after read. The initial version of these Codec classes are implemented in the new _tiled_compression.codecs subpackge. Currently these codecs have not been tested outside of their internal use in io.fits.
- Raise an error if more bytes are passed to decode than expected. Currently this segfaults.
- Modify the codecs to accept the buf= argument.
- Do not assume that the input to decode is already an array, when it's likely a byte buffer.
- Refactor
decompress_hdu_sectionso that its parts could be used by kerchunk when generating references to each tile in the heap. - Modify the codecs to zero-pad tiles which are smaller than the global tile size. see this comment.
- See if it is possible to have the codecs not have per-tile settings at all. Currently PLIO, RICE and HCOMPRESS all need to know the actual tile shape of each tile (rather than the global tile shape).
- Move
finalize_arrayinto the codec.decode()methods.
Changes not in Astropy
- The Quantize codec requires per-tile arguments
zscaleandzzero, which is not supported by numcodecs / zarr. - We might need the decompression codecs to know the expected output size (real tile shape) for each tile.
- Implement building the references for each tile in the FITS file. Things to consider here:
- Some tiles in a file can be gzipped when others are compressed with other codecs
- Data can be quantized
Stretch Goals
- Enable combining more than one FITS tile into a single zarr chunk. This would be possible if the tiles are contiguous in the heap (not guaranteed), but the codecs would need to be modified to accept the bytes for more than one tile into
.decodeat once.
Activity
Cadair commentedon Mar 24, 2023
@martindurant I have some questions for you based on the above list:
.decode? You say that the input "is likely a byte buffer", but it can also be an array? what else is it allowed to be?Quantizecodec?buf=argument be? Is this documented anywhere?.decodebe? Is it ok for it to always be a numpy array with a specific dtype as calculated by the codec? - The reason for this question is currently we do this post-processing after the call todecodebut I suspect it should be insidedecode?martindurant commentedon Mar 24, 2023
Thanks for putting together the summary, I think you've covered all the major points.
Some answers:
bufis "buffer-like" according to the docstring, which would include numpy arrays and memoryviews. In kerchunk's usage it will bebytes. The output is something you can pass to anotherdecode()function, or tonp.frombufferCadair commentedon Mar 24, 2023
Sorry I mistyped, I meant what's the behaviour of the
out=argument.Yes, that's correct.
Is this a zarr proposal? and is context proposed to be an argument to
decode?martindurant commentedon Mar 24, 2023
This is an idea within zarr, yes. It is not a formal ZEP-level discussion, because it is an implementation detail rather than any change in the spec. The thread closest to a resolution is this one: zarr-developers/zarr-python#1131 , which was originally about defining what class of array a codec makes (numpy versus cupy).
According to the documentation, this can also be any mutable buffer (memoryview, bytearray...), but in practice I think it's always numpy or numpy compatible, so you can [:] assign into it and expect it to have the correct dtype and shape for the chunk. It provides for zero-copy in the case of decompression that accepts an output pointer.