Repository navigation
Compressed HDU enhancements #3895
Description
Activity
This should probably involve some kind of array wrapper that only decompresses tiles as needed, and perhaps also caches recently used tiles
This sounds like something dask could do if we wanted - and we would also get things like parallelisation over tiles etc for free.
Yes! To manage things efficiently we need lazy instantiation of tiles plus some means of caching and "retiring" tiles (or maybe swapping out to VM, but it would likely be faster to go back and re-read and uncompress a tile rather than writing and reading an uncompressed version of it). I completely agree that dask sounds like a good basis for the development. I'd love to see this happen and am willing to help. (I'm very familiar with the low-level cfitsio compression code since I wrote most of it.)
Hey @rlwastro I too would love to see this happen, but while I have some experience with dask and
astropy.io.fitsit sounds like you have a lot more of the skills that would be needed to make this happen!Do you have time to work on this? Is there anything myself or the Astropy project could do to help you work on this problem?
@astrofrog am I correct in saying that we have now achieved
Having all of the above fixes in place would also open the door to multi-core compression/decompression. Using tiles effectively makes this ridiculously parallelizeable.
with the latest performance PR, if you know how to do the dask magic? or are we keeping that box un-ticked until we merge #14452 ?
Yeah I would say let's wait until that PR is merged to tick the last box.
While I'm thinking about the topic, this issue will serve as a clearinghouse for work that needs to be done on the FITS tile compression support:
io.fitsinto CFITSIO in a rather complicated and ad-hoc manner. It is the only feature CFITSIO is used for. With Pythonic wrappers for the supported compression code, most of the need for CFITSIO can go away. Some of the other functionality currently in CFITSIO, such as the code for breaking an image into tiles and iterating over those tiles, can be easily rewritten in Python using Numpy. edit: done in A new tiled image compression submodule mostly replacing cfitsio #14252