Repository navigation
Warn/raise error when a fits file cannot be opened in memory map mode #2295
Description
I spent some time trying to figure out why the following code created a non-memory mapped file:
x = fits.open('m33_hi.pbcorr.fits', memmap=True)[0].data # uses all my RAM
x.flags.owndata # True
I think I traced this down to the presence of the BLANK header keyword, so the array needs to be transformed at load-time (If I create a new file without this keyword, memory mapping works).
It would be helpful if:
fits.openraised an error here, to alert me why using memmap mode isn't possible.- the
do_not_scale_image_datakwarg also ignoredBLANK, so that memory mapping was possible. - when I run
hdulist[0].header.pop('BLANK'), thenhdulist[1].dataignoresBLANKand makes a memory-mapped data array
As it stands, I can't figure out a way to get a memory-mapped view into this fits file (transformed or otherwise)
Activity
ChrisBeaumont commentedon Apr 8, 2014
Also a fun fact: the data in this file is a 32-bit float and, if the information on this page is correct, then the
BLANKkeyword (specific to integer dtypes) isn't valid anyways.Raise error if fits.open(memmap=True) is impossible.
embray commentedon Apr 8, 2014
I'm going to leave this open as-is for now, but the answer to this isn't necessarily what you're suggesting. First of all, doing this as soon as a FITS file is opened is just imposing another set of checks on every file of the sort I'd like to avoid for now, especially as I'm trying to speed up file opening/reading.
Second of all, what you've hit on here affects a lot more than just
BLANK, and is a deep issue I've been wrestling almost ever since taking the reins on PyFITS. It affects any data that needs to be transformed from its on-disk, raw storage format to a more "user-visible" format. This includes any data that is scaled withBSCALEand/orBZERO. It affects boolean columns in tables. It most certainly affects ASCII tables. And worst of all it affects strings, especially, on Python 3, as Numpy does not provide a type of "unicode" array with different internal storage formats depending on the maximum bytecode in the string (as Python itself does).The string thing I actually have a partial solution for that I've been toying with. Hope to make it a real thing at some point. The others are a much deeper issue.
There are three potential solutions I've been toying with: One is to make a custom Numpy array type that can do the conversions on the fly (and would have to do them before/after any ufunc), but that hits up against some of the problems we've found in trying to implement
Quantityand may break down with use in more specialized code, like routines in scipy.The second, which will have performance issues of its own and a handful of platform-specific issues would involve allocating a temporary file to store the converted values on the fly and do some sort of paging of converted values to this temp file. That would allow limiting physical memory usage (while also not being limited by the system's pagefile), though it would have a potential tradeoff in required free disk space.
The third is may favorite, though I haven't actually tried it yet. It would work almost transparently, but thinking about the platform support issues makes my legs go wobbly, not to mention the time and effort it would take to get right.. But in principle I could use libsigsegv to implement my own page handler--then I could just allocate some region of virtual memory and pass a pointer to it off to Numpy. Page faults would be handled through my custom pager which would do the requisite data conversions on the fly, though it would still have to page to and from disk in order to avoid using up too much physical memory (so effectively re-implement mmap). This would be super cool and I would love to do it, but I have no idea even in theory how well it would work out performance-wise. I really wish I could find the time to come up with a prototype though.
(And yes,
BLANKis only intended for use with integer data to begin with)ChrisBeaumont commentedon Apr 9, 2014
Ah, sorry, I responded to the PR before I read this.
I appreciate the general problem, and I've also wanted the ability to express transformations on arrays that are evaluated transparently + lazily.
The handling of
BLANKstrikes me as a potentially very easy fix (albeit different from my first proposal). I hope we we can fix this special case independently of (and before) solving the general problem.embray commentedon Apr 9, 2014
I agree, there were really like three issues wrapped up in this. The ones explicitly related to
BLANKcan and should be resolved.ChrisBeaumont commentedon Apr 9, 2014
Regarding the ability to ignore
BLANK-- is there a solution you prefer over thedo_not_scale_image_datakwarg?embray commentedon Apr 9, 2014
I'm going to overhaul how options for reading data are passed down to HDUs. But in the meantime we need a new option. Maybe something like
replace_missing_data, though preferably less verbose if you can think of anything better.