Use io.ascii to parse FITS ASCII tables?

As mentioned in other issues, the eventual goal for io.fits is to use Astropy Table (or a specialized subclass thereof) as the native format in which to return table data read from FITS files (both at the high level Table.read() interface, and the lower-level HDU.data interface).

Since io.ascii already returns its results as Table objects, it seems worth investigating using it to parse FITS ASCII tables, rather than reinventing the wheel. My knowledge of io.ascii is still superficial so I don't know if it supports all the features that would be needed to parse FITS ASCII tables. And if not, I don't know how much effort it would be to add those features over just using a specialized parsers just for FITS. This is a reminder to look into it.

The fixed_width reader has the machinery you could use for parsing actual data fields in FITS ASCII table. Overall I am guessing this is the easy part of reading a FITS ASCII table, but it might not hurt to consolidate to one bit of code.

There are some funny FORTRAN-based formatting/parsing rules that I'm not sure how to incorporate, but I agree that it seems like the fixed_width reader would be a start at least.

Ooh, it even already handles inter-column cruft That was going to be my next question.

Possibly naïve question: is io.fits not using cfitsio under the hood for ASCII as well as binary files? That would seem to be far more mature than adapting io.ascii to FITS' specific Fortran-style flavour.
As an aside, perhaps HEASARC should be contacted to list astropy as Python interface on their FITSIO page rather than some package that has last been updated when pyfits was still alive.

@dhomeier

@dhomeier It does not use CFITSIO, no, for reasons.

That page is pretty woefully out of date. It doesn't even mention PyFITS despite it being older and more mature than the FITS library it does link to (not to mention maintained).

@embray

@embray - my recollection is that PyFITS used to be pure Python at one point, and for this reason I was pretty sure astropy.io.fits doesn't use cfitsio. So I was surprised to find cextern/cfitsio in astropy. What's up with that?

@taldcroft

@taldcroft It uses CFITSIO to support the tile compression format, and always has for as long as PyFITS has supported that format. But only for that, and it's optional. I would prefer to write support for that format from "scratch" (excepting rewriting the actual compression libs themselves), but it hasn't been worth the extra effort. (Though it would be necessary to fully fix #2150--compression of non-contiguous data. At least, to fix that issue without requiring copying of the data.)

@embray

@embray - yes, that was my point about that page. If you want to handle it internally, making use of the existing io.ascii would certainly be reasonable, and I think FITS tables are probably structured enough that the C parser could handle them most of the time.

Yes, I looked into this earlier, and found that io.ascii already has 90% of what I would need for that, with a fraction of the hassle =D

No one has really asked for this and this issue is now 8 years old. Can we close it as won't fix?

Hi humans 👋 - this issue was labeled as Close? approximately 16 hours ago. If you think this issue should not be closed, a maintainer should remove the Close? label - otherwise, I will close this issue in 7 days.

If you believe I commented on this issue incorrectly, please report this here

I think it could still be a useful avenue for cleanups inside the io.fits code. I don't think anyone is likely to get to it anytime ever, but I don't think that means it should be closed.

@embray

I also still see this as potentially useful. Would be great of course to know what specific approach @embray had in mind as a starting point.
Also we should consider switching right away to the accelerated np.loadtxt instead – which will require numpy >= 1.23, but it does not seem likely that would delay this task in any way. 😁

I'm not sure this is the best place to put our effort, and if there is much to gain here. ASCII tables are not used a lot AFAIK, and it's not a lot of code and using io.ascii (or np.loadtxt) would require handling some FITS specific cases.
But if someone is interested, here is the code involved:

astropy/astropy/io/fits/hdu/table.py

Lines 820 to 834 in d3b5af5

    
           for idx in range(len(columns)): 
        
               data_type = "S" + str(columns.spans[idx]) 
        
               if idx == len(columns) - 1: 
        
                   # The last column is padded out to the value of NAXIS1 
        
                   if self._header["NAXIS1"] > itemsize: 
        
                       data_type = "S" + str( 
        
                           columns.spans[idx] + self._header["NAXIS1"] - itemsize 
        
                       ) 
        
               dtype[columns.names[idx]] = (data_type, columns.starts[idx] - 1) 
        
           raw_data = self._get_raw_data(self._nrows, dtype, self._data_offset) 
        
           data = raw_data.view(np.rec.recarray) 
        
           self._init_tbdata(data) 
        
           return data.view(self._data_type)

astropy/astropy/io/fits/fitsrec.py

Lines 847 to 885 in d3b5af5

    
               def _convert_ascii(self, column, field): 
        
                   """ 
        
                   Special handling for ASCII table columns to convert columns containing 
        
                   numeric types to actual numeric arrays from the string representation. 
        
                   """ 
        
                   format = column.format 
        
                   recformat = getattr(format, "recformat", ASCII2NUMPY[format[0]]) 
        
                   # if the string = TNULL, return ASCIITNULL 
        
                   nullval = str(column.null).strip().encode("ascii") 
        
                   if len(nullval) > format.width: 
        
                       nullval = nullval[: format.width] 
        
                   # Before using .replace make sure that any trailing bytes in each 
        
                   # column are filled with spaces, and *not*, say, nulls; this causes 
        
                   # functions like replace to potentially leave gibberish bytes in the 
        
                   # array buffer. 
        
                   dummy = np.char.ljust(field, format.width) 
        
                   dummy = np.char.replace(dummy, encode_ascii("D"), encode_ascii("E")) 
        
                   null_fill = encode_ascii(str(ASCIITNULL).rjust(format.width)) 
        
                   # Convert all fields equal to the TNULL value (nullval) to empty fields. 
        
                   # TODO: These fields really should be converted to NaN or something else undefined. 
        
                   # Currently they are converted to empty fields, which are then set to zero. 
        
                   dummy = np.where(np.char.strip(dummy) == nullval, null_fill, dummy) 
        
                   # always replace empty fields, see https://github.com/astropy/astropy/pull/5394 
        
                   if nullval != b"": 
        
                       dummy = np.where(np.char.strip(dummy) == b"", null_fill, dummy) 
        
                   try: 
        
                       dummy = np.array(dummy, dtype=recformat) 
        
                   except ValueError as exc: 
        
                       indx = self.names.index(column.name) 
        
                       raise ValueError( 
        
                           "{}; the header may be missing the necessary TNULL{} " 
        
                           "keyword or the table contains invalid data".format(exc, indx + 1) 
        
                       ) 
        
                   return dummy

Navigation Menu

Repository navigation

Use io.ascii to parse FITS ASCII tables? #3430

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	for idx in range(len(columns)):
	data_type = "S" + str(columns.spans[idx])

	if idx == len(columns) - 1:
	# The last column is padded out to the value of NAXIS1
	if self._header["NAXIS1"] > itemsize:
	data_type = "S" + str(
	columns.spans[idx] + self._header["NAXIS1"] - itemsize
	)
	dtype[columns.names[idx]] = (data_type, columns.starts[idx] - 1)

	raw_data = self._get_raw_data(self._nrows, dtype, self._data_offset)
	data = raw_data.view(np.rec.recarray)
	self._init_tbdata(data)
	return data.view(self._data_type)

	def _convert_ascii(self, column, field):
	"""
	Special handling for ASCII table columns to convert columns containing
	numeric types to actual numeric arrays from the string representation.
	"""
	format = column.format
	recformat = getattr(format, "recformat", ASCII2NUMPY[format[0]])
	# if the string = TNULL, return ASCIITNULL
	nullval = str(column.null).strip().encode("ascii")
	if len(nullval) > format.width:
	nullval = nullval[: format.width]

	# Before using .replace make sure that any trailing bytes in each
	# column are filled with spaces, and not, say, nulls; this causes
	# functions like replace to potentially leave gibberish bytes in the
	# array buffer.
	dummy = np.char.ljust(field, format.width)
	dummy = np.char.replace(dummy, encode_ascii("D"), encode_ascii("E"))
	null_fill = encode_ascii(str(ASCIITNULL).rjust(format.width))

	# Convert all fields equal to the TNULL value (nullval) to empty fields.
	# TODO: These fields really should be converted to NaN or something else undefined.
	# Currently they are converted to empty fields, which are then set to zero.
	dummy = np.where(np.char.strip(dummy) == nullval, null_fill, dummy)

	# always replace empty fields, see https://github.com/astropy/astropy/pull/5394
	if nullval != b"":
	dummy = np.where(np.char.strip(dummy) == b"", null_fill, dummy)

	try:
	dummy = np.array(dummy, dtype=recformat)
	except ValueError as exc:
	indx = self.names.index(column.name)
	raise ValueError(
	"{}; the header may be missing the necessary TNULL{} "
	"keyword or the table contains invalid data".format(exc, indx + 1)
	)

	return dummy

Use io.ascii to parse FITS ASCII tables? #3430

Description

Activity

taldcroft commented on Feb 4, 2015

embray commented on Feb 4, 2015

embray commented on Feb 4, 2015

dhomeier commented on Apr 1, 2015

embray commented on Apr 1, 2015

taldcroft commented on Apr 1, 2015

embray commented on Apr 1, 2015

dhomeier commented on Apr 1, 2015

embray commented on Apr 2, 2015

pllim commented on Mar 21, 2023

github-actions commented on Mar 22, 2023

Cadair commented on Mar 22, 2023

dhomeier commented on Mar 22, 2023

saimn commented on Mar 22, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions