Incorrect exception in Table value formatting if type and format mismatch

In the following example:

In [4]: from astropy.table import Table

In [5]: t = Table()

In [6]: t['a'] = ['a','b','c']

In [7]: t['a'].format = "%9.5f"

In [8]: print(t)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-d6c4f5caf31e> in <module>()
----> 1 print(t)

/Volumes/Raptor/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9579-py3.4-macosx-10.8-x86_64.egg/astropy/table/table.py in __unicode__(self)
    664 
    665     def __unicode__(self):
--> 666         lines, n_header = self.formatter._pformat_table(self)
    667         return '\n'.join(lines)
    668     if six.PY3:

/Volumes/Raptor/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9579-py3.4-macosx-10.8-x86_64.egg/astropy/table/pprint.py in _pformat_table(self, table, max_lines, max_width, show_name, show_unit, html, tableid)
    296         for col in six.itervalues(table.columns):
    297             lines, n_header = self._pformat_col(col, max_lines, show_name,
--> 298                                                 show_unit)
    299             cols.append(lines)
    300 

/Volumes/Raptor/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9579-py3.4-macosx-10.8-x86_64.egg/astropy/table/pprint.py in _pformat_col(self, col, max_lines, show_name, show_unit)
    148         """
    149         outs = {}  # Some values from _pformat_col_iter iterator that are needed here
--> 150         col_strs = list(self._pformat_col_iter(col, max_lines, show_name, show_unit, outs))
    151         col_width = max(len(x) for x in col_strs)
    152 

/Volumes/Raptor/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9579-py3.4-macosx-10.8-x86_64.egg/astropy/table/pprint.py in _pformat_col_iter(self, col, max_lines, show_name, show_unit, outs)
    239                                   format_func(col.format, col[(i,) + multidim1]))
    240                 else:
--> 241                     col_str = format_func(col.format, col[i])
    242                 yield col_str
    243             elif i == i0:

/Volumes/Raptor/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9579-py3.4-macosx-10.8-x86_64.egg/astropy/table/pprint.py in _auto_format_func(format_, val)
     62             except:
     63                 raise ValueError('Unable to parse format string {0}'
---> 64                                  .format(format_))
     65     _format_funcs[format_] = format_func
     66     return out

ValueError: Unable to parse format string %9.5f

The exception is misleading. The issue is that the value needs to be a float. The 'proper' exception for this in Python is:

In [9]: "%9.5f" % 'a'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-55c5d51877aa> in <module>()
----> 1 "%9.5f" % 'a'

TypeError: a float is required

The current error makes it sound like the format is malformed.

I think we should only catch ValueError for parsing issues, but leave the TypeErrors. It would also be nice ideally if we could escalate the error higher up and give the column for which the issue occurred, or maybe instead print out the offending value in addition to format (to help identify where it went wrong).

Taking this one.

@cherti

@cherti - great, let me know if you have any questions!

Hmm, I'm actually not sure this is that easy to resolve, as the way it is determined whether the format is sensible is by trying to format a table item, and this process is repeated for all different ways we allow the format to be set. So, which exception should be passed on? I guess in principle, one could pass on all, but they are different for new and old-style formats:

In [14]: '{9.5f}'.format('a')
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-14-01ffea99124f> in <module>()
----> 1 '{9.5f}'.format('a')

IndexError: tuple index out of range

In [15]: '%9.5f' % 'a'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-c99945e30420> in <module>()
----> 1 '%9.5f' % 'a'

TypeError: a float is required

p.s. The file to look at is astropy/table/pprint.py, near the top, where one has:

        for format_func in _possible_string_format_functions(format_):
            if NUMPY_LE_1P6:
                format_func = _use_val_tolist(format_func)

            try:
                # Does this string format method work?
                out = format_func(format_, val)
                # Require that the format statement actually did something.
                assert out != format_
            except:
                continue
            else:
                break
        else:
            # None of the possible string functions passed muster.
            raise ValueError('Unable to parse format string {0}'
                             .format(format_))

But at the very least one could change the error message itself, to notice that the format could also be incompatible with the table data type.

@cherti

I think @cherti is working on a solution that includes trying to catch most common mistakes with a format property setter when the dtype is not np.object.

But for the more general case you are right that we will need to find a way to have it work with both formatting styles!

Indeed. I'd like to catch a mismatched format upon setting it as it can only be set when we already know the dtype of the column.
This should work for most cases, except dtype=np.object, for what we need to resort to execptions within print.

Trying '{9.5f}' for a column list ['a', 'b', 'c'] seems to throw a type error for me.
I'm using format_.format(val) for formatting and it seems to throw a TypeError (at least the error comes up on a way where I am only catching TypeErrors). I'll look into that, but it wouldn't be a problem to catch IndexErrors as well and raise a TypeError. The set of possibilities to format output should be finite and not too large, I suppose.

Nevermind, error on my side, as it seems.

I have switched the formatting entirely from the %-version over to the .format()-version in the code parts I am working on, throwing the TypeError at the user seems appropriate to me, only requires catching the IndexErrors thrown.

@mhvk

@mhvk
Can you state a line for the code snippet from pprint.py? I cannot find it in my file here.

@cherti

@cherti - we need to support the % syntax as well as the {...} syntax to remain backward compatible unfortunately. Any solution we come up with has to be backward-compatible. If it's not possible to catch all cases then at least making the error message more explicit will help.

Ah, '%9.5f' only works for format % float() and {9.5f} only works for format.format(float())?

In this case, we of course have to keep both for backwards compatibility. Are there more ways other than these two that we must support?

@cherti

@cherti - great, and just for info, you can run all the table tests with:

python setup.py test -P table

this runs all the tests in astropy.table. They should all pass, except if they are checking the content/type of the exception you are raising (in which case it's ok if they change of course).

@cherti

@cherti - the snippet I gave is from the latest development version; it includes #2898.
I do like the idea of (also) testing the format upon setting it, rather than during the printing phase.

submitted pull request here:
#3055

I wonder if this still needs fixing, since the previous attempt to fix it has been abandoned.

@embray

@embray - I think it still needs fixing

Is this PR[0] entirely abandoned or still to be merged? I think this issue would be fixed by that PR.

[0] #3261

@cherti

@cherti I have about a billion open PRs, and just forgot about #3261 completely, but thanks for pointing it out. I'll see if I can rebase it and get that going again.

Navigation Menu

Repository navigation

Incorrect exception in Table value formatting if type and format mismatch #2868

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Incorrect exception in Table value formatting if type and format mismatch #2868

Description

Activity

astrofrog commented on Aug 19, 2014

cherti commented on Sep 6, 2014

astrofrog commented on Sep 6, 2014

mhvk commented on Sep 8, 2014

mhvk commented on Sep 8, 2014

astrofrog commented on Sep 8, 2014

astrofrog commented on Sep 8, 2014

cherti commented on Sep 8, 2014

cherti commented on Sep 8, 2014

cherti commented on Sep 8, 2014

cherti commented on Sep 8, 2014

astrofrog commented on Sep 8, 2014

cherti commented on Sep 8, 2014

2 remaining items

astrofrog commented on Sep 8, 2014

mhvk commented on Sep 8, 2014

cherti commented on Oct 26, 2014

embray commented on Jul 31, 2015

astrofrog commented on Jul 31, 2015

cherti commented on Aug 4, 2015

embray commented on Aug 18, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions