Latex reader should accept tables without \begin/\end

While trying to read in a Latex table I copied-and-pasted from an arXiv paper, I learned an oddity: the Latex reader fails with an obscure error if you just put in the contents of a latex table (see stack trace at the end. I see no reason why it shouldn't be possible to have it parse Latex style tables without the \begin{table}/\begin{tabular}{ccccccccccc} and corresponding \end commands. It doesn't even seem to care how many markers you have in \tabular, so it's not using the header for anything. So it should be possible to just past in the table data and have it parse, rather than having to add the \begin \end commands around the content.

This could be construed as part of #1525, although that's a bit more wide-ranging and this is one particular thing in that direction.

Stack trace from attempting to Table.read(...,format='latex') just the content of a Latex table:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-43-e9b3cdb9f6d8> in <module>()
     32 """.replace('~','').replace('\,', '').replace('$', '')
     33 
---> 34 k15_tab2 = table.Table.read(k15_tab2_data.split('\n'), format='latex', names=k15_tab2_colnames, guess=False)
     35 k15_tab2

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/table/table.pyc in read(cls, *args, **kwargs)
   1896         passed through to the underlying data reader (e.g. `~astropy.io.ascii.read`).
   1897         """
-> 1898         return io_registry.read(cls, *args, **kwargs)
   1899 
   1900     def write(self, *args, **kwargs):

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/registry.pyc in read(cls, *args, **kwargs)
    330 
    331         reader = get_reader(format, cls)
--> 332         data = reader(*args, **kwargs)
    333 
    334         if not isinstance(data, cls):

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/connect.pyc in io_read(format, filename, **kwargs)
     35     from .ui import read
     36     format = re.sub(r'^ascii\.', '', format)
---> 37     return read(filename, format=format, **kwargs)
     38 
     39 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/ui.pyc in read(table, guess, **kwargs)
    273                 dat = reader.read(table)
    274         else:
--> 275             dat = reader.read(table)
    276 
    277     return dat

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/core.pyc in read(self, table)
   1038 
   1039         # Set self.data.data_lines to a slice of lines contain the data rows
-> 1040         self.data.get_data_lines(self.lines)
   1041 
   1042         # Extract table meta values (e.g. keywords, comments, etc).  Updates self.meta.

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/core.pyc in get_data_lines(self, lines)
    614         table data values."""
    615         data_lines = self.process_lines(lines)
--> 616         start_line = _get_line_index(self.start_line, data_lines)
    617         end_line = _get_line_index(self.end_line, data_lines)
    618 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/core.pyc in _get_line_index(line_or_func, lines)
    399 
    400     if hasattr(line_or_func, '__call__'):
--> 401         return line_or_func(lines)
    402     elif line_or_func:
    403         if line_or_func >= 0:

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/latex.pyc in start_line(self, lines)
    151             return find_latex_line(lines, self.data_start)
    152         else:
--> 153             return self.header.start_line(lines) + 1
    154 
    155     def end_line(self, lines):

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

I think the intent of the LaTeX reader is to allow for a fair bit of non-table cruft (i.e. the paper) around the table elements. So you need something to indicate that a table is starting and ending. But more to the point, isn't is required to have \begin/end{tabular} around a LaTeX tabular-formatted section? I agree that for a regular latex format table there should not be a need for the \begin/end{table} as that isn't required in the latex document. For aastex it's a different story.

Otherwise to handle a bare table with & delimiter and \\ at the end of each line, it's a one-line workaround from your example to strip the \\ at the line ends, after which the list of lines will parse with delimiter='&'.

@taldcroft

@taldcroft do you want to make latex reader more flexible by treating a single table with '&' delimiter and each line of the table ends with
remove the check for the \begin/end{tabular} is this so which you want to fix

fix for the nonetype error astropy#4295

@taldcroft

@taldcroft i have fix the nonetype error. but i need a help .if you want a flexible reading of table then it goes right with one table but what if we had two table .how should i identify that new table has started.should we use a delimeter for new table start.

@hamogu

@hamogu - do you have an opinion on whether the latex reader could (or should) be updated to use only \tabular tags to find the table instead of \table? Does it parse required information from the \table definition?

@eteq

Use the tabular tag?

The current LaTeX reader already uses the \tabular tag, you can read a table without a \table tag perfectly fine:

In [9]: from astropy.table import Table

In [10]: tab = r'''stuff here
\begin{tabular}{cc}
col1 & col2 \\
3 & 4\\
\end{tabular}'''

In [11]: Table.read(tab, format='ascii.latex')
Out[11]:
<Table length=1>
 col1  col2
int64 int64
----- -----
    3     4

The bug that turned up for the @eteq example is due to something else: There is a if line in the code where it should be if line is None. If line (the line with the \begin{tabular}) is 0 then that comparison does not give the result it should have.

@AMAN3003 : If you have a fix for the Nonetype error, please make that a pull request. I have a solution, too and will issue a pull request, too and then we can compare our approaches and see what we do here.

Do we need a `tabular` tag at all?

If I understand @eteq correctly, he want to read tables without even the \begin{tabular} and \end{tabular} tags, i.e. just the raw latex lines.
In principle we can do that too, but in the current form there are two problems:

As @AMAN3003 says: If we allow input of a large latex file then we need to make use of the delimiters \begin{tabular} and \end{tabular} to know where a table starts and ends.
If we allow any lines outside the tabular environment (be it even \begin{table}) then we only way in the current scheme is to try start reading in every line and see if it fails. That's very inefficient.

If we want to allow the @eteq use case, too, I suggest that we make a new reader (BareLatexTabularReader) that inherits from Latex with few modifications in start_header, start_line etc.
We can add that reader to the guess list easily because LaTeX tables are so specific that there is little chance of confusion.

Fix reading of Latex tables with ``\tabular`` tag in the first line

@hamogu

@hamogu - thanks for the explanation and the fix in #4595, that makes sense.

My understanding is that a table is not a latex table without the \tabular tag. In other words you cannot find tabular data embedded within a well-formed latex file (which runs through latex without errors) that does not have the surrounding \tabular. If so then I don't see any real reason to support the use case of bare table values. We can just tell uses to copy/paste the entire table.

@hamogu

@hamogu do you want me to make a separate PR for the issue and add test for it to solve the fix of nonetype

@AMAN3003

@AMAN3003 - I think that the solution from @hamogu is good enough, so there is no strong need to make an alternate PR here.

@taldcroft

@taldcroft but we can implement as @eteq need as i did it for one table .if you could help .. but i only need to add some function to check if the new table latex have started

finafix for the nonetype error astropy#4295

@taldcroft

@taldcroft @eteq AMAN3003@53e5a65

here is the final changes in the code wich made read the table without begin and end
sorry for my previous typos

@AMAN3003

@AMAN3003 - the commit you referenced does not appear to have any code changes. We cannot review isolated commits, so you need to follow the documented development procedure and create a pull request from a new branch.

@taldcroft

@taldcroft I am creating a pull request from new branch
also as you refernce in the comment i will like to work on latex documentation
#4295 (comment)
but could you tell where is it located and what should i add in latex documentation

@AMAN3003

@AMAN3003 : No! We do not have to tell you "l where is it located and what should i add in latex documentation" because if we pick out the file to edit and write the text that should go in there, then we might as well do the pull request ourselves - that would be easier and quicker.

So, I suggest that you read the documentation for astropy and sphinx. Then you find out how it works and which file you would have to change. Then, you make a new branch, edit the file with a suggestion for the new text. You open a new pull request for that branch and we will read over it, let you know what we think about it, and make suggestions how you should change your text.

@taldcroft

@taldcroft Sorry for the delay
AMAN3003@1b5267e
this time changes are clearly visible

@AMAN3003

@AMAN3003 : Different projects have different ways to do things. However, in astropy, we don't comment on individual commits, we comment on pull requests.
Do you know how to open a pull request from your code?
When you are on the github astropy website and you click on the "code" tab on the top left of the github website, there should be a yellow field that says "compare and pull request".
Alternatively, you go to "pull requests" and click the green button that says "new pull request".

On the next page, you fill in some details, e.g. a title and a short description. Then you click "create pull request".

@AMAN3003

@AMAN3003 - the Astropy project coordinators would like to communicate with you in private. Could you send an email to [email protected]?

Fix reading of Latex tables with ``\tabular`` tag in the first line

I'm still getting the same error when trying to read a table using python3. It looks like some of the problem is a python2-to-3 issue?

If I first load the file into a string, then do encode('unicode-escape'), then I can avoid the error:

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

Then I'm getting the new error (not sure if related):

File "/Users/lzkelley/Research/catalogs/astrocats/astrocats/blackholes/tasks/wevers_2017__1706_08965.py", line 79, in do_wevers_2017
    data = ap.table.Table.read(data.encode('unicode-escape'), format='ascii.latex')
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/table/table.py", line 2450, in read
    return io_registry.read(cls, *args, **kwargs)
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/registry.py", line 468, in read
    data = reader(*args, **kwargs)
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/connect.py", line 37, in io_read
    return read(filename, format=format, **kwargs)
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/ui.py", line 350, in read
    dat = reader.read(table)
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/core.py", line 1147, in read
    self.data.get_data_lines(self.lines)
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/core.py", line 711, in get_data_lines
    data_lines = self.process_lines(lines)
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/core.py", line 704, in process_lines
    return [x for x in nonblank_lines if not re_comment.match(x)]
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/core.py", line 704, in <listcomp>
    return [x for x in nonblank_lines if not re_comment.match(x)]
  File "/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/core.py", line 701, in <genexpr>
    nonblank_lines = (x for x in lines if x.strip())
AttributeError: 'int' object has no attribute 'strip'

I'll try to take a closer look at this when I can... but are the builds/tests still working for this module (under either python2 or python3)?

All astropy tests (incl the tests for the latex reader) are run for several versions of python 2 and 3 every time somebody opens a pull request or pushed a commit to any part of astropy. On average that happens a few times per day. Thus, I'm fairly confident that that tests are passing. However, the tests might not be complete and the error could be triggered in some situation that's not covered in the tests. If it is, please add a test for the future!

@lzkelley

@lzkelley Do you have any unicode symbols in the table? Could you try to make a copy of your table, manually delete the unicode in the text editor and try again to check if this has to do with the encoding or is an unrelated problem?
We never implemented unicode handling on python 2 for this module, so I would not be surprised if it breaks in all sorts of ways and unicode is not tested on python 3 either as fas as I know.

I think I found it... Notice the spacing at the beginning of lines

test_succeed.tex:

\begin{tabular}{cc}
  AA & BB \\
  cc & dd \\
\end{tabular}

In [178]: ap.table.Table.read('test_succeed.tex')
Out[178]:
<Table length=1>
 AA   BB
str2 str2
---- ----
  cc   dd

test_fail.tex:
(spaces before \begin and \end)

 \begin{tabular}{cc}
  AA & BB \\
  cc & dd \\
 \end{tabular}

In [179]: ap.table.Table.read('test_fail.tex')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-179-223b9272d43b> in <module>()
----> 1 ap.table.Table.read('test_fail.tex')

/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/table/table.py in read(cls, *args, **kwargs)
   2448         passed through to the underlying data reader (e.g. `~astropy.io.ascii.read`).
   2449         """
-> 2450         return io_registry.read(cls, *args, **kwargs)
   2451
   2452     def write(self, *args, **kwargs):

/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/registry.py in read(cls, *args, **kwargs)
    466
    467         reader = get_reader(format, cls)
--> 468         data = reader(*args, **kwargs)
    469
    470         if not isinstance(data, cls):

/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/connect.py in io_read(format, filename, **kwargs)
     35     from .ui import read
     36     format = re.sub(r'^ascii\.', '', format)
---> 37     return read(filename, format=format, **kwargs)
     38
     39

/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/ui.py in read(table, guess, **kwargs)
    348                                              ' with fast (no guessing)'})
    349         else:
--> 350             dat = reader.read(table)
    351             _read_trace.append({'kwargs': new_kwargs,
    352                                 'status': 'Success with specified Reader class '

/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/core.py in read(self, table)
   1145
   1146         # Set self.data.data_lines to a slice of lines contain the data rows
-> 1147         self.data.get_data_lines(self.lines)
   1148
   1149         # Extract table meta values (e.g. keywords, comments, etc).  Updates self.meta.

/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/core.py in get_data_lines(self, lines)
    710         table data values."""
    711         data_lines = self.process_lines(lines)
--> 712         start_line = _get_line_index(self.start_line, data_lines)
    713         end_line = _get_line_index(self.end_line, data_lines)
    714

/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/core.py in _get_line_index(line_or_func, lines)
    495
    496     if hasattr(line_or_func, '__call__'):
--> 497         return line_or_func(lines)
    498     elif line_or_func:
    499         if line_or_func >= 0:

/Users/lzkelley/anaconda3/lib/python3.5/site-packages/astropy/io/ascii/latex.py in start_line(self, lines)
    169             return find_latex_line(lines, self.data_start)
    170         else:
--> 171             return self.header.start_line(lines) + 1
    172
    173     def end_line(self, lines):

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

when there's whitespace before the \begin or \end statement, the read fails (note: the error is different when there's only space before the \end). Since the indentation doesn't (shouldn't?) matter, perhaps the easiest fix is to strip leading whitespace from each line before processing.
There should probably be better error handling overall when a read fails for whatever reason.
Something about using raw-strings or changing the encoding prevents the error... no idea why.

~~Removing all of the leading whitespace in my actual data file does remove the NoneType error, but continues to the next error (shown above)~~

Nevermind, this part was my fault.

fixed by #17624

Navigation Menu

Repository navigation

Latex reader should accept tables without \begin/\end #4295

Use the tabular tag?

Do we need a `tabular` tag at all?

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Latex reader should accept tables without \begin/\end #4295

Description

Activity

taldcroft commented on Nov 6, 2015

AMAN3003 commented on Feb 10, 2016

AMAN3003 commented on Feb 10, 2016

taldcroft commented on Feb 10, 2016

hamogu commented on Feb 10, 2016

Use the tabular tag?

Do we need a tabular tag at all?

taldcroft commented on Feb 10, 2016

AMAN3003 commented on Feb 10, 2016

taldcroft commented on Feb 10, 2016

AMAN3003 commented on Feb 10, 2016

10 remaining items

AMAN3003 commented on Feb 10, 2016

taldcroft commented on Feb 10, 2016

AMAN3003 commented on Feb 10, 2016

hamogu commented on Feb 10, 2016

AMAN3003 commented on Feb 10, 2016

hamogu commented on Feb 10, 2016

taldcroft commented on Feb 17, 2016

lzkelley commented on Jun 30, 2017

hamogu commented on Jun 30, 2017

hamogu commented on Jun 30, 2017

lzkelley commented on Jun 30, 2017

hamogu commented on Jan 11, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Do we need a `tabular` tag at all?