Repository navigation
Latex reader should accept tables without \begin/\end #4295
Description
While trying to read in a Latex table I copied-and-pasted from an arXiv paper, I learned an oddity: the Latex reader fails with an obscure error if you just put in the contents of a latex table (see stack trace at the end. I see no reason why it shouldn't be possible to have it parse Latex style tables without the \begin{table}/\begin{tabular}{ccccccccccc} and corresponding \end commands. It doesn't even seem to care how many markers you have in \tabular, so it's not using the header for anything. So it should be possible to just past in the table data and have it parse, rather than having to add the \begin \end commands around the content.
This could be construed as part of #1525, although that's a bit more wide-ranging and this is one particular thing in that direction.
Stack trace from attempting to Table.read(...,format='latex') just the content of a Latex table:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-43-e9b3cdb9f6d8> in <module>()
32 """.replace('~','').replace('\,', '').replace('$', '')
33
---> 34 k15_tab2 = table.Table.read(k15_tab2_data.split('\n'), format='latex', names=k15_tab2_colnames, guess=False)
35 k15_tab2
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/table/table.pyc in read(cls, *args, **kwargs)
1896 passed through to the underlying data reader (e.g. `~astropy.io.ascii.read`).
1897 """
-> 1898 return io_registry.read(cls, *args, **kwargs)
1899
1900 def write(self, *args, **kwargs):
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/registry.pyc in read(cls, *args, **kwargs)
330
331 reader = get_reader(format, cls)
--> 332 data = reader(*args, **kwargs)
333
334 if not isinstance(data, cls):
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/connect.pyc in io_read(format, filename, **kwargs)
35 from .ui import read
36 format = re.sub(r'^ascii\.', '', format)
---> 37 return read(filename, format=format, **kwargs)
38
39
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/ui.pyc in read(table, guess, **kwargs)
273 dat = reader.read(table)
274 else:
--> 275 dat = reader.read(table)
276
277 return dat
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/core.pyc in read(self, table)
1038
1039 # Set self.data.data_lines to a slice of lines contain the data rows
-> 1040 self.data.get_data_lines(self.lines)
1041
1042 # Extract table meta values (e.g. keywords, comments, etc). Updates self.meta.
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/core.pyc in get_data_lines(self, lines)
614 table data values."""
615 data_lines = self.process_lines(lines)
--> 616 start_line = _get_line_index(self.start_line, data_lines)
617 end_line = _get_line_index(self.end_line, data_lines)
618
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/core.pyc in _get_line_index(line_or_func, lines)
399
400 if hasattr(line_or_func, '__call__'):
--> 401 return line_or_func(lines)
402 elif line_or_func:
403 if line_or_func >= 0:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/io/ascii/latex.pyc in start_line(self, lines)
151 return find_latex_line(lines, self.data_start)
152 else:
--> 153 return self.header.start_line(lines) + 1
154
155 def end_line(self, lines):
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
Activity
taldcroft commentedon Nov 6, 2015
I think the intent of the LaTeX reader is to allow for a fair bit of non-table cruft (i.e. the paper) around the table elements. So you need something to indicate that a table is starting and ending. But more to the point, isn't is required to have
\begin/end{tabular}around a LaTeX tabular-formatted section? I agree that for a regularlatexformat table there should not be a need for the\begin/end{table}as that isn't required in the latex document. Foraastexit's a different story.Otherwise to handle a bare table with
&delimiter and\\at the end of each line, it's a one-line workaround from your example to strip the\\at the line ends, after which the list of lines will parse withdelimiter='&'.AMAN3003 commentedon Feb 10, 2016
@taldcroft do you want to make latex reader more flexible by treating a single table with '&' delimiter and each line of the table ends with
remove the check for the \begin/end{tabular} is this so which you want to fix
fix for the nonetype error astropy#4295
AMAN3003 commentedon Feb 10, 2016
@taldcroft i have fix the nonetype error. but i need a help .if you want a flexible reading of table then it goes right with one table but what if we had two table .how should i identify that new table has started.should we use a delimeter for new table start.
taldcroft commentedon Feb 10, 2016
@hamogu - do you have an opinion on whether the latex reader could (or should) be updated to use only
\tabulartags to find the table instead of\table? Does it parse required information from the\tabledefinition?hamogu commentedon Feb 10, 2016
Use the tabular tag?
The current LaTeX reader already uses the
\tabulartag, you can read a table without a\tabletag perfectly fine:The bug that turned up for the @eteq example is due to something else: There is a
if linein the code where it should beif line is None. Ifline(the line with the\begin{tabular}) is0then that comparison does not give the result it should have.@AMAN3003 : If you have a fix for the Nonetype error, please make that a pull request. I have a solution, too and will issue a pull request, too and then we can compare our approaches and see what we do here.
Do we need a
tabulartag at all?If I understand @eteq correctly, he want to read tables without even the
\begin{tabular}and\end{tabular}tags, i.e. just the raw latex lines.In principle we can do that too, but in the current form there are two problems:
\begin{tabular}and\end{tabular}to know where a table starts and ends.\begin{table}) then we only way in the current scheme is to try start reading in every line and see if it fails. That's very inefficient.If we want to allow the @eteq use case, too, I suggest that we make a new reader (
BareLatexTabularReader) that inherits fromLatexwith few modifications instart_header,start_lineetc.We can add that reader to the guess list easily because LaTeX tables are so specific that there is little chance of confusion.
Fix reading of Latex tables with ``\tabular`` tag in the first line
taldcroft commentedon Feb 10, 2016
@hamogu - thanks for the explanation and the fix in #4595, that makes sense.
My understanding is that a table is not a latex table without the
\tabulartag. In other words you cannot find tabular data embedded within a well-formed latex file (which runs through latex without errors) that does not have the surrounding\tabular. If so then I don't see any real reason to support the use case of bare table values. We can just tell uses to copy/paste the entire table.AMAN3003 commentedon Feb 10, 2016
@hamogu do you want me to make a separate PR for the issue and add test for it to solve the fix of nonetype
taldcroft commentedon Feb 10, 2016
@AMAN3003 - I think that the solution from @hamogu is good enough, so there is no strong need to make an alternate PR here.
AMAN3003 commentedon Feb 10, 2016
@taldcroft but we can implement as @eteq need as i did it for one table .if you could help .. but i only need to add some function to check if the new table latex have started
10 remaining items
finafix for the nonetype error astropy#4295
AMAN3003 commentedon Feb 10, 2016
@taldcroft @eteq AMAN3003@53e5a65
here is the final changes in the code wich made read the table without begin and end
sorry for my previous typos
taldcroft commentedon Feb 10, 2016
@AMAN3003 - the commit you referenced does not appear to have any code changes. We cannot review isolated commits, so you need to follow the documented development procedure and create a pull request from a new branch.
AMAN3003 commentedon Feb 10, 2016
@taldcroft I am creating a pull request from new branch
also as you refernce in the comment i will like to work on latex documentation
#4295 (comment)
but could you tell where is it located and what should i add in latex documentation
hamogu commentedon Feb 10, 2016
@AMAN3003 : No! We do not have to tell you "l where is it located and what should i add in latex documentation" because if we pick out the file to edit and write the text that should go in there, then we might as well do the pull request ourselves - that would be easier and quicker.
So, I suggest that you read the documentation for astropy and sphinx. Then you find out how it works and which file you would have to change. Then, you make a new branch, edit the file with a suggestion for the new text. You open a new pull request for that branch and we will read over it, let you know what we think about it, and make suggestions how you should change your text.
AMAN3003 commentedon Feb 10, 2016
@taldcroft Sorry for the delay
AMAN3003@1b5267e
this time changes are clearly visible
hamogu commentedon Feb 10, 2016
@AMAN3003 : Different projects have different ways to do things. However, in astropy, we don't comment on individual commits, we comment on pull requests.
Do you know how to open a pull request from your code?
When you are on the github astropy website and you click on the "code" tab on the top left of the github website, there should be a yellow field that says "compare and pull request".
Alternatively, you go to "pull requests" and click the green button that says "new pull request".
On the next page, you fill in some details, e.g. a title and a short description. Then you click "create pull request".
taldcroft commentedon Feb 17, 2016
@AMAN3003 - the Astropy project coordinators would like to communicate with you in private. Could you send an email to [email protected]?
Fix reading of Latex tables with ``\tabular`` tag in the first line
lzkelley commentedon Jun 30, 2017
I'm still getting the same error when trying to read a table using python3. It looks like some of the problem is a python2-to-3 issue?
If I first load the file into a string, then do
encode('unicode-escape'), then I can avoid the error:Then I'm getting the new error (not sure if related):
I'll try to take a closer look at this when I can... but are the builds/tests still working for this module (under either python2 or python3)?
hamogu commentedon Jun 30, 2017
All astropy tests (incl the tests for the latex reader) are run for several versions of python 2 and 3 every time somebody opens a pull request or pushed a commit to any part of astropy. On average that happens a few times per day. Thus, I'm fairly confident that that tests are passing. However, the tests might not be complete and the error could be triggered in some situation that's not covered in the tests. If it is, please add a test for the future!
hamogu commentedon Jun 30, 2017
@lzkelley Do you have any unicode symbols in the table? Could you try to make a copy of your table, manually delete the unicode in the text editor and try again to check if this has to do with the encoding or is an unrelated problem?
We never implemented unicode handling on python 2 for this module, so I would not be surprised if it breaks in all sorts of ways and unicode is not tested on python 3 either as fas as I know.
lzkelley commentedon Jun 30, 2017
I think I found it... Notice the spacing at the beginning of lines
test_succeed.tex:
test_fail.tex:
(spaces before
\beginand\end)\beginor\endstatement, the read fails (note: the error is different when there's only space before the\end). Since the indentation doesn't (shouldn't?) matter, perhaps the easiest fix is to strip leading whitespace from each line before processing.Removing all of the leading whitespace in my actual data file does remove theNoneTypeerror, but continues to the next error (shown above)Nevermind, this part was my fault.
hamogu commentedon Jan 11, 2025
fixed by #17624