reading in two large ascii files causes segfault -- macosx

@demitri

When I try to open more than one large-ish (4MB) ascii file in using astropy.io.ascii.read, python segfaults. Both files open fine individually, and only together they have a problem

@demitri @taldcroft

Here's an example

In [26]: test1 = ascii.read("cosmos_3dhst.v4.1.master.RF")

In [27]: test2 = ascii.read("cosmos_3dhst.v4.1.fout")
/Users/alex/code/anaconda/bin/python.app: line 3: 22909 Segmentation fault: 11  /Users/alex/code/anaconda/python.app/Contents/MacOS/python "$@"

@astronomeralex

@astronomeralex - I tried this locally using two 5 Mb ASCII files (100000 rows by 10 cols). This was on MacOSX and Anaconda with Python 2.7 and astropy 0.4.2.

The ascii.read code in question is (for 0.4.2) pure Python until the point of converting into numpy arrays at the end of parsing. If you run this in a debugger (e.g. pdb or within spyder) can you localize where the segfault occurs? Those files are actually small-ish (we noted problems earlier more at the 100Mb or Gb level). The pure-Python reader in 0.4.2 does use a lot more memory than the file size, but presumably memory is not the problem here.

@taldcroft

@taldcroft -- thanks for trying; i assume it works?

I'm in python 3.3.5 in anaconda using astropy 0.4.2. Unfortunately, the debugger doesn't help because its a segfault, not a python error. I can send you the files if that would be helpful?

I was thinking of doing a line trace prior to the crash. If you can send the files then I can try.

@taldcroft

@taldcroft -- the files should be in your inbox

Whatever the problem is I doubt it has to do with the size of the files.

@astronomeralex

@astronomeralex - I tried the files you supplied by email and they read in with no problem. I'm using Python 3.3.5 with numpy 1.9 and astropy 0.4.2, all from Anaconda.

Can you try making a fresh Anaconda? Go to http://conda.pydata.org/miniconda.html, download the mac miniconda installer for python 3 and do something like below from a fresh terminal shell:

bash Miniconda3-3.7.0-MacOSX-x86_64.sh -b -p ~/tmp_conda
export PATH=~/tmp_conda/bin:$PATH
which conda  # just to make sure
conda install astropy
python
>>> from astropy.io import ascii
>>> dat = ascii.read('cosmos_3dhst.v4.1_f125w.galfit')
>>> dat2 = ascii.read('cosmos_3dhst.v4.1_f160w.galfit')

I have similar issues: (here is the code)

from astropy.io import ascii
from astropy.io import registry
import astropy
import numpy
print 'Numpy version:', numpy.__version__
print 'astropy version:', astropy.__version__
print astropy.__file__

g = ascii.read('filters/total_g.dat')
print 'loaded g'
r = ascii.read('filters/total_r.dat')
print 'loaded r'  
u = ascii.read('filters/total_u.dat')
print 'loaded u'
# i = ascii.read('filters/total_i.dat')

Output:

>>> python test_ascii.py

Numpy version: 1.9.1
astropy version: 0.4.2
/usr/local/manual/anaconda/lib/python2.7/site-packages/astropy/__init__.pyc
loaded g
loaded r
loaded u

d-69-91-142-89:astropy_problem rbiswas$ vi test_ascii.py
d-69-91-142-89:astropy_problem rbiswas$ python test_ascii.py
Numpy version: 1.9.1
astropy version: 0.4.2

--------------------
Completes successfully, but uncommenting the last two lines
>>> python test_ascii.py 
/usr/local/manual/anaconda/lib/python2.7/site-packages/astropy/__init__.pyc
loaded g
loaded r
loaded u
Segmentation fault: 11

I am running Darwin Kernel Version 13.4.0 (Mavericks). The problem is not there on Linux (Tried on ubuntu 14.04, or macs with older anaconda and astropy versions. You can get all of this here: https://github.com/rbiswas4/ascii_read_problem

I also tried out the following:
a. installing astropy through virtualenv from scratch on the same machine. In this case the problem disappears. (version 0.4.3)
b. updated anaconda astropy by doing conda update astropy but this does not remove the problem.

@astronomeralex

@astronomeralex - were you using a mac when you first encountered this? And @taldcroft, are you on a mac?

@astronomeralex

yep, I was on a mac.

On Tue, Jan 27, 2015 at 2:13 PM, Erik Tollerud [email protected]
wrote:

@astronomeralex https://github.com/astronomeralex - were you using a
mac when you first encountered this? And @taldcroft
https://github.com/taldcroft, are you on a mac?

—
Reply to this email directly or view it on GitHub
#3280 (comment).

alex hagen
916.425.3581
https://astronomeralex.github.io/

Yes I was on a mac.

@rbiswas4

Somehow it feels like it must be a bad interaction with some other library because it goes away for a fresh install (according to @rbiswas4 comment (a)).

@taldcroft

@taldcroft it could like a bad interaction with some other library, but this bad interaction seems to be present in anaconda installs. The fresh install install which removed the problem was a fresh astropy install through virtualenv without using anaconda at all.

@rbiswas4

@rbiswas4 - can you replicate this with a pristine install of Anaconda? By that I mean download Miniconda-<..>.sh from the continuum downloads site, install into a new directory and then run from the root env of that new installation. Start with a minimal install, basically just python and astropy.

This issue was labeled as Close?. Remove Close? label or this will be closed after 7 days. (This is currently a dry-run.)

Very old issue that no one has reproduced since. There have been other segfaults in io.ascii that have been fixed since. Maybe one of them was related. But this was originally reported against a very old Python version and a very old Astropy version, on a MacOS version that is also obsolete. Since the issue can't be reproduced we should close this one.

@embray

@embray - agreed, closing.

Hah, at least half the bot is back alive... (the threat to close was empty threat). 😜 🤖

Navigation Menu

Repository navigation

reading in two large ascii files causes segfault -- macosx #3280

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

reading in two large ascii files causes segfault -- macosx #3280

Description

Activity

taldcroft commented on Jan 8, 2015

astronomeralex commented on Jan 8, 2015

taldcroft commented on Jan 8, 2015

astronomeralex commented on Jan 11, 2015

embray commented on Jan 13, 2015

taldcroft commented on Jan 13, 2015

rbiswas4 commented on Jan 17, 2015

rbiswas4 commented on Jan 18, 2015

eteq commented on Jan 27, 2015

astronomeralex commented on Jan 27, 2015

taldcroft commented on Jan 27, 2015

taldcroft commented on Jan 27, 2015

rbiswas4 commented on Jan 27, 2015

taldcroft commented on Jan 27, 2015

github-actions commented on Jun 21, 2021

embray commented on Jun 21, 2021

taldcroft commented on Jun 21, 2021

pllim commented on Jun 21, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions