Repository navigation
Concurrent import of astropy occasionally fails with likely race condition on config files. #7290
Description
When trying to import astropy from an MPI application ( with 128 ranks), the application crashes during first import randomly with a file system error.
It appears that some astropy instances are trying to modify a configuration file while some others are trying to read it.
from astropy.cosmology import Planck13, Planck15, WMAP5, WMAP7, WMAP9
File "/dev/shm/local/lib/python3.6/site-packages/astropy/__init__.py", line 286, in <module>
log = _init_log()
File "/dev/shm/local/lib/python3.6/site-packages/astropy/logger.py", line 97, in _init_log
log._set_defaults()
File "/dev/shm/local/lib/python3.6/site-packages/astropy/logger.py", line 473, in _set_defaults
self.setLevel(conf.log_level)
File "/dev/shm/local/lib/python3.6/site-packages/astropy/config/configuration.py", line 273, in __get__
return self()
File "/dev/shm/local/lib/python3.6/site-packages/astropy/config/configuration.py", line 396, in __call__
sec = get_config(self.module)
File "/dev/shm/local/lib/python3.6/site-packages/astropy/config/configuration.py", line 530, in get_config
cobj = configobj.ConfigObj(cfgfn, interpolation=False)
File "/dev/shm/local/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", line 1228, in __init__
self._load(infile, configspec)
File "/dev/shm/local/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", line 1317, in _load
raise error
astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00\x00\x00\x00\x00\x00\x00\x00
....
Activity
rainwoodman commentedon Mar 14, 2018
I'd add this seem to occur only when astropy is updated. Ideally a library shall not rely on any particular dot files in $HOME to properly function. The parts that rely on dot files are more or less like final, user facing applications. But in the present day the boundary between application and library is blurry.
Anyhow, I suggest the following fix:
This way astropy doesn't try to automatically migrate the configuration file during import. Yet the migration functionality is preserved when it is needed.
j-dr commentedon Aug 14, 2018
Has there been any progress on this? It is a big issue when trying to run any large mpi python jobs that have astropy dependencies.
pllim commentedon Aug 17, 2018
Maybe related to #5899 ?
rainwoodman commentedon Aug 17, 2018
Yes. It is related. Use cases in headless applications will be smoother if the automated loading of file based configurations is opted out by default.
Would a two tier initialization (think matplotlib vs matplotlib.pyplot) be handy for this? After importing the top level package one can modify / load the configuration (think matplotlib.use), but in interactive application one directly import from another namespace which triggers automated backend initialization based on profiles in the file system.
pllim commentedon Aug 17, 2018
@eteq ?
rainwoodman commentedon Mar 16, 2020
This is not fixed by #8237, and #8552 does not solve this issue.
We still have a race condition, because multiple astropy instances still tries to write to the same files.
Two possible solutions:
What about adding a magic environment variable to instruct astropy to use a per-process directory for the config and caches.
We can also try to make all write operations in the config module atomic (at least on POSIX). e.g. first write to a unique temporary file, then renamed to the target file.
pllim commentedon Mar 16, 2020
@rainwoodman , have you tried this with the latest dev version of
astropy? We have some caching problems reported but none related toconfig.rainwoodman commentedon Mar 29, 2020
No, I have not tried with the dev version; we have been using a local workaround that first imports astropy with one python process per file system hosting the astropy config file.
As the code is written currently there still is a race. Thus I am not very positive that the dev version will fix the issue.
The race is between the write at:
https://github.com/astropy/astropy/blob/master/astropy/config/configuration.py#L742
and the read at:
https://github.com/astropy/astropy/blob/master/astropy/config/configuration.py#L701
The read sees a half written file when another process is writing. Without using a lock, the write shall first write to a temporary file on the same filesystem (e.g. the same folder), then do an "atomic rename" to replace the file.
Or did I misread the meaning of cfgfn?
pllim commentedon Mar 30, 2020
pllim commentedon Mar 30, 2020
@rainwoodman , would #10090 fix your problem by avoiding the loading of
astropy.cfgaltogether? Do you want to take it for a spin?dannygoldstein commentedon Apr 27, 2020
I am also experiencing this error and am a little confused about it. Why is there any config-file writing happening at all? Once I have a astropy.cfg and a astropy.version.cfg in my ~/.astropy/config directory, why does anything new need to be written?
dannygoldstein commentedon May 9, 2020
@j-dr did you ever manage to fix this ?
j-dr commentedon May 11, 2020
I have a fix that works at NERSC for docker/shifter images. You just need to run the following at the end of your dockerfile:
Not sure if this will work for any other use cases though.
rainwoodman commentedon May 12, 2020
dannygoldstein commentedon May 12, 2020
rainwoodman commentedon May 12, 2020