Skip to content

RuntimeError in astropy.io.fits in del_col test - #4015

Merged
embray merged 1 commit into
astropy:masterfrom
embray:fits/issue-4015
Jul 29, 2015
Merged

embray merged 1 commit into
astropy:masterfrom
embray:fits/issue-4015

Conversation

@embray

@embray embray commented Jul 29, 2015

Copy link
Copy Markdown
Member

@embray - I noticed the following sporadic failures in io.fits over in #3909:

==================================== ERRORS ====================================
_____________ ERROR at teardown of TestCore.test_add_del_columns2 ______________

self = <CallInfo when='teardown' exception: File(s) not closed:
  /tmp/astropy-test-wsvFQg/lib.linux-x86_64-2.6/astropy/io/fits/tests/data/tb.fits>
func = <function <lambda> at 0xcce17d0>, when = 'teardown'

 >   ??? 

_pytest.runner:137: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

 >   ??? 

_pytest.runner:124: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

kwargs = {'item': <Function 'test_add_del_columns2'>, 'nextitem': <Function 'test_uint'>}
plugins = [<Session 'lib.linux-x86_64-2.6'>, <_pytest.config.PytestPluginManager object at 0x1991150>, <_pytest.config.Config ob...opy/extern/pytest.py/_pytest.main'>, <module '_pytest.terminal' from 'astropy/extern/pytest.py/_pytest.terminal'>, ...]

 >   ??? 

_pytest.main:161: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <HookCaller 'pytest_runtest_teardown'>
plugins = [<Session 'lib.linux-x86_64-2.6'>, <_pytest.config.PytestPluginManager object at 0x1991150>, <_pytest.config.Config ob...opy/extern/pytest.py/_pytest.main'>, <module '_pytest.terminal' from 'astropy/extern/pytest.py/_pytest.terminal'>, ...]
kwargs = {'item': <Function 'test_add_del_columns2'>, 'nextitem': <Function 'test_uint'>}
methods = [<function pytest_runtest_teardown at 0x18868c0>, <function pdbitem at 0x19f46e0>, <function pytest_runtest_teardown a...2578>, <bound method CaptureManager.pytest_runtest_teardown of <_pytest.capture.CaptureManager instance at 0x2649b90>>]

 >   ??? 

_pytest.core:380: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <HookCaller 'pytest_runtest_teardown'>
methods = [<function pytest_runtest_teardown at 0x18868c0>, <function pdbitem at 0x19f46e0>, <function pytest_runtest_teardown a...2578>, <bound method CaptureManager.pytest_runtest_teardown of <_pytest.capture.CaptureManager instance at 0x2649b90>>]
kwargs = {'item': <Function 'test_add_del_columns2'>, 'nextitem': <Function 'test_uint'>}

 >   ??? 

_pytest.core:387: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <MultiCall 0 results, 3 meths, kwargs={'item': <Function 'test_add_del_columns2'>, 'nextitem': <Function 'test_uint'>}>

 >   ??? 

_pytest.core:288: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <Function 'test_add_del_columns2'>, nextitem = <Function 'test_uint'>

     def pytest_runtest_teardown(item, nextitem): 
         if hasattr(item, 'set_temp_cache'): 
             item.set_temp_cache.__exit__() 
         if hasattr(item, 'set_temp_config'): 
             item.set_temp_config.__exit__() 

         # a "skipped" test will not have been called with 
         # pytest_runtest_setup, so therefore won't have an 
         # "open_files" member 
         if (not item.config.getvalue('open_files') or 
                 not hasattr(item, 'open_files')): 
             return 

         start_open_files = item.open_files 
         del item.open_files 

         open_files = _get_open_file_list() 

         # This works in tandem with the test_open_file_detection test to 
         # ensure that it creates one extra open file. 
         if item.name == 'test_open_file_detection': 
             assert len(start_open_files) + 1 == len(open_files) 
             return 

         not_closed = set() 
         open_files_ignore = item.config.getini('open_files_ignore') 
         for filename in open_files: 
             ignore = False 

             for ignored in open_files_ignore: 
                 if not os.path.isabs(ignored): 
                     if os.path.basename(filename) == ignored: 
                         ignore = True 
                         break 
                 else: 
                     if filename == ignored: 
                         ignore = True 
                         break 

             if ignore: 
                 continue 

             if filename not in start_open_files: 
                 not_closed.add(filename) 

         if len(not_closed): 
             msg = ['File(s) not closed:'] 
             for name in not_closed: 
                 msg.append('  {0}'.format(name)) 
 >           raise AssertionError('\n'.join(msg)) 
 �[31mE           AssertionError: File(s) not closed: 
 �[31mE             /tmp/astropy-test-wsvFQg/lib.linux-x86_64-2.6/astropy/io/fits/tests/data/tb.fits 

astropy/tests/pytest_plugins.py:526: AssertionError
=================================== FAILURES ===================================
________________________ TestCore.test_add_del_columns2 ________________________

self = <astropy.io.fits.tests.test_core.TestCore object at 0xb3f09d0>

     def test_add_del_columns2(self): 
         hdulist = fits.open(self.data('tb.fits')) 
         table = hdulist[1] 
         assert table.data.dtype.names == ('c1', 'c2', 'c3', 'c4') 
         assert table.columns.names == ['c1', 'c2', 'c3', 'c4'] 
 >       table.columns.del_col(str('c1')) 

astropy/io/fits/tests/test_core.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ColDefs(
    name = 'c2'; format = '3A'; disp = 'A3'
    name = 'c3'; format =...40000000000000002; disp = 'G15.7'
    name = 'c4'; format = '1L'; disp = 'L6'
)
col_name = 'c1'

     def del_col(self, col_name): 
         """ 
             Delete (the definition of) one `Column`. 

             col_name : str or int 
                 The column's name or index 
             """ 

         indx = _get_index(self.names, col_name) 
         col = self.columns[indx] 

         del self._arrays[indx] 
         # Obliterate caches of certain things 
         del self.dtype 
         del self._recformats 
         del self._dims 

         del self.columns[indx] 

         col._remove_listener(self) 

         # If this ColDefs is being tracked by a table HDU, inform the HDU (or 
         # any other listeners) that the column has been removed 
         # Just send a reference to self, and the index of the column that was 
         # removed 
 >       self._notify('column_removed', self, indx) 

astropy/io/fits/column.py:1403: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ColDefs(
    name = 'c2'; format = '3A'; disp = 'A3'
    name = 'c3'; format =...40000000000000002; disp = 'G15.7'
    name = 'c4'; format = '1L'; disp = 'L6'
)
notification = 'column_removed'
args = (ColDefs(
    name = 'c2'; format = '3A'; disp = 'A3'
    name = 'c3'; format =...40000000000000002; disp = 'G15.7'
    name = 'c4'; format = '1L'; disp = 'L6'
), 0)
kwargs = {}, method_name = '_update_column_removed'
listener = <astropy.io.fits.hdu.table.BinTableHDU object at 0xb318a10>
method = <bound method BinTableHDU._update_column_removed of <astropy.io.fits.hdu.table.BinTableHDU object at 0xb318a10>>

     def _notify(self, notification, *args, **kwargs): 
         """ 
             Notify all listeners of some particular state change by calling their 
             ``_update_<notification>`` method with the given ``*args`` and 
             ``**kwargs``. 

             The notification does not by default include the object that actually 
             changed (``self``), but it certainly may if required. 
             """ 

         if self._listeners is None: 
             return 

         method_name = '_update_{0}'.format(notification) 
 >       for listener in self._listeners.itervaluerefs(): 
             listener = listener()  # dereference weakref 
 �[31mE           RuntimeError: dictionary changed size during iteration 

astropy/io/fits/util.py:137: RuntimeError
________ TestColumnFunctions.test_column_attribute_change_after_removal ________

self = <astropy.io.fits.tests.test_table.TestColumnFunctions object at 0xd694f90>

     def test_column_attribute_change_after_removal(self): 
         """ 
             This is a test of the column attribute change notification system. 

             After a column has been removed from a table (but other references 
             are kept to that same column) changes to that column's attributes 
             should not trigger a notification on the table it was removed from. 
             """ 

         # One way we can check this is to ensure there are no further changes 
         # to the header 
         table = fits.BinTableHDU.from_columns([ 
             fits.Column('a', format='D'), 
             fits.Column('b', format='D')]) 

         b = table.columns['b'] 

 >       table.columns.del_col('b') 

astropy/io/fits/tests/test_table.py:2587: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ColDefs(
    name = 'a'; format = 'D'
), col_name = 'b'

     def del_col(self, col_name): 
         """ 
             Delete (the definition of) one `Column`. 

             col_name : str or int 
                 The column's name or index 
             """ 

         indx = _get_index(self.names, col_name) 
         col = self.columns[indx] 

         del self._arrays[indx] 
         # Obliterate caches of certain things 
         del self.dtype 
         del self._recformats 
         del self._dims 

         del self.columns[indx] 

         col._remove_listener(self) 

         # If this ColDefs is being tracked by a table HDU, inform the HDU (or 
         # any other listeners) that the column has been removed 
         # Just send a reference to self, and the index of the column that was 
         # removed 
 >       self._notify('column_removed', self, indx) 

astropy/io/fits/column.py:1403: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ColDefs(
    name = 'a'; format = 'D'
), notification = 'column_removed'
args = (ColDefs(
    name = 'a'; format = 'D'
), 1), kwargs = {}
method_name = '_update_column_removed'
listener = <astropy.io.fits.hdu.table.BinTableHDU object at 0xd694cd0>
method = <bound method BinTableHDU._update_column_removed of <astropy.io.fits.hdu.table.BinTableHDU object at 0xd694cd0>>

     def _notify(self, notification, *args, **kwargs): 
         """ 
             Notify all listeners of some particular state change by calling their 
             ``_update_<notification>`` method with the given ``*args`` and 
             ``**kwargs``. 

             The notification does not by default include the object that actually 
             changed (``self``), but it certainly may if required. 
             """ 

         if self._listeners is None: 
             return 

         method_name = '_update_{0}'.format(notification) 
 >       for listener in self._listeners.itervaluerefs(): 
             listener = listener()  # dereference weakref 
 �[31mE           RuntimeError: dictionary changed size during iteration 

astropy/io/fits/util.py:137: RuntimeError
 �[31m 2 failed, 9136 passed, 390 skipped, 75 xfailed, 2 xpassed, 1 error in 415.64 seconds  
travis_time:end:02867130:start=1438115296196612299,finish=1438115804923683841,duration=508727071542
�[0K
�[31;1mThe command "$MAIN_CMD $SETUP_CMD" exited with 1. 

Done. Your build exited with 1.

@embray embray added Bug Affects-dev PRs and issues that do not impact an existing Astropy release labels Jul 29, 2015
@embray

embray commented Jul 29, 2015

Copy link
Copy Markdown
Member

This would be related to #4012 which I just merged...

@embray embray added this to the v1.1.0 milestone Jul 29, 2015
@embray

embray commented Jul 29, 2015

Copy link
Copy Markdown
Member

That's tricky--I guess if you're looping over the values of a WeakValueDictionary and then doing some complicated action that possibly involves other values in that dict (which is entirely likely in cases like removing columns from a table), then you could wind up with the values dropping out of the dict mid-iteration as their reference counts go to zero.

Incidentally I don't think --open-files has anything directly to do with this. If it's showing up when using it that would just be a coincidence.

@embray embray changed the title Failures with --open-files in astropy.io.fits RuntimeError in astropy.io.fits in del_col test Jul 29, 2015
…ation can themselves cause those listeners or others to be deleted resulting in values being dropped out of the dict. Fixes astropy#4015
@embray

embray commented Jul 29, 2015

Copy link
Copy Markdown
Member

This should un-break CI builds.

embray added a commit that referenced this pull request Jul 29, 2015
RuntimeError in astropy.io.fits in del_col test
@embray
embray merged commit a4a93ff into astropy:master Jul 29, 2015
@embray
embray deleted the fits/issue-4015 branch July 29, 2015 19:44
@embray embray modified the milestones: v1.0.4, v1.1.0 Aug 5, 2015
@embray

embray commented Aug 5, 2015

Copy link
Copy Markdown
Member

Oops, turns out this needs to be in the v1.0.x branch as well.

embray added a commit that referenced this pull request Aug 5, 2015
RuntimeError in astropy.io.fits in del_col test
embray added a commit that referenced this pull request Aug 7, 2015
RuntimeError in astropy.io.fits in del_col test
dhomeier pushed a commit to dhomeier/astropy that referenced this pull request Aug 11, 2015
…ation can themselves cause those listeners or others to be deleted resulting in values being dropped out of the dict. Fixes astropy#4015
embray added a commit that referenced this pull request Aug 11, 2015
RuntimeError in astropy.io.fits in del_col test
embray added a commit that referenced this pull request Aug 11, 2015
RuntimeError in astropy.io.fits in del_col test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Affects-dev PRs and issues that do not impact an existing Astropy release Bug io.fits

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants