astropy Table to pandas DataFrame #2804

Activity

1. astrofrog

   added

   Feature Request

   on Jul 30, 2014
2. eteq commented on Jul 30, 2014

   

   eteq

   on Jul 30, 2014

   Member

   More actions

   @taldcroft - have you thought much about this? An idea here might be a Table.to_pandas method (which of course fails if pandas is not installed), and perhaps also a Table.from_pandas class method (although Table(pandasdataframe) might be a more natural way?)
3. taldcroft commented on Jul 30, 2014

   

   taldcroft

   on Jul 30, 2014

   Member

   More actions

   For unmasked tables do:

   from astropy.table import Table
   from pandas import DataFrame
   
   t = Table(...)
   df = DataFrame(...)
   
   df_t = DataFrame(np.array(t))
   t_df = Table(np.array(df))
   

   If you have an astropy masked table this will not propagate the masked values, and unfortunately DataFrame does not appear to accept a numpy masked array as input. I don't know enough about Pandas to know the right way to do this. It looks like you need to set missing values to np.nan, but that's just a guess.
4. taldcroft commented on Jul 30, 2014

   

   taldcroft

   on Jul 30, 2014

   Member

   More actions

   @eteq - this has definitely been discussed. I thought there was an issue open but apparently not. From our side it would just be Table(pandas_dataframe) and probably Table.to_pandas(). It would be nice if pandas provided a conversion hook like __array__. Does anyone know if something like that exists?
5. eteq commented on Jul 30, 2014

   

   eteq

   on Jul 30, 2014

   Member

   More actions

   @taldcroft - this may be sort of a bug in pandas, but note the following:

   >>> f = pandas.DataFrame(randn(100,2),columns=['a','b'])
   >>> t = astropy.table.Table(np.array(f))
   >>> t.colnames
    ['col0', 'col1']
   

   It looks like array doesn't correctly convert a DataFrame to a numpy structured array... But the other way does work:

   >>> t = astropy.table.Table(data=randn(100,2),names=['a','b'])
   >>> f = pandas.DataFrame(np.array(t))
   >>> f.columns
   Index([u'a', u'b'], dtype='object')
   

   So you can do astropy -> pandas easily, but not vice-versa... so perhaps a reason to at least accept it in the Table initializer.
6. cdeil commented on Sep 30, 2014

   

   cdeil

   on Sep 30, 2014

   Member

   More actions

   There's an open issue that's very similar to this one: #1156.
7. taldcroft

   closed this as completedin #3504on Apr 25, 2015
8. embray

   added this to the v1.1.0 milestone on May 4, 2015
9. Malewhaler commented on Oct 9, 2022

   

   Malewhaler

   on Oct 9, 2022

   More actions

   @taldcroft - this may be sort of a bug in pandas, but note the following:

   >>> f = pandas.DataFrame(randn(100,2),columns=['a','b'])
   >>> t = astropy.table.Table(np.array(f))
   >>> t.colnames
    ['col0', 'col1']
   

   It looks like array doesn't correctly convert a DataFrame to a numpy structured array... But the other way does work:

   >>> t = astropy.table.Table(data=randn(100,2),names=['a','b'])
   >>> f = pandas.DataFrame(np.array(t))
   >>> f.columns
   Index([u'a', u'b'], dtype='object')
   

   So you can do astropy -> pandas easily, but not vice-versa... so perhaps a reason to at least accept it in the Table initializer.

   A way from DataFrame to Table:
   t=astropy.table.Table(f.values,names=f.columns)
10. taldcroft commented on Oct 9, 2022

    

    taldcroft

    on Oct 9, 2022

    Member

    More actions

    A way from DataFrame to Table:
    t=astropy.table.Table(f.values,names=f.columns)

    @Malewhaler - the preferred way from DataFrame to Table is below, which has some options and deals with a few subtleties.

    t = astropy.table.Table.from_pandas(f)