Skip to content

loadtxt support for blocks of data (Trac #1107) #1705

Closed
Closed
loadtxt support for blocks of data (Trac #1107)#1705
Labels
01 - EnhancementPatchcomponent: numpy.lib

Description

@thouis
thouis
opened on Oct 19, 2012

Original ticket http://projects.scipy.org/numpy/ticket/1107 on 2009-05-13 by trac user ctenney, assigned to unknown.

Problem:
I wanted to be able to easily read multiple blocks of array data into Numpy from a single text file (e.g. xyz positions of a collection of atoms at different times). I was unable to find an existing tool to accomplish this.

Solution:
I modified the loadtxt function in numpy/lib/io.py (diff pasted below).

Summary:
I added two new parameter options: 'numcols' and 'datablocks'. The defaults were chosen to reproduce previous behavior.

'numcols' is an optional int that defaults to None. If set, 'numcols' specifies the number of columns in a valid data line. This is useful for filtering out non-comment header info (particularly from files with multiple blocks of data).

'datablocks' is a boolean that defaults to False. If True, data blocks within the file are assumed to be separated by two or more non-data lines (e.g. blank, comment, or invalid 'numcols'). An additional leading dimension is added to each returned array if two or more data blocks are present. (e.g. If X = np.loadtxt(..., datablocks=True), then X[0] is the first block, X[1] the second, et cetera; and X.ravel returns the same result as np.loadtxt(..., datablocks=False).) The result is unchanged for single-block data sets.

Notes:

  • I successfully ran the changes against numpy/lib/tests/test_io.py.
  • When faced with multi-block datasets, use of either or both of these options makes plotting in matplotlib almost as easy as plotting in gnuplot, which is particularly handy for data exploration via ipython -pylab.

diff for numpy/lib/io.py for v1.3.0:

332,333c332
<             skiprows=0, usecols=None, unpack=False,
<             numcols=None, datablocks=False):

---
>             skiprows=0, usecols=None, unpack=False):
368,379d366
<     numcols : int
<         If set, 'numcols' specifies the number of columns in a valid data line.
<         This is useful for filtering out non-comment header info (particularly
<         from files with multiple blocks of data).
<     datablocks : bool
<         If True, data blocks within the file are assumed to be separated by two
<         or more non-data lines (e.g. blank, comment, or invalid 'numcols'). An
<         additional leading dimension is added to each returned array if two or
<         more data blocks are present. (e.g. If X = np.loadtxt(...,
<         datablocks=True), then X[0] is the first block, X[1] the second, et
<         cetera; and X.ravel returns the same result as np.loadtxt(...,
<         datablocks=False).) The result is unchanged for single-block data sets.
471,472d457
<             if numcols and len(first_vals) != numcols:
<                 first_vals = None
495,497d479
<         X.append([])
<         block = 0
<         blanks = 0
501d482
<                 blanks += 1
503,509d483
<             elif numcols and len(vals) != numcols:
<                 blanks += 1
<                 continue
<             elif datablocks and blanks >= 2:
<                 block += 1
<                 X.append([])
<             blanks = 0
515,516c489
<             X[block].append(tuple([conv(val) for (conv, val) in zip(converters, vals)]))
< 

---
>             X.append(tuple([conv(val) for (conv, val) in zip(converters, vals)]))
536c509
<         return np.rollaxis(X, -1, 0)

---
>         return X.T

Activity

  1. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    trac user ctenney wrote on 2009-05-13

    Replying to [ticket:1107 ctenney]:

    It just dawned on me that each block of data needs to be the same size. Quite the brain-fade... No 'delete ticket' button?

    While it works for predictably consistent data, this deficiency reduces the general utility and robustness of my 'datablocks' option. Handling more pathological cases will likely require a list rather than an array approach, which puts it outside the immediate numpy realm.

  2. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    trac user ctenney wrote on 2009-05-15

    Replying to [ticket:1107 ctenney]:

    To allow datablocks to differ in size, I changed my strategy. If requested, I now return (as a tuple) the original, unmodified data array, plus a 1-D integer array listing the datablock from which each row in the data array originates. The second array can be used to filter the first by datablock. The existing tests in numpy/lib/test/test_io.py run successfully.

    Again, if there already exists something somewhere for easily pulling multi-block data into Numpy from a file, feel free to let me know. Several hours of searching turned up nothing, but I certainly could have missed a magic keyword.

    The new patch is attached. It is pasted below for ease of reference.

    332c332,333
<             skiprows=0, usecols=None, unpack=False):
---
>             skiprows=0, usecols=None, unpack=False,
>             numcols=None, datablocks=False):
366a368,379
>     numcols : int
>         If set, 'numcols' specifies the number of columns in a valid data line.
>         This is useful for filtering out non-comment header info (particularly
>         from files with multiple blocks of data).
>     datablocks : bool
>         If True, individual datablocks within the input file are assumed to be
>         separated by two or more non-data lines (e.g. blank, comment, or
>         invalid 'numcols'). Instead of returning a single array, the function
>         will now return a two-element tuple composed of the original data array
>         plus a 1-D integer array listing the block number for each row of data.
>         (e.g. If X,Y = np.loadtxt(..., unpack=False, datablocks=True), then the
>         value in Y[i] specifies the datablock from which X[i] originated.)
457a471,472
>             if numcols and len(first_vals) != numcols:
>                 first_vals = None
479a495,497
>         block = 0
>         blocklist = []
>         blanks = 0
482a501
>                 blanks += 1
483a503,511
>             elif numcols and len(vals) != numcols:
>                 blanks += 1
>                 continue
> 
>             if datablocks:
>                 if blanks >= 2:
>                     block += 1
>                 blocklist.append(block)
>                 blanks = 0
508,509c536,541
<     if unpack:
<         return X.T
---
>     if datablocks:
>         blocklist = np.array(blocklist, dtype=int)
>         if unpack:
>             return X.T, blocklist
>         else:
>             return X, blocklist
511c543,546
<         return X
---
>         if unpack:
>             return X.T
>         else:
>             return X

  3. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    @pv wrote on 2009-05-16

    Quick note: please attach patches as files, and in the unified diff format (ie. use "diff -u"); then they will also have a preview available.

  4. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    Attachment added by trac user ctenney on 2009-05-16: numpy-lib-io_datablocks.patch

  5. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    trac user ctenney wrote on 2009-05-16

    Replying to [comment:3 pv]:

    Quick note: please attach patches as files, and in the unified diff format (ie. use "diff -u"); then they will also have a preview available.

    Thank you. The attached patch has been updated to a version created via 'diff -u'.

  6. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    Milestone changed to Unscheduled by @cournape on 2009-11-25

  7. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    @charris wrote on 2011-01-23

    Needs tests.

  8. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    Milestone changed to 1.6.0 by @rgommers on 2011-03-31

  9. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    @rgommers wrote on 2011-04-02

    It seems like this functionality should go into genfromtxt instead. loadtxt is a simple interface, genfromtxt a very flexible and complete one.

  10. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    @rgommers wrote on 2011-04-17

    Too late for 1.6.0 now, moving to 2.0 milestone. #2326 is related.

  11. numpy-gitbot commented on Oct 23, 2012

    @numpy-gitbot
    numpy-gitbot
    on Oct 23, 2012

    Milestone changed to NumPy 2.0 by @rgommers on 2011-04-17

  12. charris
    added
    Patch
    on Feb 18, 2014
  13. mattip
    removed
    priority: normal
    on Oct 21, 2018
  14. WarrenWeckesser
    removed this from the NumPy 2.0 milestone on Apr 16, 2020

  15. WarrenWeckesser commented on Apr 16, 2020

    @WarrenWeckesser
    WarrenWeckesser
    on Apr 16, 2020
    Member

    The change proposed here seems too specialized to actually incorporate into loadtxt or genfomtxt. Does anyone mind if we close this?

  16. seberg commented on Apr 16, 2020

    @seberg
    seberg
    on Apr 16, 2020
    Member

    No, I don't mind. Its a super old issue. Lets close it for now, if someone wants to bring it up again, we can reopen or just open a new issue.

  17. seberg
    closed this as completedon Apr 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    01 - EnhancementPatchcomponent: numpy.lib

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions