allow loadtxt() to read given number of rows (Trac #1731) #2326

Activity

1. numpy-gitbot commented on Oct 23, 2012

   

   numpy-gitbot

   on Oct 23, 2012

   Author

   More actions

   trac user dynetrekk wrote on 2011-03-26

   A suggested implementation of the numrows keyword can be found here. It simply slices the number of lines in the file. One can of course discuss whether one should count blank or commented lines - I believe not, as one would want to be able to relate the line number of a file with the number of rows to read. (If this is desired, one could write a break statement for the for loop.)

   diff -r 2763b87dd7e8 numpy/lib/npyio.py
   --- a/numpy/lib/npyio.py    Fri Mar 25 22:37:19 2011 -0600
   +++ b/numpy/lib/npyio.py    Sat Mar 26 15:40:11 2011 +0100
   @@ -579,7 +579,8 @@
   
   
    def loadtxt(fname, dtype=float, comments='#', delimiter=None,
   -            converters=None, skiprows=0, usecols=None, unpack=False):
   +            converters=None, skiprows=0, numrows=None, usecols=None,
   +            unpack=False):
        """
        Load data from a text file.
   
   @@ -609,6 +610,9 @@
            ``converters = {3: lambda s: float(s or 0)}``.  Default: None.
        skiprows : int, optional
            Skip the first `skiprows` lines; default: 0.
   +    numrows : int, optional
   +        Read only `numrows` lines; The default, None, results in all rows
   +        being read.
        usecols : sequence, optional
            Which columns to read, with 0 being the first.  For example,
            ``usecols = (1,4,5)`` will extract the 2nd, 5th and 6th columns.
   @@ -771,7 +775,10 @@
                converters[i] = conv
   
            # Parse each line, including the first
   -        for i, line in enumerate(itertools.chain([first_line], fh)):
   +        all_lines = itertools.chain([first_line], fh)
   +        if not (numrows is None):
   +            all_lines = itertools.islice(all_lines, numrows)
   +        for i, line in enumerate(all_lines):
                vals = split_line(line)
                if len(vals) == 0:
                    continue
   
2. numpy-gitbot commented on Oct 23, 2012

   

   numpy-gitbot

   on Oct 23, 2012

   Author

   More actions

   @rgommers wrote on 2011-03-31

   A comment on the call signature: new keywords should always be added at the end, not in the middle. If added in the middle it can break users' code when they specify those keywords as positional arguments.
3. numpy-gitbot commented on Oct 23, 2012

   

   numpy-gitbot

   on Oct 23, 2012

   Author

   More actions

   Milestone changed to 1.6.0 by @rgommers on 2011-03-31
4. numpy-gitbot commented on Oct 23, 2012

   

   numpy-gitbot

   on Oct 23, 2012

   Author

   More actions

   @rgommers wrote on 2011-04-03

   Same comment as for #1705: it seems like this functionality should go into genfromtxt instead. loadtxt is a simple interface, genfromtxt a very flexible and complete one.
5. numpy-gitbot commented on Oct 23, 2012

   

   numpy-gitbot

   on Oct 23, 2012

   Author

   More actions

   @rc wrote on 2011-04-04

   On the other hand, there is already skiprows... I mean "not reading" the file till its end seems to me simple enough. With the above patch one would get a nice functionality without extra work, and still get the speed of the simple loadtxt().

   PS: Thanks Paul for looking at this!
6. numpy-gitbot commented on Oct 23, 2012

   

   numpy-gitbot

   on Oct 23, 2012

   Author

   More actions

   @rgommers wrote on 2011-04-17

   Since this is an enhancement and not a bugfix, it's now too late for 1.6.0. Moving to 2.0 milestone.
7. numpy-gitbot commented on Oct 23, 2012

   

   numpy-gitbot

   on Oct 23, 2012

   Author

   More actions

   Milestone changed to NumPy 2.0 by @rgommers on 2011-04-17
8. charris commented on Jan 25, 2015

   

   charris

   on Jan 25, 2015

   Member

   More actions

   Closed by #5496.
9. charris

   closed this as completedon Jan 25, 2015