You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As noted by @AmrAS1, there is a performance flaw in the current table.pprint() implementation:
text_type() or some other formatter is called on each individual numpy value, which causes numpy to call an array-to-string method internally for each value.
Hm...after looking into this more carefully, I don't think there's a bug here. The reason why iter_str_vals() was so slow in the fast reader was that csv.writer can do a much faster job with floats or ints than just converting floats or ints manually and then writing with writer.writerow(). When I first noticed how slow iter_str_vals() seemed to be, I just tested the relative performance of str(val) and '{0}'.format(val) where val is a numpy value; my finding that format() was much faster was actually just a fluke because I used the same value repeatedly and apparently string formatting is more efficient when given the same input over and over.
I tried a couple methods quickly to get the speed down, like moving away from a lambda function, but nothing dropped more than 10-15% or so. My guess (or hope) is that the speed of iter_str_vals() probably isn't a huge issue anyway, so maybe leaving the system as flexible as possible is best.
Activity
mdmueller commentedon Aug 30, 2014
Hm...after looking into this more carefully, I don't think there's a bug here. The reason why
iter_str_vals()was so slow in the fast reader was thatcsv.writercan do a much faster job with floats or ints than just converting floats or ints manually and then writing withwriter.writerow(). When I first noticed how slowiter_str_vals()seemed to be, I just tested the relative performance ofstr(val)and'{0}'.format(val)wherevalis a numpy value; my finding thatformat()was much faster was actually just a fluke because I used the same value repeatedly and apparently string formatting is more efficient when given the same input over and over.I tried a couple methods quickly to get the speed down, like moving away from a lambda function, but nothing dropped more than 10-15% or so. My guess (or hope) is that the speed of
iter_str_vals()probably isn't a huge issue anyway, so maybe leaving the system as flexible as possible is best.taldcroft commentedon Aug 31, 2014
OK @AmrAS1 so you are recommending that we close this?
mdmueller commentedon Aug 31, 2014
@taldcroft - Yes, unless anyone else notices something wrong here.