Error with dec=',', sep = 'auto' but detected as ',' #4483

Activity

1. MichaelChirico

   added

   fread

   on May 24, 2020
2. st-pasha commented on May 25, 2020

   

   st-pasha

   on May 25, 2020

   Last edited by st-pasha

   Contributor

   More actions

   When dec=',' and sep=',', we should not attempt to use the float parser at all, unless the field is quoted.
   Otherwise we'll end up reading lines like 1,2,3,4 as two numbers 1.2 and 3.4, which is rather insane...
3. MichaelChirico commented on May 26, 2020

   

   MichaelChirico

   on May 26, 2020

   MemberAuthor

   More actions

   rather insane...

   Actually found a file on some Spanish government website for which that's the correct parsing 😂 but yes, indeed insane as a general thing.

   This issue is slightly different. We already block sep==dec:

   data.table/src/fread.c

   Line 1180 in ff3e7d4

   if (args.sep == dec) STOP(_("sep == dec ('%c') is not allowed"), dec);

   Here what happens is, since sep detection comes first, right now we aren't enforcing sep != dec when sep starts out as 'auto' but ends up as ','.

   Maybe the best option is to change here:

   data.table/src/fread.c

   Line 1455 in ff3e7d4

   char seps[]=",|;\t "; // default seps in order of preference. See ?fread.

   to

   char seps[] = dec == ',' ? "|;\t " : ",|;\t ";
   

   i.e. if user has manually specified dec=',', prevent sep='auto' from finding sep=','
4. mattdowle

   added this to the 1.14.1 milestone on Aug 20, 2021
5. mattdowle

   closed this as completedin #4495on Aug 20, 2021
6. jangorecki

   modified the milestones: 1.14.9, 1.15.0 on Oct 29, 2023