Error reading big CNT file

MNE Group:
I tried to read several cnt files of different sizes using mne.io.read_raw_cnt.

data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None, stim_channel=True)

If the cnt file is lower than 2G, the file can be successfully loaded.
However, if the file size exceeds 2G, I will receive the following error:

<ipython-input-25-acf0568a07e0>:1: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None, stim_channel=True)
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-25-acf0568a07e0> in <module>
----> 1 data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None, stim_channel=True)

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in read_raw_cnt(input_fname, montage, eog, misc, ecg, emg, data_format, date_format, preload, stim_channel, verbose)
    176     return RawCNT(input_fname, montage=montage, eog=eog, misc=misc, ecg=ecg,
    177                   emg=emg, data_format=data_format, date_format=date_format,
--> 178                   preload=preload, stim_channel=stim_channel, verbose=verbose)
    179
    180

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in __init__(self, input_fname, montage, eog, misc, ecg, emg, data_format, date_format, preload, stim_channel, verbose)
    430         input_fname = path.abspath(input_fname)
    431         info, cnt_info = _get_cnt_info(input_fname, eog, ecg, emg, misc,
--> 432                                        data_format, _date_format, stim_channel)
    433         last_samps = [cnt_info['n_samples'] - 1]
    434         _check_update_montage(info, montage)

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in _get_cnt_info(input_fname, eog, ecg, emg, misc, data_format, date_format, stim_channel_toggle)
    293         if stim_channel_toggle:
    294             data_format = 'int32' if n_bytes == 4 else 'int16'
--> 295             annot = _read_annotations_cnt(input_fname, data_format=data_format)
    296             events = (np.stack((annot.onset * sfreq,
    297                                 annot.duration * sfreq,

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in _read_annotations_cnt(fname, data_format)
     68
     69     with open(fname, 'rb') as fid:
---> 70         teeg = _read_teeg(fid, teeg_offset=event_table_pos)
     71
     72     event_parser = _get_event_parser(event_type=teeg.event_type)

~/.local/lib/python3.5/site-packages/mne/io/cnt/_utils.py in _read_teeg(f, teeg_offset)
     26     teeg_parser = Struct('<Bll')
     27
---> 28     f.seek(teeg_offset)
     29     return Teeg(*teeg_parser.unpack(f.read(teeg_parser.size)))
     30

OSError: [Errno 22] Invalid argument

Also:
If I use:

data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None)

I got different error:

<ipython-input-30-dbbb56936797>:1: FutureWarning: The parameter `stim_channel` controlling the stim channel synthesis has not been specified. In 0.18 it defaults to True but will change to False in 0.19 (when no stim channel synthesis will be allowed) and be removed in 0.20; migrate code to use `stim_channel=False` and :func:`mne.events_from_annotations` or set `stim_channel=True` to avoid this warning.
  data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None)
<ipython-input-30-dbbb56936797>:1: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None)
Reading 0 ... 4528779  =      0.000 ...  4528.779 secs...
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-30-dbbb56936797> in <module>
----> 1 data = mne.io.read_raw_cnt(test.cnt', preload=True, montage=None)

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in read_raw_cnt(input_fname, montage, eog, misc, ecg, emg, data_format, date_format, preload, stim_channel, verbose)
    176     return RawCNT(input_fname, montage=montage, eog=eog, misc=misc, ecg=ecg,
    177                   emg=emg, data_format=data_format, date_format=date_format,
--> 178                   preload=preload, stim_channel=stim_channel, verbose=verbose)
    179
    180

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in __init__(self, input_fname, montage, eog, misc, ecg, emg, data_format, date_format, preload, stim_channel, verbose)
    439         data_format = 'int32' if cnt_info['n_bytes'] == 4 else 'int16'
    440         self.set_annotations(
--> 441             _read_annotations_cnt(input_fname, data_format=data_format))
    442
    443     @verbose

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in _read_annotations_cnt(fname, data_format)
     68
     69     with open(fname, 'rb') as fid:
---> 70         teeg = _read_teeg(fid, teeg_offset=event_table_pos)
     71
     72     event_parser = _get_event_parser(event_type=teeg.event_type)

~/.local/lib/python3.5/site-packages/mne/io/cnt/_utils.py in _read_teeg(f, teeg_offset)
     26     teeg_parser = Struct('<Bll')
     27
---> 28     f.seek(teeg_offset)
     29     return Teeg(*teeg_parser.unpack(f.read(teeg_parser.size)))
     30

OSError: [Errno 22] Invalid argument

Here is my mne info

mne.sys_info()
Platform:      Linux-4.4.0-151-generic-x86_64-with-Ubuntu-16.04-xenial
Python:        3.5.2 (default, Nov 12 2018, 13:43:14)  [GCC 5.4.0 20160609]
Executable:    /usr/bin/python3
CPU:           x86_64: 12 cores
Memory:        Unavailable (requires "psutil" package)
mne:           0.18.1
numpy:         1.16.3 {blas=openblas, lapack=openblas}
scipy:         1.3.0
matplotlib:    3.0.1 {backend=TkAgg}

sklearn:       0.19.1
nibabel:       Not found
mayavi:        Not found
cupy:          Not found
pandas:        0.22.0
dipy:          Not found

Since the CNT file is huge, which means the samples section in the CNT file (ref this: Structure of CNT file ) is also huge.

And mne choose to read the event table position from the SETUP section, which read as a long (32 bit) integer.

with open(fname, 'rb') as fid:
        ....
        fid.seek(SETUP_EVENTTABLEPOS_OFFSET)
        (event_table_pos,) = np.frombuffer(fid.read(4), dtype='<i4')`

As a result, the maximum length of the CNT file mne supports is equals to :

maximum_seconds = (np.iinfo(np.int32).max - 900 - 75 * n_channels) / sfreq / n_channels / 4

"900" is the size of the SETUP section, 75*n_channels is the size of ELECTLOC, and "4" is the bytes each sample point takes.

For the situation n_channels=62 and sfreq=1000, the result is 8659 seconds (about 2.4 hours). When the length of your CNT file is too long that the value of EVENT_TABLE_POS will overflow.

To solve this problem, you need to change the way to obtain the value of EVENT_TABLE_POS: calculate it using follow formula instead of read it from SETUP section:

--        # fid.seek(SETUP_EVENTTABLEPOS_OFFSET)         # Line 66
--        # (event_table_pos,) = np.frombuffer(fid.read(4), dtype='<i4')  # Line 67

++        fid.seek(864)
++        (n_samples,) = np.frombuffer(fid.read(4), dtype='<i4')
++        event_table_pos = 900 + 75 * int(n_channels) + 4 * int(n_channels) * int(n_samples)

--          event_offset = np.fromfile(fid, dtype='<i4', count=1)[0]     # Line 242
++        event_offset = 900 + 75 * int(n_channels) + 4 * int(n_channels) * int(n_samples)

any chance you can open a PR @AimerLee97 ? thx

…

Thank you all. It works!
However, the author said in his code that '# Header has a field for number of samples, but it does not seem to be too reliable. That's why we have option for setting n_bytes manually.' So I want to know if this update is stable. Because you use the nsamples in header to calculate the correct event_offset.

@AimerLee

@AimerLee nice catch. I guess I broke it in #6047. I'll open a PR.

can anyone share (privately or publicly) one of those long CNT files ??

When trying these large files, the code I'm reading in Teeg field of TEEG is not 1 or 2 is a random number (usually 129 or 116) our codebase allows for 3 as well. Is there any extension that I'm not aware of with different structures or am I doing something wrong?

typedef struct {
   char Teeg;                   /* Either 1 or 2                           */
   long Size;                   /* Total length of all the events          */
   long Offset;                 /* Hopefully always 0                      */
} TEEG;

@massich

@massich Please download the cnt file from the following link:
https://bcmi.cloud:5001/sharing/ezXci1zax
The size of the cnt file is 2.23G. And it has multiple triggers of integer types named 1, 11, 2, 22.

@LM-thinking thx for sharing the file, #6537 seems to be working, can you double check?

Hi all, It seems that I'm still getting this error reading cnt files.
Any idea why ?
Thanks !

@sKinreich

@sKinreich are you sure you are using the latest mne version ? Is so can you share the file privately ?

Hi agramfort,

Thank you for your reply !!!
I’m using version 0.22 of MNE.
How can I share the file privately ? can you send me your email address ? ([email protected])
Thanks you again for any help.

Sivan

Navigation Menu

Repository navigation

Error reading big CNT file #6535

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Error reading big CNT file #6535

Description

Activity

AimerLee commented on Jul 7, 2019

agramfort commented on Jul 7, 2019

emo-farmer commented on Jul 8, 2019

massich commented on Jul 8, 2019

massich commented on Jul 9, 2019

massich commented on Jul 9, 2019

emo-farmer commented on Jul 10, 2019

massich commented on Jul 10, 2019

sKinreich commented on Feb 26, 2021

agramfort commented on Feb 27, 2021

sKinreich commented on Feb 27, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions