Skip to content

[models] Add NeuroRVQ EEG foundation model - #1218

Merged
bruAristimunha merged 10 commits into
braindecode:masterfrom
lindicaphxag-tech:feat/neurorvq-1090
Oct 6, 2026
Merged

bruAristimunha merged 10 commits into
braindecode:masterfrom
lindicaphxag-tech:feat/neurorvq-1090

Conversation

@lindicaphxag-tech

@lindicaphxag-tech lindicaphxag-tech commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add NeuroRVQ-EEG as a channel-aware Braindecode classifier with the released four-scale temporal patch embedding and shared Transformer encoder.
  • Add a pinned Hugging Face checkpoint loader that validates the expected encoder schema while leaving the task-specific classifier head initialized for the downstream task.
  • Document the 200 Hz / 200-sample patch contract, supported electrode mapping, preprocessing expectations, CC BY-NC 4.0 terms, API entry, model table, and architecture diagram.
  • This contribution covers the EEG foundation model only; the separate NeuroRVQ tokenizer is not included.

Closes #1090

Implementation fidelity

Reference: https://github.com/KonstantinosBarmpas/NeuroRVQ (CC BY-NC 4.0; attribution and noncommercial terms are recorded in the module header and NOTICE.txt). The checkpoint is fetched from the pinned Hugging Face revision d944b87f44ae0ba2923b2f10d0518f23f6803b76.

A CPU parity probe loads the released checkpoint into the reference model and this implementation, copies the task head, and compares logits plus input gradients for a 3-channel, 4-patch input. All 263 shared encoder tensors load; maximum absolute logit and input-gradient differences are both 0.0. The checkpoint loader also rejects missing or unexpected encoder tensors outside the documented pretraining-only heads.

The port also preserves the upstream zero-based spatial-slot convention: create_embedding_ix maps the first electrode to slot 0 and the reference forward path separately pads the CLS position with another 0; a focused regression locks this unusual but checkpoint-relevant behavior.

Licensing / distribution boundary

  • No third-party checkpoint bytes are vendored in this PR. Pretrained weights are downloaded at runtime from the upstream Hugging Face repository at the pinned revision above.
  • The adapted braindecode/models/neurorvq.py module is explicitly marked CC BY-NC 4.0 and is listed in NOTICE.txt, following Braindecode's existing per-file third-party licensing pattern.
  • The loader does not relicense the upstream checkpoint: users remain subject to the NeuroRVQ checkpoint/source noncommercial terms.
  • Tests use locally initialized models unless the pretrained-loader path is explicitly exercised; the package remains usable without downloading the checkpoint.

Validation

  • pytest test/unit_tests/models/test_neurorvq.py test/unit_tests/models/test_integration.py -k NeuroRVQ -q — 21 passed, 3 skipped.
  • pytest test/unit_tests/models/test_return_features.py test/unit_tests/models/test_models.py -k 'NeuroRVQ or completeness_summary_table' -q — 4 passed.
  • pytest test/unit_tests/models/test_integration.py -k 'completeness_summary_table and NeuroRVQ' -q — 1 passed.
  • ruff check braindecode/models/neurorvq.py test/unit_tests/models/test_neurorvq.py — passed.
  • ruff format --check braindecode/models/neurorvq.py test/unit_tests/models/test_neurorvq.py braindecode/models/util.py — passed.
  • git diff --check — passed.

The paper's dataset benchmark was not rerun here. The parity result validates implementation agreement with the released checkpoint, not reproduction of published downstream benchmark scores.

Channel provenance

Released pretrained weights require explicit channel_names or chs_info. Randomly initialized models may use the first-N reference-channel fallback for integration/training-from-scratch, but load_pretrained_weights() rejects that ambiguous mapping so pretrained spatial embeddings cannot be silently assigned to the wrong electrodes.

Copy link
Copy Markdown
Contributor Author

@bruAristimunha this is ready for maintainer review when convenient. The current head has exact CPU parity against the released NeuroRVQ checkpoint (263/263 shared encoder tensors; max logit and input-gradient error 0.0), and I clarified the licensing boundary in the PR body: no checkpoint bytes are vendored, the adapted module is explicitly CC BY-NC 4.0 in NOTICE.txt, and weights are fetched from a pinned upstream revision at runtime. I’ll hold the head stable unless review identifies a concrete issue.

@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.21461% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.98%. Comparing base (5e00a5b) to head (69021ba).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1218      +/-   ##
==========================================
- Coverage   87.99%   87.98%   -0.01%     
==========================================
  Files         151      152       +1     
  Lines       17628    17847     +219     
==========================================
+ Hits        15511    15702     +191     
- Misses       2117     2145      +28     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Resolved docs/whats_new.rst conflict by keeping both the NeuroRVQ entry and
master's new llms.txt/docs entry. All other paths auto-merged, including
training/losses.py and test/unit_tests/training/test_losses.py, which now
exactly match origin/master (the braindecode#1198 CroppedLoss fix and its regression
test are restored).
- Replace NeuroRVQ's private _DropPath/_Mlp with braindecode.modules
  DropPath/MLP; load_pretrained_weights remaps the released checkpoint's
  mlp.fc1/mlp.fc2 keys to mlp.0/mlp.2 to match the new module. Parity
  verified at the _Block level with a state-dict-mapped random init:
  max-abs diff 0.0 in both eval and train mode (drop_prob=0.0 by default).
- Fold test_neurorvq.py into test_models.py, keeping every case.
- Trim the NOTICE.txt NeuroRVQ entry to the file-list-only convention and
  add the upstream LICENSE link to the model docstring.
@bruAristimunha bruAristimunha added model Adds a new model needs-replication Model PR: paper number must be replicated (NeuralBench) before merge labels Oct 5, 2026
Resolved docs/whats_new.rst by keeping both entries.
@bruAristimunha

bruAristimunha commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Integration gate (braindecode maintainers), covering #1218 (model) and #1223 (standalone tokenizer)

  • Target: arXiv 2510.13068v4, Table 1(A), Eyes (PhysioNet MI, eyes open vs. closed), NeuroRVQ-EEG 86.9 ± 2.6 balanced accuracy. We count it as replicated if our 5-seed mean falls within 5 % relative of that value, i.e. in [82.6, 91.2]. We picked this cell because it is the smallest public one. No EEG cell in the paper uses TUAB/TUEV.
  • Protocol: we follow the released fine_tuning/NeuroRVQ_EEG_FM_FineTuning.py and flags/NeuroRVQ_EEG_v1.yml (upstream 926e770).
    • Data: the pinned FM checkpoint d944b87, R01/R02 runs with CAR, notch, 0.5–44.5 Hz band-pass, ±500 µV clip, resampled to 200 Hz, cut into 4 s windows (64 ch × 4 patches = 256 tokens).
    • Training: AdamW, lr 5e-4, wd 1e-2, the per-tensor layer decay 0.975 in upstream parameter order, warmup 4 epochs then linear decay, batch 32, 20 epochs, class-weighted BCE.
    • Evaluation: 10-fold subject-independent CV on the last epoch, seeds 41–45.
    • Things we chose ourselves because they are not published: the window length, the fold assignment, and fp32 on CPU instead of CUDA bf16.
  • Already verified: the [models] Add NeuroRVQ residual tokenizer #1223 tokenizer matches upstream exactly on the released tokenizer weights (codes torch.equal, reconstruction max-abs 0.0; see the [models] Add NeuroRVQ residual tokenizer #1223 section below). The canary run on [models] Add NeuroRVQ EEG foundation model #1218 @ 69021ba reached 0.726 held-out BAcc after one warmup epoch.
  • Running: 50 Voyager Jobs (5 seeds × 10 folds), ETA early Tue 6 Oct. The number will be added here when it lands.
  • Declared deviation: Table 16 gives the Eyes head as 801 parameters, which points to a mean-pooled 800-d head; the released code and this port flatten all tokens instead. The campaign follows the released code. Window length (4 s) and fold split are not published; both are fixed in the replication config.

Result (2026-10-06 03:56 Paris, 5 % gate): REPLICATED: 5-seed mean 0.8370 ± 0.0028 (SD over seeds; mean fold-SD 0.0324) vs paper 0.869 ± 0.026, band [0.8256, 0.9125], rel gap 3.7 %. Numbers recomputed from the per-cell files; full record in experiments/replications/neurorvq/replicate.yaml (NeuralBench fork feat/replication-yaml @ cb17a22f).

#1223 tokenizer (updated 2026-10-06 21:30 Paris, head 19a26c1)

  • Code: the tokenizer imports _Block, _MultiScaleTemporalConv and the channel list from braindecode.models.neurorvq and remaps the released mlp.fc1/fc2 keys on load (4e3bd7a). 19a26c1 adds a shared encoder/decoder Transformer base, one input check for forward/tokenize, a docstring example, two tests (per-window standardization, tokenize codes = training-path codes) and the released preprocessing chain in the docs page. Same-seed random init before/after: ordered state-dict keys equal; outputs, codes, gradients and EMA state max-abs 0.0. Released weights on real HGD windows: 4e3bd7a vs 19a26c1 reconstructions max-abs 0.0.

  • Parity, released tokenizer weights (d944b87) vs upstream 926e770: eval target/reconstruction max-abs 0.0, codes identical, 345 parameter gradients max-abs 1.4e-14, EMA state equal after one step.

  • Cell: Table 10, NeuroRVQ rows: High Gamma raw MSE 0.084 (δ 0.046, θ 0.006, α 0.009, β 0.022, γ 0.010) and Pavlov 2022 raw MSE 0.090 (δ 0.032, θ 0.010, α 0.017, β 0.038, γ 0.009). Table 10 has no released script. For the data preparation we follow the benchmark the NeuroRVQ README points to (EEG-Benchmarking @ dbb87b9): 200 Hz, 0.5-45 Hz band-pass, HGD 0-4 s after each cue, Pavlov 14-18 s after each 13-digit trial cue (memory + control), common average reference, its 65 Pavlov / 14 HGD subjects. The filter implementation is not stated there; we use MNE defaults.

  • Numbers (raw MSE, mean over windows; the authors' code and the port give the same value in every row, reconstruction max-abs ≤ 9.5e-7):

    protocol HGD (13,484 windows) Pavlov (3,505 windows)
    released inference example (notch, 0.5-44.5 Hz, ±500 µV clip; window from cue, 256 // channels patches) 0.0627 0.0523
    + benchmark trial window only 0.0610 0.0482
    + common average reference only 0.0742 0.0567
    + benchmark preprocessing order only (MNE resample, then 0.5-45 Hz) 0.0656 0.0931
    benchmark recipe (all three) 0.0748 0.0858
    paper 0.084 0.090

    Per band, benchmark recipe: HGD δ 0.036 / θ 0.006 / α 0.010 / β 0.024 / γ 0.015; Pavlov δ 0.049 / θ 0.005 / α 0.006 / β 0.019 / γ 0.014. The protocol variants were fixed before running. Record: experiments/replications/neurorvq/replicate.yaml tokenizer_cell (NeuralBench feat/replication-yaml).

# Conflicts:
#	braindecode/models/summary.csv
#	test/unit_tests/models/test_models.py

@bruAristimunha bruAristimunha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the port! Replication on Voyager via NeuralBench, paper Table 1(A) Eyes (PhysioNet eyes open vs closed), released fine-tuning recipe, 5 seeds x 10 folds: balanced accuracy 0.837 +/- 0.003 vs paper 0.869 +/- 0.026 (gap 3.7%, inside the 5% gate). Tokenizer parity vs the reference code 2.98e-8. I merged master in (summary.csv / test_models.py keep-both). CI green.

@bruAristimunha
bruAristimunha merged commit 02f6b55 into braindecode:master Oct 6, 2026
19 of 21 checks passed
bruAristimunha added a commit to bruAristimunha/braindecode that referenced this pull request Oct 6, 2026
whats_new: keep both; refs for NeuroRVQ (braindecode#1218), BrainTokenizer (braindecode#1230), BrainOmni (braindecode#1231) point to the merged PRs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model Adds a new model needs-replication Model PR: paper number must be replicated (NeuralBench) before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New model https://github.com/KonstantinosBarmpas/NeuroRVQ

2 participants