Skip to content

[models] Add EEG-CLIP dual encoder - #1200

Open
lindicaphxag-tech wants to merge 30 commits into
braindecode:masterfrom
lindicaphxag-tech:feat/eeg-clip-multimodal-model
Open

lindicaphxag-tech wants to merge 30 commits into
braindecode:masterfrom
lindicaphxag-tech:feat/eeg-clip-multimodal-model

Conversation

@lindicaphxag-tech

@lindicaphxag-tech lindicaphxag-tech commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer map

Fast path (~60 s):

  1. braindecode/models/eeg_clip.py — model/API semantics;
  2. test/unit_tests/models/test_eeg_clip.py — released-source parity and edge-case contracts;
  3. registry/docs files — standard Braindecode integration only.

Four invariants worth checking first:

  • default forward(X) stays Tensor-valued for Braindecode/skorch;
  • paired EEG/text behavior is isolated in forward_paired(...); no Transformers core dependency;
  • released-source projection / raw learned-logit-scale contrastive semantics are preserved;
  • reconstructable default models save/load through the Hub config path, while custom encoders fail explicitly instead of serializing an incomplete config.

Review-ready: latest head bc52ca7 passed its complete public validation suite: six cross-platform test jobs (Ubuntu/Windows/macOS × Python 3.12/3.13), acceptance, Codecov project, Codacy, docs, pre-commit and What's New (12/12 green). GitHub reports mergeable=true; maintainer review is the remaining gate.

Summary

Adds EEGCLIP, a Braindecode-native EEG/text dual encoder based on the 2025 EEG-CLIP work.

Closes #999.

API contract

  • forward(X) remains Tensor-valued for Braindecode/skorch integration and returns raw projected EEG vectors.
  • forward_paired(...) is the explicit eager multimodal API for paired EEG/text embeddings and bidirectional logits.
  • encode_text(...) accepts dependency-injected text encoders or precomputed text features; Transformers is not added as a core dependency.
  • compute_logits(...) preserves the released model's raw projection geometry without hidden L2 normalization.

Pinned-source fidelity

Audited public source:

tidiane-camaret/EEGClip@1d6b89b33d56edcf35037ed7cc27e445885532ea

The implemented training path preserves the pinned released source semantics:

  • EEG/text projection-head structure matches the current public source.
  • learned logit_scale initializes as log(1 / 0.07) and is used directly as the released ClipLoss multiplier;
  • bidirectional logits and symmetric cross-entropy match under copied parameters;
  • input, projection-layer, and learned-logit_scale gradients were included in the parity audit.

Observed numerical envelope in the pinned-source CPU parity audit:

  • EEG→text logits max abs error: 0.0
  • text→EEG logits max abs error: 4.47e-8
  • symmetric loss max abs error: 1.19e-7
  • learned-logit_scale gradient max abs error: 2.79e-9

The repository's historical modelsexample.ckpt predates the public source's projection-head simplification. This PR therefore does not force-map that checkpoint and does not claim historical-checkpoint parity.

Braindecode integration

  • EEGModuleMixin registration, model summary, API docs, categorization, reference and What's New entry;
  • default Deep4Net EEG path plus custom EEG/text encoders;
  • masked and CLS text pooling;
  • paired contrastive loss and zero-shot logits;
  • reset_head() preserves device, dtype, and module train/eval state;
  • Hub/config round-trip for the reconstructable default-encoder path;
  • custom encoder architectures fail explicitly rather than serializing an incomplete reconstruction config.

Current validation state

Current head: bc52ca754c676d3a33a7106e2c41c7ef704d989f.
Upstream master at last synchronization: 63b0d026a62e4094f706170f04e27b663876a2ee.
GitHub reports the PR as mergeable and 0 commits behind, with the same 9-file EEG-CLIP-only diff.

New upstream regression resolved: after #1252 landed, test_pretrained_compat.py::test_compat_covers_every_pretrained_model failed solely because its source-scanning _SHIPS_WEIGHTS regex matched AutoModel.from_pretrained(...) inside the EEGCLIP class docstring example. It incorrectly classified EEGCLIP as distributing an official pretrained EEG model. The optional external text-encoder example now lives in the module docstring, outside inspect.getsource(EEGCLIP); the class retains a dependency-free paired-tensor example. The new upstream source-scanner regex was reproduced against the updated class text with no matches. No model parameter, pretrained artifact, or runtime logic was added/changed in this fix.

Latest-head CI (confirmed at bc52ca7): 12/12 checks completed successfully — Ubuntu 3.12/3.13, Windows 3.12/3.13, macOS 3.12/3.13, acceptance, Codecov project, Codacy, build_docs, pre-commit, check-whats-news. No remaining CI failures or pending checks were visible at verification time. The earlier source-regex false positive has been resolved without changing the model implementation.

Claim boundary

This PR claims a library-native EEG-CLIP implementation with pinned projection and active contrastive-loss semantics fidelity to the current public source.

It does not claim:

  • reproduction of the paper's TUH/TUAB training results;
  • full-model parity to the historical public checkpoint;
  • that the released source's raw learned-logit-scale convention is identical to standard CLIP temperature parameterization.

Reviewer-facing diff hygiene

The branch was re-synchronized after upstream advanced again. Shared registry and documentation files were rebuilt from current master plus only the EEG-CLIP additions. The PR therefore remains a 9-file method/integration diff with no unrelated upstream deletions carried by the feature branch.

No further code changes are planned unless CI or review identifies a reproducible defect.

Copy link
Copy Markdown
Contributor Author

Small follow-up hardening on the current head: reset_head() now moves each replacement projection to the device/dtype of the projection it replaces before preserving its training-mode structure. This matches the reset-head behavior used by other Braindecode models and avoids a fine-tuned CUDA/bfloat16 model silently receiving fresh CPU/float32 heads. I added a CPU-only float64 regression that also runs a paired forward pass, so the invariant is testable without GPU CI. The PR remains mergeable; upstream workflows are still waiting for fork-CI approval.

lindicaphxag-tech commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

CI follow-up on the current head (575ac67):

  • The generic TorchScript failure came from the multimodal forward(..., **text_kwargs) signature.
  • The default forward(X) is now strictly Tensor-valued and returns the raw projected EEG vectors, without L2 normalization, matching the released model's projection geometry.
  • Paired EEG/text training remains available through the explicit eager API forward_paired(...); model-specific tests cover that path.
  • Local focused validation on the exact PR head, CPU/Windows with the uv environment: uv run --python .venv\Scripts\python.exe pytest test/unit_tests/models/test_eeg_clip.py -q → 15 passed.
  • Upstream GitHub Actions remain action_required pending base-repository workflow approval; this local run does not replace upstream CI.

No contrastive objective or encoder/projection semantics were changed. Could a maintainer with write access approve the upstream checks when convenient?

@lindicaphxag-tech
lindicaphxag-tech force-pushed the feat/eeg-clip-multimodal-model branch from 575ac67 to 37b85b7 Compare October 4, 2026 07:40

Copy link
Copy Markdown
Contributor Author

Review-history cleanup only: I squashed the branch to a single commit before maintainer review. The Git tree is unchanged (20a4f79f6b50f88370600ccc44723d832285651a), so the existing focused/parity evidence remains applicable. I’ll hold this head unless review identifies a concrete issue.

Copy link
Copy Markdown
Contributor Author

Self-review clarification: I tightened the reference-fidelity claim without changing the model code. The projection heads match the pinned current source, and the symmetric CLIP loss/logits match when both implementations are given the same effective similarity multiplier. However, the public source initializes logit_scale = log(1/0.07) and currently passes that raw value to ClipLoss, whereas this PR deliberately exponentiates the log-space parameter so initial_temperature=0.07 yields the paper/standard CLIP inverse-temperature scale. The PR body and external parity record now state this explicitly; the original numerical workflow/artifact is unchanged and is no longer described as temperature-parameterization or training-dynamics parity.

Copy link
Copy Markdown
Contributor Author

Current-head source-fidelity update on 0484661c: I removed the deliberate temperature-parameterization divergence and now preserve the pinned released implementation's active ClipLoss semantics (logit_scale = log(1/0.07) is used raw as the similarity multiplier). The refreshed CPU assay compares the actual source parameterization, not a manually matched effective scale. It passes for projection outputs, bidirectional logits, symmetric loss, feature/projection gradients, and the learned logit-scale gradient itself. Max abs differences: EEG→text logits 0.0, text→EEG logits 4.47e-8, loss 1.19e-7, logit-scale gradient 2.79e-9. Run: https://github.com/lindicaphxag-tech/lindicaphxag-tech/actions/runs/37206486699. Evidence record: https://github.com/lindicaphxag-tech/lindicaphxag-tech/blob/main/research/bci_parity/EEGCLIP_CURRENT_SOURCE_PARITY_RESULT.md. I’m freezing model semantics here pending maintainer review.

Copy link
Copy Markdown
Contributor Author

Exact-head follow-up after switching to released-source logit-scale semantics: 0484661c is green on 16 EEG-CLIP model tests and 11 selected shared integration checks (1 skip). More importantly, the pinned-source CPU parity now compares the actual raw learned logit_scale path rather than a matched effective multiplier, including logits, symmetric loss, input/projection gradients, and the logit_scale gradient itself. The parity run passed; contrastive-loss max abs error is 1.19e-7 and logit-scale-gradient max abs error is 2.79e-9. I’ve updated the PR body with the exact runs and will hold this head for review.

@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.13%. Comparing base (63b0d02) to head (bc52ca7).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1200      +/-   ##
==========================================
+ Coverage   89.05%   89.13%   +0.07%     
==========================================
  Files         157      158       +1     
  Lines       19654    19798     +144     
==========================================
+ Hits        17503    17647     +144     
  Misses       2151     2151              
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bruAristimunha bruAristimunha added model Adds a new model needs-replication Model PR: paper number must be replicated (NeuralBench) before merge labels Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

CI note on current head fafca24:

  • Windows 3.13 failed because the GitHub-hosted runner ran out of disk space (There is not enough space on the disk), not because an EEG-CLIP assertion failed.
  • Before the runner exhausted disk, the shared EEG-CLIP save/load and reset_head integration checks passed.
  • Windows 3.12, Ubuntu 3.12/3.13, acceptance, Codacy, pre-commit, and What's New are green on this head; macOS/docs are still completing.

I attempted to re-run only the failed job, but the fork integration does not have Actions write permission. I am leaving the code frozen unless a maintainer review or a reproducible code failure identifies a change.

Copy link
Copy Markdown
Contributor Author

@bruAristimunha the current EEG-CLIP head 8bcbf49 is now fully green across Windows/Ubuntu/macOS 3.12/3.13, docs, acceptance, Codacy, pre-commit, What's New and Codecov; Codecov also reports all modified and coverable lines covered. The branch is mergeable and I am freezing it here. When convenient, I would appreciate maintainer review; I will only change the head again if review identifies a concrete issue.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model Adds a new model needs-replication Model PR: paper number must be replicated (NeuralBench) before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adding EEG-CLIP

2 participants