Repository navigation
[models] Add EEG-CLIP dual encoder - #1200
lindicaphxag-tech wants to merge 30 commits into
Conversation
|
Small follow-up hardening on the current head: |
|
CI follow-up on the current head (
No contrastive objective or encoder/projection semantics were changed. Could a maintainer with write access approve the upstream checks when convenient? |
575ac67 to
37b85b7
Compare
|
Review-history cleanup only: I squashed the branch to a single commit before maintainer review. The Git tree is unchanged ( |
|
Self-review clarification: I tightened the reference-fidelity claim without changing the model code. The projection heads match the pinned current source, and the symmetric CLIP loss/logits match when both implementations are given the same effective similarity multiplier. However, the public source initializes |
|
Current-head source-fidelity update on |
|
Exact-head follow-up after switching to released-source logit-scale semantics: |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1200 +/- ##
==========================================
+ Coverage 89.05% 89.13% +0.07%
==========================================
Files 157 158 +1
Lines 19654 19798 +144
==========================================
+ Hits 17503 17647 +144
Misses 2151 2151 🚀 New features to boost your workflow:
|
Resolved docs/whats_new.rst by keeping both entries.
|
CI note on current head
I attempted to re-run only the failed job, but the fork integration does not have Actions write permission. I am leaving the code frozen unless a maintainer review or a reproducible code failure identifies a change. |
|
@bruAristimunha the current EEG-CLIP head |
…erving EEG-CLIP entry
…ouching pretrained contracts
Reviewer map
Fast path (~60 s):
braindecode/models/eeg_clip.py— model/API semantics;test/unit_tests/models/test_eeg_clip.py— released-source parity and edge-case contracts;Four invariants worth checking first:
forward(X)stays Tensor-valued for Braindecode/skorch;forward_paired(...); no Transformers core dependency;Review-ready: latest head
bc52ca7passed its complete public validation suite: six cross-platform test jobs (Ubuntu/Windows/macOS × Python 3.12/3.13), acceptance, Codecov project, Codacy, docs, pre-commit and What's New (12/12 green). GitHub reportsmergeable=true; maintainer review is the remaining gate.Summary
Adds EEGCLIP, a Braindecode-native EEG/text dual encoder based on the 2025 EEG-CLIP work.
Closes #999.
API contract
forward(X)remains Tensor-valued for Braindecode/skorch integration and returns raw projected EEG vectors.forward_paired(...)is the explicit eager multimodal API for paired EEG/text embeddings and bidirectional logits.encode_text(...)accepts dependency-injected text encoders or precomputed text features; Transformers is not added as a core dependency.compute_logits(...)preserves the released model's raw projection geometry without hidden L2 normalization.Pinned-source fidelity
Audited public source:
tidiane-camaret/EEGClip@1d6b89b33d56edcf35037ed7cc27e445885532eaThe implemented training path preserves the pinned released source semantics:
logit_scaleinitializes aslog(1 / 0.07)and is used directly as the releasedClipLossmultiplier;logit_scalegradients were included in the parity audit.Observed numerical envelope in the pinned-source CPU parity audit:
0.04.47e-81.19e-7logit_scalegradient max abs error:2.79e-9The repository's historical
modelsexample.ckptpredates the public source's projection-head simplification. This PR therefore does not force-map that checkpoint and does not claim historical-checkpoint parity.Braindecode integration
EEGModuleMixinregistration, model summary, API docs, categorization, reference and What's New entry;reset_head()preserves device, dtype, and module train/eval state;Current validation state
Current head:
bc52ca754c676d3a33a7106e2c41c7ef704d989f.Upstream master at last synchronization:
63b0d026a62e4094f706170f04e27b663876a2ee.GitHub reports the PR as mergeable and 0 commits behind, with the same 9-file EEG-CLIP-only diff.
New upstream regression resolved: after #1252 landed,
test_pretrained_compat.py::test_compat_covers_every_pretrained_modelfailed solely because its source-scanning_SHIPS_WEIGHTSregex matchedAutoModel.from_pretrained(...)inside the EEGCLIP class docstring example. It incorrectly classified EEGCLIP as distributing an official pretrained EEG model. The optional external text-encoder example now lives in the module docstring, outsideinspect.getsource(EEGCLIP); the class retains a dependency-free paired-tensor example. The new upstream source-scanner regex was reproduced against the updated class text with no matches. No model parameter, pretrained artifact, or runtime logic was added/changed in this fix.Latest-head CI (confirmed at
bc52ca7): 12/12 checks completed successfully — Ubuntu 3.12/3.13, Windows 3.12/3.13, macOS 3.12/3.13, acceptance, Codecov project, Codacy, build_docs, pre-commit, check-whats-news. No remaining CI failures or pending checks were visible at verification time. The earlier source-regex false positive has been resolved without changing the model implementation.Claim boundary
This PR claims a library-native EEG-CLIP implementation with pinned projection and active contrastive-loss semantics fidelity to the current public source.
It does not claim:
Reviewer-facing diff hygiene
The branch was re-synchronized after upstream advanced again. Shared registry and documentation files were rebuilt from current
masterplus only the EEG-CLIP additions. The PR therefore remains a 9-file method/integration diff with no unrelated upstream deletions carried by the feature branch.No further code changes are planned unless CI or review identifies a reproducible defect.