You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add Brant: foundation model for intracranial (sEEG/iEEG) signals - #1100
Towards #1097 — adds a Braindecode-native implementation of Brant
(Zhang et al., Brant: Foundation Model for Intracranial Neural Signal,
NeurIPS 2023) for intracranial EEG signals.
Brant models intracranial signals (sEEG/iEEG) in four stages: it splits each
channel into non-overlapping temporal patches, embeds them with a learned
patch projection plus a spectral band-power feature and temporal position, runs
a temporal Transformer within each channel followed by a spatial Transformer
across channels at each patch index, then mean-pools the channel-patch grid
into a Braindecode downstream head (bare nn.Linear, no bespoke MLP).
Verified weights
The official pretrained weights are converted and hosted at braindecode/brant-pretrained (revision d716c94bcfee43b0f35fb0bccd3f640e43583dce).
All 210 shared tensors of the official checkpoint (Brant_pre_trained_weights.zip,
sha256 de4f28f4...) load unchanged, and with identical inputs the port's
encoders match the official code (Daoze/Brant on the Hugging Face Hub,
revision 7e6f9156) within 1e-5 (scripts/brant_parity_check.py). The
classification head is Braindecode's and is not pretrained. The Hub config is
left unchanged (sfreq: 250, no band_power_sfreq); the port's default band_power_sfreq=256.0 applies on load.
Band-power sampling rate
Brant's spectral front end computes log band-power at a fixed 256 Hz
regardless of the true data rate (the paper's data is 250 Hz). The port keeps
this upstream behavior explicit as its own band_power_sfreq parameter
(default 256.0), separate from sfreq, which only describes the data. Passing
a different band_power_sfreq shifts the band edges and changes the learned
band embeddings' effective input (up to ~0.55 on the feature scale in our
checks); the default reproduces the released weights faithfully.
Replication
Brant's paper reports every headline number (Tables 1-6) on the authors'
private clinical SEEG data, so none of it is reproducible from public
resources. Two public-only reference runs are in progress and will be posted
here once complete, both evaluated against sets other groups have already used
with Brant so the numbers are comparable, not a reproduction of a private
table:
BaRISTA's sentence-onset/speech protocol on Brain Treebank, run through
NeuralBench with the official frozen encoder weights (brant_barista_protocol
recipe; random and chronological splits).
The MAYO/FNUSA pathology probe (Nejedly et al. 2020, public, CC0): a
frozen-encoder logistic probe over the same clip set as the paper's
Appendix G evaluation, reported next to (not as) that table, since the
paper's own heads were trained on private data and were never released.
Frequency-domain features are computed inside forward, preserving the
standard (B, C, T) model input. The spatial path has no channel-specific
parameters; temporal geometry (patch count, channel count) is validated
against the configured n_times/n_chans at call time. The model is
TorchScript-compatible (torch.jit.script), matching eager output; under
scripting, return_features=True returns logits instead of the feature dict
(torch.jit.is_scripting() guard).
Included
Brant(EEGModuleMixin, nn.Module) with in-model band-power features, patch
tokenization, temporal and spatial Transformer encoders, and a bare nn.Linear prediction head.
Standard Braindecode contracts for runtime signal geometry, return_features, reset_head, model registration, summary metadata, and
serialized configuration.
Reduced-precision-safe spectral computation with float32/float64 numerical
coverage against SciPy.
TorchScript export support.
Tests: a single # Brant section at the end of test/unit_tests/models/test_models.py (band-power correctness against
SciPy's periodogram, sfreq-independence of the band-power rate, bare-linear
head + reset, channel-count and time-length validation, channel-order
preservation through the spatial encoder, and TorchScript/eager parity) plus
the shared parametrized suites every model goes through (registration,
summary CSV, return_features, Hub round-trip, configuration). There is no
bespoke test_brant.py.
Model documentation, examples, a 1.8 changelog entry, and the parity-check
script (scripts/brant_parity_check.py).
The exact upstream Apache-2.0 license plus attribution and modification
notices in source, sdist, and wheel distributions. Braindecode remains
BSD-3-Clause overall.
Scaffold for a braindecode-native port of Brant (Zhang et al., NeurIPS
2023). Fixes the public contract (class, mandatory params, forward /
reset_head); architecture and parity check follow. Not yet registered,
so CI stays green. Towards #1097.
Set encoder defaults to the paper's config (§3.2): embed_dim=2048,
ffn_dim=3072, 12 temporal / 5 spatial layers, 16 heads. Add the 8
rhythmic frequency bands (BRANT_FREQ_BANDS) and document the input
encoding (linear projection + positional + frequency). Towards #1097.
Add braindecode/modules/brant_modules.py: faithful port of the upstream
TimeEncoder / ChannelEncoder (stock nn.TransformerEncoder, no new
dependency) plus the input encoding (learnable band + positional) and the
3-layer MLP head. Band-power is computed inside the forward pass so the
model keeps the (B, C, T) signature.
Wire Brant.forward: patch -> band-power -> temporal encoder -> spatial
encoder -> mean-pool over channels/patches -> MLP head.
scripts/brant_parity_check.py (dev-only) proves the port: both encoders
are bit-exact vs upstream (max|diff|=0), and the internal band-power
matches scipy.periodogram to 1e-16. Still unregistered; weight loading
and registration to follow. Towards #1097.
Export Brant from models/__init__.py, add its mandatory-parameter entry in
util.py and a summary.csv row (Foundation Model / Attention-Transformer,
iEEG). Give the class modest, CI-friendly defaults (~3M params) and document
the paper's ~500M pretrained configuration in the docstring; add an
'activation' parameter for the head. Skip Brant in the jit.script test, like
the other Transformer foundation models. Towards #1097.
test/unit_tests/models/test_brant.py: forward/reset_head shapes, the
patch-size and frequency-band guards, and a band-power sanity check. Adds a
parity gate that, when BRANT_SRC points at the gated upstream Brant_src,
asserts the temporal and spatial encoders are bit-exact vs upstream; the gate
skips cleanly when the reference is absent, so CI stays green. Towards #1097.
The registration and checkpoint-loading note referenced #1097 as pending;
both are now done in this PR, so point to the from_pretrained entry point
instead.
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Adds a braindecode-native implementation of the Brant iEEG/sEEG foundation model (plus supporting modules, tests, and docs) so it can be instantiated and exercised by the standard model test/config infrastructure.
Changes:
Introduce braindecode.models.Brant and its core building blocks in braindecode.modules.brant_modules.
Add Brant-specific unit tests and a dev-only upstream parity-check script.
Register/document Brant in the shared model registries and release notes (incl. summary.csv and integration test lists).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.
Show a summary per file
File
Description
test/unit_tests/models/test_integration.py
Adds Brant to the TorchScript skip list used by the integration suite.
test/unit_tests/models/test_brant.py
Adds Brant-specific tests and an optional upstream parity gate.
scripts/brant_parity_check.py
Adds a developer script for numerical parity against upstream encoders.
docs/whats_new.rst
Documents Brant as a new enhancement for the current dev release.
braindecode/modules/brant_modules.py
Implements Brant’s band-power features, input embedding, temporal/spatial encoders, and head modules.
braindecode/models/util.py
Adds Brant to models_mandatory_parameters, making it part of shared model test coverage.
braindecode/models/summary.csv
Adds Brant to the model catalog and categorization registry.
braindecode/models/brant.py
Introduces the Brant model implementation and public API (init/reset_head/forward).
braindecode/models/init.py
Exposes Brant from braindecode.models (enabling discovery/registration).
Add a return_features path to Brant.forward (pooled encoder embedding,
cls_token=None, matching the BENDR/STEEGFormer convention) so the model
exposes the same feature-extraction API as its peers, with a unit test.
List Brant in the pretrained-models compatibility table and in the
foundation-model finetuning tutorial, noting it is the intracranial
(sEEG/iEEG) model of the collection. Brant is intentionally kept out of
the executed checkpoint bar chart to avoid a ~1.9 GB download at doc
build time.
Replace Brant's inline reshape-into-patches with the shared
braindecode.modules.PatchTokenizer (non-learnable, crop mode), so Brant
uses the same tokenization primitive as the other transformer foundation
models. The tokenizer is a pure reshape with no parameters, so the change
is numerically identical (forward diff 0.0) and leaves the state dict and
from_pretrained loading untouched.
I fed identical inputs through the upstream encoders (Brant_src/pretrain/pre_model.py, Apache-2.0) and the braindecode port with the weights copied over. Both are bit-exact:
Check
Upstream ref
braindecode port
max|diff|
Temporal encoder
TimeEncoder
_BrantTemporalEncoder
0.0
Spatial encoder
ChannelEncoder
_BrantSpatialEncoder
0.0
from_pretrained
Daoze/Brant (508M)
braindecode/brant-pretrained (508M)
loads + forward OK
Reproduce with scripts/brant_parity_check.py --brant-src <Brant_src>; gated in CI by test_brant.py::test_encoders_are_bit_exact_with_upstream (skips without BRANT_SRC). Weights re-hosted at braindecode/brant-pretrained (Apache-2.0): Brant.from_pretrained("braindecode/brant-pretrained", n_outputs=2).
Validate reset_head output count through _set_n_outputs
braindecode/models/brant.py:254
This bypasses EEGModuleMixin._set_n_outputs, so reset_head(0) or a negative value is accepted even though the shared contract rejects non-positive output counts (base.py:522-545). That can create an invalid zero-width head and serialize an unusable configuration; validate through _set_n_outputs before replacing the head.
Document increased setuptools build requirement
pyproject.toml:2
The minimum build dependency is being raised from setuptools 64 to 77, but the current docs/whats_new.rst Requirements section does not document this packaging requirement change. Please add a Requirements entry so users can identify the new build-tool constraint.
Result: weights verified; no public benchmark to reproduce.
Why there is no benchmark number. Brant's paper reports results only with
downstream heads trained on the authors' private clinical data (Appendix G).
Those heads and that data are not released, so no public score can be
recomputed. Following the rule "only public results count", this card checks
the part that can be checked: that braindecode loads the official weights and
computes the same outputs as the official code.
What was checked (scripts in pr-evidence-20260924/pr1100_brant_*.py):
Check
Result
Weights vs the official release (HF Daoze/Brant @ 7e6f9156, time and channel encoders)
210 of 210 shared tensors identical (max difference 0). The 6 extra tensors are the new classification head. Two dropped mask embeddings are all zeros in the official file.
Band-power front end vs upstream compute_power
Same result to float precision (max difference 4e-16 in float64, 2e-7 in float32) at 250 Hz and 256 Hz.
Full forward pass vs the official model, same input
Pooled features match to 6.5e-5 (relative 4e-6); token cosine similarity 1.000000.
Same power tensor fed to both models
Identical pooled output (max difference 0).
One thing users must know. Brant's band-power features depend on the
sampling rate. Upstream computes them at the rate you pass, so sfreq must
match the data: at the wrong rate (250 vs 256 Hz) features differ by up to 0.19
(per-band up to 0.19 on a feature scale of 1.6). The PR documents this.
Not done, and why. No NeuralBench run: there is no public target to compare
against, so a run would only show that the model trains, which the unit tests
already cover.
Drop the bundled LICENSES/Apache-2.0.txt and the packaging changes that
shipped it; list brant.py under Apache-2.0 in NOTICE.txt and declare
license="apache-2.0" on the class, as LUNA, ZUNA and MVPFormer do.
This parameter is the fixed sampling rate used to build the periodogram frequency grid, not necessarily the input signal's actual sampling rate: Brant passes band_power_sfreq here while Brant.sfreq may be 250 Hz. The current description contradicts that documented distinction and could lead future callers to use the wrong rate; describe it as the periodogram/band-edge sampling rate instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Towards #1097 — adds a Braindecode-native implementation of Brant
(Zhang et al., Brant: Foundation Model for Intracranial Neural Signal,
NeurIPS 2023) for intracranial EEG signals.
Brant models intracranial signals (sEEG/iEEG) in four stages: it splits each
channel into non-overlapping temporal patches, embeds them with a learned
patch projection plus a spectral band-power feature and temporal position, runs
a temporal Transformer within each channel followed by a spatial Transformer
across channels at each patch index, then mean-pools the channel-patch grid
into a Braindecode downstream head (bare
nn.Linear, no bespoke MLP).Verified weights
The official pretrained weights are converted and hosted at
braindecode/brant-pretrained(revisiond716c94bcfee43b0f35fb0bccd3f640e43583dce).All 210 shared tensors of the official checkpoint (
Brant_pre_trained_weights.zip,sha256
de4f28f4...) load unchanged, and with identical inputs the port'sencoders match the official code (
Daoze/Branton the Hugging Face Hub,revision
7e6f9156) within 1e-5 (scripts/brant_parity_check.py). Theclassification head is Braindecode's and is not pretrained. The Hub config is
left unchanged (
sfreq: 250, noband_power_sfreq); the port's defaultband_power_sfreq=256.0applies on load.Band-power sampling rate
Brant's spectral front end computes log band-power at a fixed 256 Hz
regardless of the true data rate (the paper's data is 250 Hz). The port keeps
this upstream behavior explicit as its own
band_power_sfreqparameter(default 256.0), separate from
sfreq, which only describes the data. Passinga different
band_power_sfreqshifts the band edges and changes the learnedband embeddings' effective input (up to ~0.55 on the feature scale in our
checks); the default reproduces the released weights faithfully.
Replication
Brant's paper reports every headline number (Tables 1-6) on the authors'
private clinical SEEG data, so none of it is reproducible from public
resources. Two public-only reference runs are in progress and will be posted
here once complete, both evaluated against sets other groups have already used
with Brant so the numbers are comparable, not a reproduction of a private
table:
NeuralBench with the official frozen encoder weights (
brant_barista_protocolrecipe; random and chronological splits).
frozen-encoder logistic probe over the same clip set as the paper's
Appendix G evaluation, reported next to (not as) that table, since the
paper's own heads were trained on private data and were never released.
Architecture
(B, C, T) -> patch + band-power embedding -> temporal Transformer -> spatial Transformer -> prediction headFrequency-domain features are computed inside
forward, preserving thestandard
(B, C, T)model input. The spatial path has no channel-specificparameters; temporal geometry (patch count, channel count) is validated
against the configured
n_times/n_chansat call time. The model isTorchScript-compatible (
torch.jit.script), matching eager output; underscripting,
return_features=Truereturns logits instead of the feature dict(
torch.jit.is_scripting()guard).Included
Brant(EEGModuleMixin, nn.Module)with in-model band-power features, patchtokenization, temporal and spatial Transformer encoders, and a bare
nn.Linearprediction head.return_features,reset_head, model registration, summary metadata, andserialized configuration.
coverage against SciPy.
# Brantsection at the end oftest/unit_tests/models/test_models.py(band-power correctness againstSciPy's periodogram, sfreq-independence of the band-power rate, bare-linear
head + reset, channel-count and time-length validation, channel-order
preservation through the spatial encoder, and TorchScript/eager parity) plus
the shared parametrized suites every model goes through (registration,
summary CSV,
return_features, Hub round-trip, configuration). There is nobespoke
test_brant.py.script (
scripts/brant_parity_check.py).notices in source, sdist, and wheel distributions. Braindecode remains
BSD-3-Clause overall.
Upstream and scope boundary
in
NOTICE.txt).Checklist
braindecode/brant-pretrained).once the runs finish.