Skip to content

Add registry-wide model contract gates - #1208

Merged
bruAristimunha merged 8 commits into
braindecode:masterfrom
lindicaphxag-tech:feat/model-contract-gate-1156
Oct 5, 2026
Merged

bruAristimunha merged 8 commits into
braindecode:masterfrom
lindicaphxag-tech:feat/model-contract-gate-1156

Conversation

@lindicaphxag-tech

@lindicaphxag-tech lindicaphxag-tech commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Closes part of #1156.

Summary

Turn a focused subset of Braindecode's model conventions into a registry-wide, machine-enforced first validation layer instead of another manual checklist.

The gate is intentionally test-only and reuses the existing models_mandatory_parameters registry, so every newly registered model is exercised automatically without a second opt-in list.

Enforced contracts

  • eval-mode forward must not mutate the input tensor;
  • after legitimate lazy first-forward materialization, eval-mode inference must not mutate persistent model state;
  • reordering independent batch samples must reorder batched outputs in the same way (batch-permutation equivariance);
  • changing one batch member while holding another fixed must not change the fixed sample's outputs (batch-composition invariance / cross-sample leakage guard);
  • changing one batch member must not change another member's eval-mode outputs (batch-composition invariance), catching symmetric cross-sample leakage that permutation equivariance alone cannot detect;
  • changing one batch member must not change another member's eval output (batch-composition invariance), which catches symmetric cross-sample leakage that permutation equivariance alone cannot detect;
  • model outputs must contain tensor data that preserves the batch dimension;
  • floating/complex outputs must be finite;
  • get_config() must be JSON serializable;
  • from_config(config) + state_dict must reconstruct the same eval-mode tensor output structure and values.

The state-purity and permutation checks share one second forward pass. A third composition probe keeps sample 0 fixed while perturbing only sample 1; this catches symmetric cross-sample mixing that permutation equivariance alone cannot detect.

Failure policy

The common gate is fail-closed for newly registered models. Legitimate task-specific exceptions should be explicit data with a reason rather than ad-hoc skips, so CI can distinguish a real model-contract violation from an intentionally unsupported contract.

Validation

The exact current head ebe5f1a75f54dd02d92185b80154ee484596aa0f passes 142 contract cases with 2 existing/intentional skips and no model-specific exception skips. This includes the new cross-sample leakage probe across the full registered-model matrix. Independent run: https://github.com/lindicaphxag-tech/lindicaphxag-tech/actions/runs/37206037736

Non-goals

This is the first shared validation layer, not a replacement for model-specific fidelity testing. Pretrained/reference parity, task-specific output contracts, and broader variable-length/channel guarantees remain separate, more expensive layers and can be added independently without making this gate heavy.

lindicaphxag-tech commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Independent exact-head validation is green on current head 3a010bab978936c5598b47e021a064f6c36d889f: the registry-wide contract suite passes 142 tests with 2 skips. The gate remains registry-driven (new models opt in by being registered, not by adding another test list) and covers input immutability, finite/batched runtime output, and config + state_dict reconstruction where supported. Evidence run: https://github.com/lindicaphxag-tech/lindicaphxag-tech/actions/runs/37136260152. I’ll keep this PR scoped to the shared machine gate unless maintainer review asks for a broader validation layer.

@lindicaphxag-tech
lindicaphxag-tech marked this pull request as ready for review October 3, 2026 13:59

Copy link
Copy Markdown
Contributor Author

@bruAristimunha — this is now ready for the maintainer pass on #1156. The shared gate is registry-driven (new models opt in by being registered, not by adding another test list), and the exact current head passes 142 contract cases with 2 skips without model-specific exception skips. It covers input immutability, finite/batched runtime output, and JSON config + state_dict reconstruction. I’m holding scope here so review can decide whether this is the right first official gate.

Copy link
Copy Markdown
Contributor Author

Deeper contract review found one gap in permutation-only testing: a model can mix samples through a symmetric batch aggregate and still remain batch-permutation equivariant. Current head ebe5f1a7 adds a composition-invariance probe: sample 0 is held fixed while only sample 1 is changed, and sample 0's batched tensor outputs must remain unchanged. Exact-head registry validation is green: 142 passed, 2 skipped on run https://github.com/lindicaphxag-tech/lindicaphxag-tech/actions/runs/37206037736. No model-specific exceptions were needed. I’m freezing the gate here for maintainer review.

Copy link
Copy Markdown
Contributor Author

One deeper invariant landed on current head ebe5f1a7: permutation equivariance alone can miss symmetric batch mixing (for example, adding a batch aggregate to every sample still permutes correctly). The shared gate now also holds sample 0 fixed, changes only sample 1, and requires sample 0's batched tensor leaves to remain unchanged. Exact-head registry validation is green: 142 passed, 2 skipped. This adds a true cross-sample leakage check without any model-specific exceptions; I’ve updated the PR body and am holding the head here.

@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.96%. Comparing base (2426ddc) to head (cd9f29a).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1208      +/-   ##
==========================================
+ Coverage   87.84%   87.96%   +0.12%     
==========================================
  Files         151      151              
  Lines       17602    17609       +7     
==========================================
+ Hits        15462    15490      +28     
+ Misses       2140     2119      -21     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bruAristimunha bruAristimunha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for turning #1156 into tests. We ran the file locally: 142 passed / 2 network-gated skips in ~22 s, no downloads, every gate maps to one of the asks in the issue, and the open model PRs (TMSA-Net, TFMTokenizer) pass it unchanged. CI green.

Resolved docs/whats_new.rst by keeping both changelog entries.
@bruAristimunha bruAristimunha added the maintenance Bug fix / refactor / tests — not a new model label Oct 5, 2026

@bruAristimunha bruAristimunha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for turning #1156 into tests: 142 passed / 2 network-gated skips locally in ~22 s, no downloads, every gate maps to an ask in the issue; the open model PRs pass it unchanged.

@bruAristimunha
bruAristimunha merged commit 48178cb into braindecode:master Oct 5, 2026
11 of 12 checks passed
@raghav-rathi raghav-rathi mentioned this pull request Oct 5, 2026
22 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Bug fix / refactor / tests — not a new model

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants