Repository navigation
Add registry-wide model contract gates - #1208
bruAristimunha merged 8 commits into
Conversation
|
Independent exact-head validation is green on current head |
|
@bruAristimunha — this is now ready for the maintainer pass on #1156. The shared gate is registry-driven (new models opt in by being registered, not by adding another test list), and the exact current head passes 142 contract cases with 2 skips without model-specific exception skips. It covers input immutability, finite/batched runtime output, and JSON config + state_dict reconstruction. I’m holding scope here so review can decide whether this is the right first official gate. |
|
Deeper contract review found one gap in permutation-only testing: a model can mix samples through a symmetric batch aggregate and still remain batch-permutation equivariant. Current head |
|
One deeper invariant landed on current head |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1208 +/- ##
==========================================
+ Coverage 87.84% 87.96% +0.12%
==========================================
Files 151 151
Lines 17602 17609 +7
==========================================
+ Hits 15462 15490 +28
+ Misses 2140 2119 -21 🚀 New features to boost your workflow:
|
bruAristimunha
left a comment
There was a problem hiding this comment.
Thanks for turning #1156 into tests. We ran the file locally: 142 passed / 2 network-gated skips in ~22 s, no downloads, every gate maps to one of the asks in the issue, and the open model PRs (TMSA-Net, TFMTokenizer) pass it unchanged. CI green.
Resolved docs/whats_new.rst by keeping both changelog entries.
bruAristimunha
left a comment
There was a problem hiding this comment.
Thanks for turning #1156 into tests: 142 passed / 2 network-gated skips locally in ~22 s, no downloads, every gate maps to an ask in the issue; the open model PRs pass it unchanged.
Closes part of #1156.
Summary
Turn a focused subset of Braindecode's model conventions into a registry-wide, machine-enforced first validation layer instead of another manual checklist.
The gate is intentionally test-only and reuses the existing
models_mandatory_parametersregistry, so every newly registered model is exercised automatically without a second opt-in list.Enforced contracts
get_config()must be JSON serializable;from_config(config) + state_dictmust reconstruct the same eval-mode tensor output structure and values.The state-purity and permutation checks share one second forward pass. A third composition probe keeps sample 0 fixed while perturbing only sample 1; this catches symmetric cross-sample mixing that permutation equivariance alone cannot detect.
Failure policy
The common gate is fail-closed for newly registered models. Legitimate task-specific exceptions should be explicit data with a reason rather than ad-hoc skips, so CI can distinguish a real model-contract violation from an intentionally unsupported contract.
Validation
The exact current head
ebe5f1a75f54dd02d92185b80154ee484596aa0fpasses 142 contract cases with 2 existing/intentional skips and no model-specific exception skips. This includes the new cross-sample leakage probe across the full registered-model matrix. Independent run: https://github.com/lindicaphxag-tech/lindicaphxag-tech/actions/runs/37206037736Non-goals
This is the first shared validation layer, not a replacement for model-specific fidelity testing. Pretrained/reference parity, task-specific output contracts, and broader variable-length/channel guarantees remain separate, more expensive layers and can be added independently without making this gate heavy.