Make the Colab pip cache able to shrink, which #11270 assumed it already could - #11271
Merged
Merged
Conversation
…ady could #11270 removed 39 pins from the Colab freeze replay and freed nothing. The cache is still 7,486,845,334 bytes. Two independent reasons, both of which predate that PR. The mapping was not a cache key input. The job keyed on colab_pip_freeze.gpu.txt and notebooks-ci.yml, so editing colab_to_cpu_pin.json -- where the skips, spoofs and index rewrites actually live -- changed what the job installs while the key hash stayed put. The restore then hit exactly, and pip-cache-save is gated on `cache-hit != 'true'`, so the entry holding the removed wheels was never rewritten. The existing comment shows the reasoning was one step short: it explains that the workflow belongs in the key "because the seed step rewrites those pins in place -- CPU index mapping, skips, spoofs", and those three things are all read out of the mapping file it omitted. And pip's HTTP cache is cumulative. pip never evicts from it, the save step saves the whole directory, and restore-keys means a run that misses its exact key still restores the previous generation's wheels into that directory and saves them straight back out. So dropping pins cannot shrink the entry even once the key does move. The two live generations are the proof: 7,486,810,247 bytes on 2026-09-15 against 7,486,845,334 on 2026-09-18, 35 KB apart, because the second run inherited the first rather than re-downloading 5 GiB of wheels it no longer installs. Adding the mapping to key-files fixes the first. The second needs the prefix retired once, so this renames the family to notebooks-colab-cpu: the next run on main starts from an empty pip directory and saves only what the trimmed pin set downloads. Costs one cold install, once. Renaming this job rather than bumping the shared `v2` segment is deliberate. v3 would invalidate all 13 pip families at once, and it would fall outside cache-janitor.yml's `pip-v2-` glob, so the new entries could never be pruned. The janitor's keep=1 rule matches by prefix and so covers the new name and the retired one both, which drains the orphan to a single entry for the 7-day idle timer to finish. Expected: about 2.6 GB for this family against 13.95 GB before #11270, once a run on main writes the first clean entry. Guarded by a mechanical rule -- every repo file the seed step opens must be a key input -- plus explicit checks that the mapping is named and that every key-files path exists, since a glob matching nothing makes hashFiles return empty. All four verified to fail when inverted.
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #11270, which removed 39 pins and freed nothing. The entry is still 7,486,845,334 bytes. Two reasons, both older than that PR.
1. The mapping was not a cache key input
The job keyed on
colab_pip_freeze.gpu.txtandnotebooks-ci.yml. It did not key oncolab_to_cpu_pin.json— where the skips, spoofs and index rewrites actually live. So editing the mapping changed what the job installs while the key hash stayed put, the restore hit exactly, andpip-cache-saveskipped oncache-hit != 'true'. The entry holding the removed wheels was never rewritten.The existing comment was one step short of catching it. It explains the workflow belongs in the key "because the seed step rewrites those pins in place -- CPU index mapping, skips, spoofs" — and all three of those are read out of the file it omitted.
A stale entry here cannot serve wrong content, since pip's cache is addressed by URL and hash. What it does is pin the entry's size to a pin set that no longer exists.
2. pip's HTTP cache is cumulative
pip never evicts from it, the save step saves the whole directory, and
restore-keysmeans a run that misses its exact key still restores the previous generation's wheels into that directory and saves them straight back out. Dropping pins therefore cannot shrink the entry even once the key does move.The two live generations are the proof:
52b8aa64ac8eb9c47ffdfeda35 KB apart. The 09-18 run missed its exact key, restored 09-15 by prefix fallback, installed, and re-saved the same directory. It never re-downloaded the 5 GiB of wheels it no longer needed.
The fix
colab_to_cpu_pin.jsonjoinskey-files.notebooks-colab->notebooks-colab-cpu, retiring the old prefix so nothing falls back to it. The next run on main starts from an empty pip directory and saves only what the trimmed set downloads. Costs one cold install, once.Renaming rather than bumping the shared
v2segment is deliberate:v3would invalidate all 13 pip families at once, and fall outsidecache-janitor.yml'spip-v2-glob, leaving the new entries unpruneable. The janitor'skeep=1rule matches by prefix, so it covers the new name and the retired one both — verified below — which drains the orphan to one entry for the 7-day idle timer to finish.Verification
Janitor routing, run through the real
caseblock with live key shapes:Tests: 146 pass across
test_smoke_install_contract.pyandtest_pip_cache_naming.py. Four new guards, each verified to fail when inverted (drop the mapping fromkey-files; drop the freeze; pointkey-filesat a non-existent file; reusenotebooks-api's cache name):open()s must be a key input — the mechanical rule that would have caught thiskey-filespath exists, since a glob matching nothing makeshashFilesreturn emptyExpected result
About 2.6 GB for this family, against 13.95 GB before #11270, once a run on main writes the first clean entry. The
keep=1half of #11270 is independent and lands at the next janitor sweep regardless.Note the job's matrix is currently red at "Verify imports under spoof", unrelated to caching — the install and save steps succeed, so the cache still gets written.