Skip to content

Make the Colab pip cache able to shrink, which #11270 assumed it already could - #11271

Merged
danielhanchen merged 2 commits into
mainfrom
ci/colab-pip-cache-key-and-prefix
Sep 18, 2026
Merged

danielhanchen merged 2 commits into
mainfrom
ci/colab-pip-cache-key-and-prefix

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

Follow-up to #11270, which removed 39 pins and freed nothing. The entry is still 7,486,845,334 bytes. Two reasons, both older than that PR.

1. The mapping was not a cache key input

The job keyed on colab_pip_freeze.gpu.txt and notebooks-ci.yml. It did not key on colab_to_cpu_pin.json — where the skips, spoofs and index rewrites actually live. So editing the mapping changed what the job installs while the key hash stayed put, the restore hit exactly, and pip-cache-save skipped on cache-hit != 'true'. The entry holding the removed wheels was never rewritten.

The existing comment was one step short of catching it. It explains the workflow belongs in the key "because the seed step rewrites those pins in place -- CPU index mapping, skips, spoofs" — and all three of those are read out of the file it omitted.

A stale entry here cannot serve wrong content, since pip's cache is addressed by URL and hash. What it does is pin the entry's size to a pin set that no longer exists.

2. pip's HTTP cache is cumulative

pip never evicts from it, the save step saves the whole directory, and restore-keys means a run that misses its exact key still restores the previous generation's wheels into that directory and saves them straight back out. Dropping pins therefore cannot shrink the entry even once the key does move.

The two live generations are the proof:

created bytes key hash
2026-09-15 7,486,810,247 52b8aa64ac8e
2026-09-18 7,486,845,334 b9c47ffdfeda

35 KB apart. The 09-18 run missed its exact key, restored 09-15 by prefix fallback, installed, and re-saved the same directory. It never re-downloaded the 5 GiB of wheels it no longer needed.

The fix

  • colab_to_cpu_pin.json joins key-files.
  • The family is renamed notebooks-colab -> notebooks-colab-cpu, retiring the old prefix so nothing falls back to it. The next run on main starts from an empty pip directory and saves only what the trimmed set downloads. Costs one cold install, once.

Renaming rather than bumping the shared v2 segment is deliberate: v3 would invalidate all 13 pip families at once, and fall outside cache-janitor.yml's pip-v2- glob, leaving the new entries unpruneable. The janitor's keep=1 rule matches by prefix, so it covers the new name and the retired one both — verified below — which drains the orphan to one entry for the 7-day idle timer to finish.

Verification

Janitor routing, run through the real case block with live key shapes:

keep=1  pip-v2-notebooks-colab-cpu-Linux-X64-py3.13     <- new
keep=1  pip-v2-notebooks-colab-Linux-X64-py3.13         <- retired, drains
keep=2  pip-v2-notebooks-api-Linux-X64-py3.12           <- untouched
keep=2  pip-v2-consolidated-Linux-X64-py3.12            <- untouched

Tests: 146 pass across test_smoke_install_contract.py and test_pip_cache_naming.py. Four new guards, each verified to fail when inverted (drop the mapping from key-files; drop the freeze; point key-files at a non-existent file; reuse notebooks-api's cache name):

  • every repo file the seed step open()s must be a key input — the mechanical rule that would have caught this
  • the mapping is named explicitly, so deleting the rule above cannot quietly drop the one file that caused the bug
  • every key-files path exists, since a glob matching nothing makes hashFiles return empty
  • the existing distinct-name and shared-key-files rules still hold

Expected result

About 2.6 GB for this family, against 13.95 GB before #11270, once a run on main writes the first clean entry. The keep=1 half of #11270 is independent and lands at the next janitor sweep regardless.

Note the job's matrix is currently red at "Verify imports under spoof", unrelated to caching — the install and save steps succeed, so the cache still gets written.

danielhanchen and others added 2 commits September 18, 2026 14:54
…ady could

#11270 removed 39 pins from the Colab freeze replay and freed nothing. The
cache is still 7,486,845,334 bytes. Two independent reasons, both of which
predate that PR.

The mapping was not a cache key input. The job keyed on
colab_pip_freeze.gpu.txt and notebooks-ci.yml, so editing
colab_to_cpu_pin.json -- where the skips, spoofs and index rewrites actually
live -- changed what the job installs while the key hash stayed put. The
restore then hit exactly, and pip-cache-save is gated on `cache-hit != 'true'`,
so the entry holding the removed wheels was never rewritten. The existing
comment shows the reasoning was one step short: it explains that the workflow
belongs in the key "because the seed step rewrites those pins in place -- CPU
index mapping, skips, spoofs", and those three things are all read out of the
mapping file it omitted.

And pip's HTTP cache is cumulative. pip never evicts from it, the save step
saves the whole directory, and restore-keys means a run that misses its exact
key still restores the previous generation's wheels into that directory and
saves them straight back out. So dropping pins cannot shrink the entry even
once the key does move. The two live generations are the proof:
7,486,810,247 bytes on 2026-09-15 against 7,486,845,334 on 2026-09-18, 35 KB
apart, because the second run inherited the first rather than re-downloading
5 GiB of wheels it no longer installs.

Adding the mapping to key-files fixes the first. The second needs the prefix
retired once, so this renames the family to notebooks-colab-cpu: the next run
on main starts from an empty pip directory and saves only what the trimmed pin
set downloads. Costs one cold install, once.

Renaming this job rather than bumping the shared `v2` segment is deliberate.
v3 would invalidate all 13 pip families at once, and it would fall outside
cache-janitor.yml's `pip-v2-` glob, so the new entries could never be pruned.
The janitor's keep=1 rule matches by prefix and so covers the new name and the
retired one both, which drains the orphan to a single entry for the 7-day idle
timer to finish.

Expected: about 2.6 GB for this family against 13.95 GB before #11270, once a
run on main writes the first clean entry.

Guarded by a mechanical rule -- every repo file the seed step opens must be a
key input -- plus explicit checks that the mapping is named and that every
key-files path exists, since a glob matching nothing makes hashFiles return
empty. All four verified to fail when inverted.
@danielhanchen
danielhanchen merged commit a6ffbed into main Sep 18, 2026
15 of 28 checks passed
@danielhanchen
danielhanchen deleted the ci/colab-pip-cache-key-and-prefix branch September 18, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant