Stage the release builder: bound the dense reconcile working set (~0.5TB measured) via cache spill + checkpointed passes #463

Activity

1. MaxGhenis commented on Jul 20, 2026

   

   MaxGhenis

   on Jul 20, 2026

   ContributorAuthor

   More actions

   Claiming this. Plan of record, per the spec:

   1. Read the Bound release-builder memory: one engine system per target family, explicit simulation release #461 post-fix structure and the June codex/jct-materialization-memory-20260617 branch for prior art, then map the two crests against the builder's actual stage sequence (eligibility/full-pool high-water; SSI reconcile passes).
   2. Spill materialized reform families through the existing target_materialization_cache (materialize → persist → drop; compile/solve streams from cache), preserving cache identity semantics — any layout change is explicitly versioned, not silent.
   3. Checkpoint-bounded SSI reconcile passes on the Stage the base builder into checkpointed processes with a bit-equivalence proof #406 lossless frame-checkpoint machinery, so each pass's working set dies with the pass.
   4. stage_profile.json twin with per-stage RSS/wall high-water, asserted in tests as a budget number.
   5. Acceptance: bit-identical exports vs the monolith on a pinned subsample (Stage the base builder into checkpointed processes with a bit-equivalence proof #406 equivalence-proof pattern) + the builder test suite; full-scale 256GB proof run deferred until the live GCP build completes.

   Diff stays off the target-registry data surfaces (fiscal_targets.py, release_input_coverage_manifest.json) per the #451 Phase B lane coordination. Working in a fresh worktree off current origin/main.
2. MaxGhenis commented on Jul 21, 2026

   

   MaxGhenis

   on Jul 21, 2026

   ContributorAuthor

   More actions

   Third full-scale datapoint, and it reclassifies the problem: the n2-highmem-80 rerun (RID …20260720T102929Z, 629GB RAM + 419GB swap) was oom-killed after 13h11m at anon-rss 623GB / total-vm 1.10TB (dmesg 23:39:26Z; pressure log shows MemAvailable at 0 for the final minutes).

   The decisive evidence is the phase behavior, not the peak: the release log shows 73 eligibility/ACA materialization cycles, still looping at death — i.e. the SSI take-up reconcile ran ~a dozen-plus passes without exiting, and the footprint grew across passes (~536GB peak on the 32-vCPU run at fewer passes → 1TB+ here at more passes). Two coupled defects, both dense-arm-specific:

   1. Per-pass accretion: each reconcile pass rematerializes the eligibility/target families and something pass-scoped survives the Bound release-builder memory: one engine system per target family, explicit simulation release #461 sweeps at full-pool scale (candidate: pass-variant keys accumulating in target_materialization_cache, or pass-scoped frames pinned by the checkpoint layer). Bound release-builder memory: one engine system per target family, explicit simulation release #461's fix is visibly working within passes (RSS returns between families) — the staircase is across passes.
   2. Dense reconcile convergence: sparse M exits in 2 passes (710,076 → 464,107 under the 0.10 cap). The dense arm (0.12 cap, Dense-arm SSI swap runaway cap at 0.12 × fresh national (#447) #452/Key the SSI swap cap on the selection arm, not the export-mode flag #455 semantics) apparently oscillates or creeps without hitting the fresh-pair exit. Even with perfect memory hygiene, a ~dozen-pass reconcile at 68-batch × full-pool scale is not a viable release path.

   So #463's scope confirms and extends: the staged/checkpointed-pass design must make pass N's memory die with it (fix 1 structurally), and the reconcile-pass audit should treat dense convergence as its own acceptance criterion (pass-count bound or explicit divergence failure, not an OOM). Evidence preserved on the stopped VM (buildm-dense-runner, disk retained): release/pressure/chain logs for all three runs, dmesg records, warm caches. The M-dense artifact is blocked on this work — no box size rescues a non-converging loop.
3. MaxGhenis commented on Jul 21, 2026

   

   MaxGhenis

   on Jul 21, 2026

   ContributorAuthor

   More actions

   Scope update absorbed into the implementation contract:

   1. "Pass N's memory dies with pass N" is now a HARD acceptance criterion for the checkpoint-bounded passes, verified two ways: the stage profiler asserts a flat parent RSS floor across pass boundaries on a synthetic multi-pass fixture (no staircase), and the PR body will carry a targeted audit of both named suspects — where pass-variant cache identities are created (disk accretion acceptable, parent-resident handles not) and what the checkpoint layer pins between passes.

   2. Divergence bound added as its own intervention: a trajectory-reading contraction requirement evaluated at each pass boundary, orthogonal to max_passes, reading the Carry the SSI reconcile per-pass trajectory through raise and record (#447) #448 per-pass swap-delta trajectory already in the reconcile payload. Plateau, oscillation, and growth all fail with the full per-pass numbers in the gate error — minutes, not 13 hours. Unit-tested with teeth (contracting passes; plateau/oscillation/growth each fail naming the trajectory).

   Please ship the raw logs to this issue (or a gist linked here) — most useful, in order: the pressure log with pass-boundary timestamps (for the staircase step sizes), the release log section covering the reconcile passes (per-pass swap-delta/count-error trajectory to calibrate the contraction factor against real divergence), and the dmesg OOM record. Implementation order: profiler → divergence bound → subprocess passes → family spill-through; equivalence harness built alongside.
4. MaxGhenis commented on Jul 21, 2026

   

   MaxGhenis

   on Jul 21, 2026

   ContributorAuthor

   More actions

   Design reframe (supersedes the checkpointed-pass framing above; from Max's review of the OOM evidence): the reconcile loop should be removed, not staged. The calibration solve is already a static matrix descent; the only reason the engine re-enters is that SSI take-up participation depends on weights. That coupling is expressible without re-simulation:

   • Materialize the take-up-sensitive columns once, conditional on participation state — for each candidate unit, the participant-state and nonparticipant-state contributions to every affected target family (SSI amounts, categorical Medicaid, ACA MAGI class — the dependency set is narrow and enumerable, not "everything").
   • A reconcile pass is then: flip participation indicators to hit the count target under current weights → recompose the affected matrix rows from the precomputed conditional columns → re-solve. No engine, no rematerialization, milliseconds per pass instead of hours.
   • The engine runs exactly once per input-frame state, at the start, to build the conditional matrix. Memory = the matrix (full dense pool ≈ tens of GB) plus two variants of the affected column subset — a single-box problem with no pass staircase by construction.

   Acceptance for #463 accordingly: (1) dense full-pool build completes on n2-highmem-32 with engine materialization occurring exactly once (assert via a materialization-count telemetry, not just RSS); (2) reconcile trajectory identical-or-better vs the sparse-certified semantics (same fresh-pair exit, caps per #452/#455) — proven on the sparse arm by bit-comparison of the certified M reconcile outcome recomputed through the matrix path; (3) the pass-count bound + loud divergence failure stands as the backstop.

   The staged/spill machinery (#406 pattern) remains useful for the one-shot materialization's peak, but it is the secondary lever now.
5. MaxGhenis commented on Jul 21, 2026

   

   MaxGhenis

   on Jul 21, 2026

   ContributorAuthor

   More actions

   Plan-of-record simplification (Max, on review of the reconcile's rationale): SSI loses its special treatment entirely. The loop's own docstring gives three motivations; each dissolves:

   1. Dependent surfaces (Medicaid categorical / ACA / other-health) — needs ordering, not iteration: assign take-up flags once from the prior, materialize dependent columns once under those frozen flags, calibrate. Inputs stay self-consistent with zero passes.
   2. Two official measures ~115k apart — already solved by rescaling onto the registry's single ssi_recipients total; that is prior work, not loop work.
   3. "Flags must be exactly count-faithful under final weights" — retired as an invariant. ssi_recipients is an ordinary indicator-column count target (it already exists at fiscal_targets.py:2434); the solve hits it as well as it hits any other count, and the residual miss is recorded in diagnostics like every other program's. M's own trajectory bounds what we are accepting: the loop was closing ~0.46–0.71M on ~7.5M recipients (6–10%), typical of the misses SNAP/Medicaid/EITC live with today. The structural fix for that miss is the Build N SIPP donor stage (SIPP+SCF blend + SSI-aware refit: land the asset magnitudes for the Build J re-certification #374/Child SSI structurally absent: ages 0–14 have no disability signal (baseline child SSI 0.12M vs SSA 0.98M); seed qualifying disability per the #430 pattern #453) — a better prior, not a runtime loop.

   Revised #463 scope, in order: (a) delete _reconcile_ssi_take_up_and_refit from the release path — take-up assignment happens once, pre-materialization; (b) engine-once materialization with the #406-pattern spill for the one-shot peak; (c) the materialization-count telemetry and the divergence backstop become trivial (no loop to bound); (d) the conditional-columns design from the previous comment is demoted to a documented fallback, to be built only if the accepted SSI count miss is later adjudicated unacceptable after the #374 priors land. Acceptance adds: N's scorecard reports the SSI recipient-count relative error alongside the other count targets, same contract, no special casing.

   This also retires the #452/#455 swap-cap machinery and the #448 pass telemetry from the live path once the deletion lands (they stay in history as the loop's honest instrumentation). Sparse arm equivalence: N is a fresh certification — the M-vs-N scorecard comparison surfaces the count-miss delta transparently rather than hiding it behind reconciliation.
6. MaxGhenis

   added a commit that references this issue on Jul 24, 2026

   SSI take-up: delivered-weight prior basis + hard band delivery gate (#…

   ...

   15e5b7a
7. PavelMakarchuk commented on Oct 5, 2026

   

   PavelMakarchuk

   on Oct 5, 2026

   Contributor

   More actions

   Throughput: the batch loop runs on about one core

   • Attempt 310842b: target compilation took 9,951 s at 1.04 CPU-s per wall second across about 2,124 independent batches. The last release rerun spent 22 h at about one core.
   • The batches are independent, so sharding them across processes is the largest available speedup. About 8 processes would have saved roughly 8,500 s on that attempt (derived from the receipts, not measured).
   • Each process holds its own batch working set, so the shard count must fit the memory bound this issue sets.
   • The next Route A attempt runs with staging on (plus Record process telemetry for US release builds #1083 once merged). The Build progress tab will then show per-stage cores and memory and batch progress, which is the baseline for a sharding change.