feat: remove the memory feature family and adopt agent-native session compaction - #710

Summary

Implements the Memory removal spec (.compozy/tasks/memory-removal: _spec.md, _dx.md, _uiux.md, _tests.md, ADR-001–004).

The change does two things:

• Hard cut of the memory feature family. Agent memory, Dream consolidation, the Knowledge app and the workspace knowledge/ augmenter are removed from the daemon, API, CLI, native tools, MCP, Host API, SDKs, config, Web, docs and the official skill.
• CompozyOS no longer compacts sessions itself. Every rebuild into a new agent session is bounded the Superset way. The agent's native compaction is observed over ACP and can be requested on demand ("Compact now", Goal executor).

Upgrades never block: retired leftovers in user-owned files are archived or ignored losslessly, layouts reconcile, database state migrates, and no file on disk is deleted.

~1,515 files changed, ~+25k / −123k. Most of the diff is deletion: internal/memory/** (~19k prod / 17k test lines), 40 HTTP/UDS operations, the compozy memory CLI tree, 35 native tools, the Knowledge app and the memory docs.

What changed

1. Bounded rebuilds replace CompozyOS compaction (ADR-002)

• boundReplay is shared by every rebuild path: resume, runtime replacement, affinity switch, continue/fork, native-bind fallback and launch.
  • It uses the [session.derive] budget (max_replay_bytes 131072 / max_message_bytes 16384) and protects the newest messages while they fit.
  • It pins the earliest user message ahead of [N earlier messages omitted to fit the context budget], adds the workspace-authority line to every header, and adds the compozy__session_history pointer only when messages were omitted and the tool is available.
  • It logs session.replay.bounded once per rebuild that omitted messages. Nested derives keep inherited omission evidence.
• Deleted: the pressure-triggered summarizer (compaction*.go, checkpoint runtime/summarizer), lifecycle continuation, ResumeContextProvider, TransientModel, LedgerMaterializer, AllowStoppedParent, the checkpoint-summary spawn role, and [session.compaction].
• Session migration 00009_unarchive_compaction_spans.sql:
  • restores events archived by the old compactor, excluding conversation-rewind ranges;
  • flags stale rewind baselines (baseline_stale);
  • rebuilds them with the commit formula at boot and lazily before any replay;
  • restores the transcript projection (entries and tool routes) for the restored spans, so the restored history appears in transcript, search, outline and fork/rewind anchors.

2. Native compaction over ACP (ADR-004, experimental)

• SDK fork: compozy/acp-go-sdk branch compaction-unstable adds the UNSTABLE compaction schema; it is pinned via go.mod replace to v0.13.6-0.20261008041747-ebd00ee4ba87.
• ACP layer:
  • advertises clientCapabilities.session.compaction: {};
  • a defensive raw-discriminator decoder turns unknown session/update kinds into raw system events (never messages);
  • a per-process, per-id assembler emits materialized compaction snapshots with ACP patch semantics (omitted/null/value, chunks buffered until terminal, corrections in receive order, late chunks dropped), a 16 KiB summary cap, and a claim-token scrub across chunk boundaries even with redaction off.
• Session:
  • redacted snapshots are persisted;
  • exactly one session.compaction_fired {compaction_id, trigger, context_used, context_size} and one context.pre_compact per id, and at most one context.post_compact;
  • one Compaction transcript item per id, while /history stays raw ledger rows;
  • a freshness boundary in contextusage and the Goal readers: occupancy is unknown after a terminal compaction until a genuine later report, and prompt totals never restore it;
  • session_usage_changed fires on compaction.
• Hooks are observation-only: ContextCompactionPayload, a labels-only patch, and the compaction_trigger matcher (requested|agent).
• Compact now: Manager.RequestCompaction with the shared ResolveCompactionCommand (compact, then compress).
  • The prompt is sent in maintenance delivery mode: the literal command, no skills or augmenters, and no startup or replay consumption.
  • A deferred replay obligation survives maintenance, stop/restart and native resume.
  • Errors: session_busy, compaction_unsupported, not-active.
• Surfaces:
  • POST /api/workspaces/{ws}/sessions/{id}/compact on HTTP and UDS (202 receipt / 409);
  • compozy session compact (waits; -o json);
  • native compozy__session_compact (structural error codes);
  • Web Compact now in the context rail.
• Goal: the compaction turn keeps the managed PrepareActionPrompt path (prompt id, fences, recovery) and sends /<command> in maintenance mode. The outcome follows the current terminal snapshot status; agents without the command go straight to reseed.

3. Memory runtime and public surfaces removed (ADR-001, ADR-003)

• Daemon:
  • no memory store, catalog, Dream scheduler, extractor, write controller, provider registry, memory prompt section, <turn-recall> or <workspace-knowledge-snapshot>;
  • bootMemorySessionRuntime becomes bootSessionRuntime;
  • the auto-title deadline is a 60s constant;
  • the SOUL startup provider is wired directly (a regression found while removing the memory assembler).
• Deleted packages: internal/memory/** and internal/sessions/ledger, along with the boundary pairs, the memory migration stream, events/observe/sse memory members, the automation Dream observer and memory.consolidated.
• API/CLI/MCP/extension/SDK:
  • the 40 memory operations and the memory tag are gone; StatusSchemaVersion is 2026-10-07;
  • settings: 18 sections, only coordinator and auto_title roles;
  • memory_entries, memory_policy and the dream type are removed;
  • the compozy memory tree, compozy_host__memory__*, Host API memory/*, memory.backend, memory.read|write and the memory-backend-ts scaffold are removed;
  • the SDKs are regenerated.
• Config: [memory], [session.compaction], the four memory roles, HomePaths.MemoryDir and builtin dreaming-curator are removed. Explicit config set of retired keys is still refused.

4. Upgrade state and non-blocking shims (one shim generation, deleted in v0.6.0)

• Global migration 00128_retire_memory.sql:
  • drops the memory stream objects and goose_db_version_memory;
  • deletes memory.consolidated triggers with their overlays, catalog, resource records, ingress bindings and owned secrets (run history stays);
  • deletes dream / memory-extractor / checkpoint-summary sessions and their dependents;
  • leaves PRAGMA foreign_key_check empty.
• Config archive on load (archiveRetiredMemorySettings):
  • covers global, profile and workspace overlays, archiving [memory], the retired roles, [session.compaction], memory.consolidated trigger entries (including non-contiguous sub-tables) and the retired hook matcher keys;
  • writes them into a commented # Archived retired memory and compaction settings; these values are inactive. block;
  • is lossless and idempotent, with a race guard;
  • non-fatal: a refused or failed rewrite logs a WARN, loads with the values inactive, and retries next load.
• Ignored with a warning, never fatal:
  • SOUL memory_policy;
  • retired compozy__memory* tool IDs and toolsets in policies;
  • retired extension manifest entries and memory.consolidated trigger resources;
  • retired compaction matcher keys in extension, AGENT.md and SKILL.md hooks.
• Layout reconcile: saved snapshots and window_layout resources drop unregistered apps (Knowledge), rewrite /settings/memory to /settings, persist once, and log windowmanager.snapshot_reconciled.
• Session recovery boundary (permanent): orphaned directories of deleted legacy sessions (unknown session type, or the retired internal spawn roles) are never recataloged, and their files are untouched. Custom advisory roles still recover.

5. Web (_uiux.md S1–S18)

• Removed:
  • the Knowledge app and /knowledge; the Settings Memory page and memory role panels; the Home Memory tile (6 → 5);
  • the automation Memory event group, the profile Memory row, the SOUL memory field and memory tool cards;
  • the palette Knowledge domain and the meter threshold warning;
  • @compozy/ui ContextBox.
• Unknown apps are guarded on restore.
• Added:
  • Compact now in the context rail meter section (present only when compact/compress is advertised; disabled while busy; inline session_busy / compaction_unsupported errors);
  • the Compaction timeline item (in progress / completed with optional summary / failed / cancelled / vendor status);
  • "Agent/Requested compaction" markers (before → after from the daemon's context_after);
  • post-compaction meter invalidation keyed off the daemon's cleared_by, with compaction-aware wording.
• System-role projections are carried as assistant rows so assistant-ui never throws on the compaction part.

6. Docs, skill, QA, release

• Site:
  • the memory docs, the CLI memory subtree and the landing Memory/Dream content are removed (bento has two cards);
  • config-toml.mdx is rewritten;
  • new sessions/compaction.mdx;
  • hooks docs are reshaped;
  • the retired-surface guards are extended (UT-W07).
• Official skill: references/memory.md is deleted; compaction and Compact now are documented.
• Root docs: README, PRODUCT, COPY (Session Compaction Terms), glossary, change-impact.md §Memory removal (including the v0.6.0 shim list) and the root + site migration guides #memory-removal.
• Breaking release note: .release-notes/retire-memory-and-session-compaction-*.md.
• QA: 32 memory-only scenarios are deleted, compaction scenarios are rewritten, and the upgrade walk, Compact now and real-adapter scenarios are added.

How it was built

• Workers (orchestrated via herdr):
  • Codex GPT-6.1-Sol (medium): bounded replay, ACP compaction layer, upgrade shims, public surfaces, runtime removal, native-compaction integration.
  • Claude Sonnet 5.5 (xhigh): Web removals and additions, docs/skill/QA.
• Each slice lived in its own worktree and was merged into memory-removal by the controller.
• Independent review: GPT-6-Astra (xhigh), four rounds over the integrated branch, with every finding fixed at the root and given an adversarial regression in its owning suite. No BLOCKER was found in any round:
  • Round 1 (whole branch): 8 MAJOR / 3 MINOR / 3 NIT. Highlights:
    • restored legacy history missing from the transcript projection;
    • non-contiguous TOML trigger tables;
    • extension trigger resources blocking load;
    • deferred replay lost across maintenance and restart;
    • prompt totals restoring occupancy;
    • corrective terminal statuses;
    • Web meter retention;
    • raw SSE losing compaction fields.
  • Round 2 (fixes + Web phase 2): 3 MAJOR / 3 MINOR:
    • replay acknowledgment failure;
    • persisted replay re-bounded under the current budget;
    • 409 transport mapping for native Compact codes;
    • projection completion ordering;
    • authoritative context_after / cleared_by for the S14 markers.
  • Round 3: 2 MAJOR / 2 MINOR: interleaved-turn projection restore; prompt recovery accumulating replay blocks (pre-existing, now one bounded replay per dispatch); stalled-prompt Compact now; the compact 400 in OpenAPI.
  • Round 4: 1 MINOR (failed historical compaction attempts and the restoration cutoff), fixed.
• Also folded in from the docs reconciliation:
  • the config archive is non-fatal;
  • AGENT.md/SKILL.md matcher shims;
  • 00128 also retires checkpoint-summary sessions;
  • a permanent session-recovery boundary for orphaned legacy session directories.

CI fixes found on the PR

• Lint: 14 findings (goconst, gocritic, golines, unused helpers) fixed; full make go-lint is clean.
• E2E runtime: two UDS/HTTP parity fixtures encoded removed behavior. One opened a window with an unregistered app id (sessions is a palette view; the OS app is session), which the new load-time reconcile now correctly drops. The other expected the removed workspace_knowledge / durable_memory augmenters.
• E2E-008: the retired Knowledge window can no longer be created through the reconciling window-manager API. A Go fixture seeder (same pattern as the E2E-020 seeder) writes the previous-release layout into the stopped isolated daemon's store, and the test then cold-restarts the daemon.
• E2E-005 found a real bug. Cloning an empty DeliveryManifest.Spans produced null, because append([]T(nil), empty...) returns nil. A maintenance prompt (/compact, no sections) therefore serialized "spans": null. The Web's strict transcript schema rejected the whole frame, which dropped the agent's reply in that turn.
  • Fixed at the source: slices.Clone in the ACP clone and in transcript redaction, with a regression in the transcript codec contract suite.
  • The Web schema also tolerates null for already-stored events.

Compatibility (SD-013)

Verification

• Local, per slice (heavy gates delegated to CI):
  • go build ./...;
  • go vet ./... and go vet -tags=integration ./...;
  • make codegen-check (OpenAPI, SDKs, native tool catalog, CLI docs, migration-guide parity);
  • owning Go suites with both tags, plus focused race runs;
  • scoped golangci-lint;
  • Turborepo typecheck/lint/test/build-storybook for web and packages/ui (6,546 + 816 tests);
  • site typecheck/lint/test (256 tests);
  • product-language-check.
• Real-process tests:
  • acpmock subprocess + SQLite: native compaction, hooks, maintenance after runtime replacement and restart, legacy upgrade read path, Compact over HTTP/UDS/CLI/Web attribution;
  • real daemon boot: config archive, legacy-session recovery boundary.
• Known limits:
  • The compaction fixtures are representative, source-derived Claude/Codex adapter frames, not live recordings. The real-adapter walk (E2E-009) and the upgraded-home QA walk (E2E-007) are manual scenarios in docs/qa/scenarios, marked untested.
  • Some full race suites hit timeouts under local machine contention; CI is the authority.

Test plan

• CI green on all required checks (Go race shards 1–8, build/boundaries, format/lint, Windows/Darwin, SDK, Frontend, Desktop, Preflight, E2E runtime, E2E web shards 1–4, E2E desktop)
• Manual: RT-upgrade-memory-removal-home (previous release home with memory enabled + compacted sessions → upgrade)
• Manual: RT-session-compact-real-adapters (Compact now + Goal compaction on claude-agent-acp and codex-acp)

🤖 Generated with Claude Code