Skip to content

A reloaded number is read against the field it fills - #713

Merged
David-Noble-at-work merged 4 commits into
developfrom
number-fidelity
Aug 27, 2026
Merged

David-Noble-at-work merged 4 commits into
developfrom
number-fidelity

Conversation

@David-Noble-at-work

Copy link
Copy Markdown
Contributor

Closes #711. Unblocks #709.

JSON has one number type, so unmarshalling into an any slot cannot say whether 420 was written as an
integer or a float. LoadGraph used plain json.Unmarshal, so every number in a reloaded graph came back as
float64 — an authored 0o644 reached an fs.FileMode as a float. Nothing noticed because op.Convert
truncated it back: the conversion layer was compensating for a serialization defect.

Two things were already true before #709 touched anything:

  • The codecs disagreed. yaml.v3 decodes an integer as int; JSON yielded float64. The same graph
    reloaded to different in-memory values depending on which codec wrote it.
  • float64 cannot represent every int64. A large enough integer in a saved graph was already
    corrupted on reload. Truncation made the corruption look like a clean number.

The rule

Convert from the value we serialize to the type of the field it fills.

LoadGraph decodes with UseNumber, so json.Number keeps the literal text. assembleBindings reads it
against the parameter — assembleNode has already resolved the action through the registry, so the declared
types are in scope at exactly the point the value is bound. YAML goes through the same path: it keeps an
integer an integer, but has no notion of the fs.FileMode the field declares.

Numbers only, and the first implementation got this wrong

Reading every value against its field is the literal reading of the rule, and it breaks a ruling. Convert
has a registered Resource construction step, so the URI string a resource-typed slot carries was built into
a live Resource at load time. §5.6 of docs/plans/resource-construction.md:

a reloaded graph's resource-typed slot carries the URI string the save left behind; at graph dispatch
that string is a KEY — it resolves through the section-rehydrated run catalog

The canonical form is computed from those values, so constructing early moved the checksum:

op.LoadGraph: checksum mismatch:
  document   "sha256:8e5bf1a1…"
  recomputed "sha256:1bcc33d7…"

That mismatch was the integrity check refusing a rule violation, not a fragile hash. Narrowing to numbers
is a scope statement rather than a workaround: only numbers are ambiguous coming out of a codec. A string is
already a string, and interpreting it is a decision the design assigned to dispatch.

Step 5 was the experiment, and it answered

The plan named checksum stability as the question that decides whether this reaches graph identity. Verified
rather than assumed: checksums are unchanged. This is contained in deserialization.

Two tests that passed for the wrong reason

Both caught by asking which path ran, not by trusting a green result.

The range test edited a saved document, so LoadGraph refused it for a checksum mismatch and the test's
early return fired without ever reaching range handling. Moved to Convert, where the behavior lives. The
ordering it exposed is worth keeping: integrity is a stronger gate than range and runs first, so a
graph-level range test cannot reach the code it claims to cover.

Step 8 asked whether this resolves the five save/load failures #709 produced. Running those five here
proves nothing — they pass on develop too, and only fail once #709's rule removes the truncation. Pasting
that rule on temporarily: all five pass, and the only remaining failure is #709's own
convert_struct_test.go, a literal float64 in a map reaching an int field with no JSON involved.

Out of scope

An any parameter, by ruling — #712. json.Marshal(float64(42)) emits 42, so a float in an any slot
loses its type at save and no load-side fix can reach it. There the document must carry the type.

Also included

docs/plans/TEMPLATE.md gains a Test Plan section. Zero of 118 plans had one, and the template not
providing it is why. It asks for a "fails when" column per row, failing tests written first, and an explicit
"not covered" list.

Verified

make check, make test-race, make test-scenario — all pass.

David Noble added 4 commits August 27, 2026 08:38
Zero of 118 plans in docs/plans carry one, and the template not providing a section is
why. Added between Implementation Phases and Migration Path, so new plans inherit it.

The section asks for three things a bare list of tests does not:

  A "fails when" column per row. A test that has only ever passed has not been shown to
  be a test. Twice this week writing that column was the thing that caught a problem: an
  assertion that turned out to be unwritable in the harness at all, and a case that did
  not fail the way the design document claimed.

  Failing tests written first. A test authored after its fix has to be re-proved by
  breaking the fix, which is slower and sometimes impossible -- on #708 that deadlocked
  the build, because removing the directive broke the tool that would put it back.

  An explicit "not covered" list. An untested property is a decision, not an oversight,
  and belongs where a reviewer can weigh it.
…ield

Plan for #711. json has one number type, so LoadGraph's plain json.Unmarshal yields
float64 for every number and an authored 0o644 reaches an fs.FileMode as a float.
Convert truncates it back, which is why nothing has noticed.

The rule: convert from the value we serialize to the type of the field it fills. Decode
with UseNumber so the literal survives, and convert at ASSEMBLY -- assembleGraph already
resolves each unit's action through the registry, so the declared parameter types are in
scope there. Converting later leaves json.Number in the graph's in-memory form, and the
canonical form is computed from that: a json.Number renders as a quoted string in yaml
where a float64 renders bare, which is the checksum mismatch a prototype produced.

Scope is parameters with a DECLARED type. An `any` slot is excluded by ruling (#712):
json.Marshal(float64(42)) emits 42, so a float in an `any` slot loses its type at SAVE
and no load-side fix can reach it. There the document must store the type.

The test plan's row 5 is the experiment rather than a regression guard: verify the
checksum does not move, BEFORE writing the conversion. Unchanged means the fix is
contained in deserialization; changed means graph identity is in scope and needs a
ruling before anything merges.
json has ONE number type, so unmarshalling into an `any` slot cannot say whether 420 was
written as an integer or a float. LoadGraph used plain json.Unmarshal, so every number in
a reloaded graph came back as float64 and an authored 0o644 reached an fs.FileMode as a
float. Nothing noticed because op.Convert truncated it back -- the conversion layer was
compensating for a serialization defect.

Two things were already true, before #709 touched anything
  The codecs disagreed. yaml.v3 decodes an integer as int; json yielded float64. The same
  graph reloaded to different in-memory values depending on which codec wrote it.

  float64 cannot represent every int64. A large enough integer in a saved graph was
  ALREADY corrupted on reload; truncation made the corruption look like a clean number.

The rule: convert from the value we serialize to the type of the field it fills
  LoadGraph decodes with UseNumber, so json.Number keeps the literal text. assembleBindings
  reads it against the parameter -- assembleNode has already resolved the action through the
  registry, so the declared types are in scope at exactly the point the value is bound.

  Yaml goes through the same path. It keeps an integer an integer, but has no notion of the
  fs.FileMode the field declares, so both codecs now reload to identical values.

NUMBERS ONLY, which the first implementation got wrong
  Reading EVERY value against its field is the honest reading of the rule and it breaks a
  ruling. Convert has a registered-Resource-construction step, so the URI string a
  resource-typed slot carries was built into a Resource at load time -- and §5.6 of
  docs/plans/resource-construction.md says that string stays a KEY until dispatch resolves
  it through the run catalog. The canonical form is computed from these values, so
  constructing early moved the checksum:

    op.LoadGraph: checksum mismatch:
      document   "sha256:8e5bf1a1..."
      recomputed "sha256:1bcc33d7..."

  That mismatch was the design refusing a rule violation, not a fragile hash. Narrowed to
  numbers, checksums are unchanged -- verified, which was the plan's step 5 and the question
  that decided whether this reached graph identity. It does not.

Two tests that passed for the wrong reason, both caught by asking which path ran
  The range test edited a saved document, so LoadGraph refused it for a CHECKSUM mismatch
  and the test's early return fired without ever reaching range handling. Moved to Convert,
  where the behavior lives. The ordering it exposed is worth keeping: integrity is a
  stronger gate than range and runs first, so a graph-level range test cannot reach the code
  it claims to cover.

  Step 8 asked whether this resolves the five save/load failures #709 produced. Running
  those five here proves nothing -- they pass on develop too, and only fail once #709's rule
  removes the truncation. Pasting that rule on temporarily: all five pass, and the only
  remaining failure is #709's own convert_struct_test.go, a literal float64 in a map
  reaching an int field with no json involved.

An `any` parameter is out of scope by ruling (#712): json.Marshal(float64(42)) emits 42, so
a float in an `any` slot loses its type at SAVE and no load-side fix can reach it. There the
document must carry the type.
…lly measured

Step 5 was written as the experiment rather than a regression guard: verify the checksum
does not move. It answered yes, but only after the first implementation moved it by
violating §5.6's resource-construction ruling -- which is a more useful answer than the
one the step was written to get. It says which values may be interpreted at load time at
all, not merely whether the hash is stable.

Step 8 records why checking #709's five failures on this branch proves nothing, and what
was done instead.
@David-Noble-at-work David-Noble-at-work self-assigned this Aug 27, 2026
@David-Noble-at-work
David-Noble-at-work merged commit d5b1375 into develop Aug 27, 2026
12 checks passed
@David-Noble-at-work
David-Noble-at-work deleted the number-fidelity branch August 27, 2026 17:50
David-Noble-at-work pushed a commit that referenced this pull request Aug 27, 2026
…on rules

The conflict was in convert.go and was expected: both branches added a step to the same
conversion ladder. Resolved by keeping BOTH, because they are complementary rather than
competing.

  tryParseSerializedNumber (#713) reads a serialized number AGAINST its field.
  losesMeaning (#709) refuses conversions that change meaning.

Neither subsumes the other. Verified after resolving: both are wired into Convert, each
of the four functions appears exactly once, and the suite left exactly one failure --
convert_struct_test.go, which is what step 8's measurement predicted before the merge.
David-Noble-at-work added a commit that referenced this pull request Sep 8, 2026
…catalog id, and the checksum covers the catalog rows (#858)

* test(op): failing tests for the any-slot type loss (#712)

A value in an `any` slot has no declared type at either end, so whatever
the document fails to record is gone. One failing test per finding, each
run and seen red for the reason it names.

Phase 1 measured two things the issue did not record:

- A number in an `any` slot makes the graph UNLOADABLE. The canonical
  form is computed from the reloaded values, so a number returning as a
  json.Number changes the recomputed checksum and LoadGraph rejects the
  document as tampered. A string and a bool reload cleanly; an integer
  and a float do not. This is a regression from #713, which replaced
  json.Unmarshal with a UseNumber decoder.

- A large integer loses digits on resume. recovery_stack.go decodes with
  plain json.Unmarshal into `Result any`, so an int64 of
  9007199254740993 reloads as 9007199254740992.

Also proven red: a []byte reloads as a base64 string, a Resource reloads
as its URI string, a non-finite float cannot be saved as JSON at all, and
IsTruthy(json.Number("0")) is true where float64(0) is falsy.

The plan document carries the design: the envelope is MongoDB Extended
JSON's shape with Go-native type names, written only where the schema
cannot answer -- `any` positions, plus non-finite floats at every float
position.

* wip(op): the type wrapper, wired at the graph slot seam (#712)

PRESERVES WORK IN PROGRESS. The tree is RED: three executor tests fail
with "encodeTypeWrapper: $resource ... has no catalog to name it".

Landed and green on its own:

- pkg/op/type_wrapper.go — the wrapper codec. MongoDB Extended JSON's
  shape with Go-native names ($bool, $bytes, $float64, $int64, $list,
  $map, $nil, $resource, $string), string payloads for numerics.
- pkg/op/type_wrapper_test.go — round trips over json and yaml, the
  2^53+1 integer, +Inf/-Inf/NaN, bytes, a map keyed like a wrapper, and
  refusal of bare values and unknown type names.

Wired, and six of the nine Phase 1 tests now pass:

- marshalBindings wraps a slot value whose parameter is `any` or absent,
  and any non-finite float regardless of declared type.
- assembleBindings unwraps structurally: a single $-prefixed key is a
  wrapper, everything else is a bare literal read against its field.

Known wrong, to fix first next session: encodeResource reads the catalog
from resource.RuntimeEnvironment(), which is the planning environment's,
not the graph's. Graph.resourceCatalog's own doc says a serializer must
read from the graph. The obstacle is that Node.MarshalYAML takes no
context and the canonical form marshals nodes through it, so reaching
graph.resourceCatalog needs a structural decision that has not been made.

The plan document carries the full work-in-progress state: the
uncommitted inventory, the devlore-test findings, and the open questions.

* docs(op): a cataloged resource names itself (#712)

encodeResource needs no catalog. A resource that has been cataloged already
carries its catalog id, so the write side asks the resource:

    func encodeResource(resource Resource) (map[string]any, error) {
        id := resource.ID()
        if id == "" {
            return nil, fmt.Errorf("op.encodeTypeWrapper: %s %q is not cataloged",
                typeNameResource, resource.URI())
        }
        return map[string]any{typeNameResource: id}, nil
    }

The catalog is the stamper, not the namer. ResourceBase holds id and
producerID, and its doc comment says who owns them: "stamped by the
ResourceCatalog when the resource is cataloged; they are not a concern of
the resource itself" (resource.go:144-149). The stamping is literal --
base.id = id at resource_catalog.go:828 and :936 -- and ID() returns that
field (resource.go:218). Identity is settled once, at catalog time.

What the code does today is wrong twice in one line. It asks
resource.RuntimeEnvironment().ResourceCatalog to Resolve the resource back
into an id. That catalog is nil by design at this point:
plan.Provider.Assemble nils it at provider.go:204 once NewGraph has taken
ownership, which is exactly what the three executor tests report. And
Resolve MUTATES -- it catalogs a resource that is not already present. A
serializer that interns into a catalog is writing where it was asked to
read.

Fixes, once applied: TestRun_AKindMismatchStopsEvenUnderIgnore,
TestRun_ScopedPreflightFailsOnConsumedMissingClaim,
TestRun_ScopedPreflightVerifiesConsumedClaims, and
TestLoadGraph_AResourceInAnAnySlotReloadsAsAResource. One defect, four
tests.

Supersedes the ruling recorded on the issue on 2026-08-29, which had
buildGraph stamp every node with its graph's catalog. That was wrong: it
re-plumbs a lookup that does not need to exist, adds a field to every node
to answer a question the resource already answers, and leaves the mutating
Resolve on the write path. No node changes, no graph.go changes.

Decoding is not symmetric and stays open: it holds an id and must find what
the id names, so it does need a catalog. assembleUnits / assembleNode /
assembleBindings thread env.ResourceCatalog; the graph's own catalog,
rebuilt by unpackCatalog from the document's resources section, is probably
the right one.

Also refreshes the work-in-progress section, which had gone stale: it still
listed the type wrapper and its call-site edits as uncommitted and warned
they would be lost, but ff33a2f landed all of them. The scratch tests and
test_path_* litter it told the next session to delete are already gone.

The tree on this branch is RED and stays that way: three executor tests
fail pending the fix above. This commit changes no code.

* Name this plan and branch after issue #712

The branch was feature/any-slot-types and the plan docs/plans/any-slot-types.md -- neither naming the
issue, so neither was reachable from the other or from #712. Both now carry it: the branch is
fix/712-any-slot-types and the plan is docs/plans/fix/712-any-slot-types.md, per the convention in #769.
The `bug` label puts it under fix/ rather than feature/, which is also what it always was.

develop merged at the same time. The branch was four behind and had already needed one catch-up merge;
the divergence only grows while the work waits its turn.

WHERE THIS SITS. Thread 2, resource management, sequenced AFTER #649 rather than before it. Decision 8
makes an uncataloged resource a hard error at save, and #649 is the defect that mints them: every
provider's ConvertFrom returns a value whose embedded op.ResourceBase stays zero, so URI() returns "" --
eight sites across appnet, git, pkg, service, and file's four variants. Landing this first would turn a
latent defect into a save-time failure across every provider, discovered mid-phase.

The thread's order is #644, #645, #649, #712, #646, #647.

#735 is the same seam from the other side and belongs to this thread. #734, #758 and #661 do not: a
number in an any slot and two document-format defects, an Epic:Serialization cluster with no thread.
Recorded rather than forced in.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* Record the resumed state of the any-slot plan: develop merged, the red set unchanged, one block lifted

Phase 8 of the sealed-resources plan landed first, as this plan's order asked, so Decision 8's save-time error is
unreachable by construction. #738 closed, lifting the devlore-test block. The six red tests are the six recorded
on 2026-09-01; the correction to apply first is unchanged.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* A cataloged resource names itself: encodeResource reads the stamped id, and two decisions the red tests forced are recorded

Decision 8 applied: the write side asks the resource for its catalog id and refuses an uncataloged one, needing
no catalog of its own; two of the three executor tests go green. The remaining red recorded two rulings in the
plan: framework-owned slots (on_missing, claim) are declared by the framework, not the method, so they are bare
like any declared parameter; and a slot's resource id resolves against the document's own unpacked catalog,
which assembleGraph must unpack before it assembles the units.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* Framework-owned slots are declared by the framework, and a slot's resource id resolves against the document's own catalog

Decision 9: frameworkSlotTypes declares the slots the executor reads by name (claim, on_missing); the serializer
leaves them bare like any declared parameter and the reader parses a bare policy back through UnmarshalText, so a
reloaded graph keeps the policy its author set. Decision 10: assembleGraph unpacks the document's catalog before it
assembles the units, and assembleUnits takes that catalog, so a $resource id resolves against the ledger the
document carries rather than the caller's empty environment. The any-slot Resource test catalogs its resource and
carries the catalog, on a fixture type whose URI the catalog can read back. Red set: the two phase-2 items.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* A decoded number with nothing to read it against is a load error, and a json.Number is falsy at zero

Phase 2 of the any-slot plan (#712). readAgainstField no longer hands a decoder artifact forward: a number in a
slot with no action, in a slot no parameter declares, or one its parameter's type cannot hold is an error at load
naming the slot, the parameter, and the declared type; the framework-slot reader refuses a non-text value the same
way. scalarTruthy gains the json.Number guard rail, with the numeric cases split out to stay under the complexity
gate. Two tests pin the refusals on hand-rewritten documents. Red set: the recovery-stack seam, phase 3.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* Receipts and the recovery stack envelope their results and slots, and a resumed resource resolves by its recorded id

Phase 3 of the any-slot plan (#712), first half. A receipt's result and slot values and the stack's stamped result
travel enveloped when the envelope names their type, and bare when result_type declares a product the registry
retypes; the large-integer resume test is green and the reloaded stack reproduces its document byte for byte. A
$resource decoded where no catalog exists is kept as a typed recorded id, and resolveRecordedResource looks it up
at rehydration instead of resolving a URI to whichever generation is current (#735); the tests pin the retirement.
RestoreEncoded restores the transaction id and the slots it had been dropping. The tree is green; make check gates.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* Size the second half of the any-slot plan's phase 3: declared resource slots by id, dispatch by id, the annotation seam

Five items against the tree at 62fadf3: a declared resource slot writes and reads its catalog id (#735 at the
declared seam); dispatch resolves the slot's resource onto the run clone by id and refuses a string; origin
annotations are enveloped per value; the variable seam is checked before it is touched; the providers' URI-constructing
unmarshalers are a follow-up sweep, not this phase.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* Resource slots record the catalog id and the checksum covers the catalog rows

A declared resource slot writes the resource's catalog id and reloads the
catalog entry that id names (#735); dispatch resolves a Resource by id and a
run-time string by key, as before. CanonicalContent carries the catalog's
intent rows (#712 decision 11), so a doctored `resources` row is a checksum
mismatch at load rather than an entry the plan never claimed.

Refs #712, #735.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* Annotations and variables carry their types, and the codec accepts Go's natural shapes

Origin and receipt annotations are enveloped per value and read back strictly;
a receipt's encoded restore now restores its annotations at all. A variable's
value is enveloped on write and decoded on read, since a paused run reads its
frame back from the trace, and a resource among them dispatches by its recorded
id. The codec names every integer kind, float32, typed slices and arrays, and
string-keyed maps by the document's own types (#712 decision 12).

Refs #712.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* A bare value or an unknown envelope in an any slot is refused, and the format is documented

Phase 4: a slot that carries its type by rule holds an envelope or nothing; a
bare value is refused naming the slot, the unit, and the type found, and an
envelope under an unknown name is refused by that name. The declared-float half
of the non-finite requirement is pinned.

Phase 5: the type envelope is stated in 5-graph-trace-integrity.md, the typed
slots and recovery serialization documents link there, and Tier 1 records that
the canonical form carries the envelope and the catalog's intent rows. The plan
is complete; #854 is the follow-up for the providers' unmarshal constructors.

Refs #712, #735.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

* Pin #735's acceptance: a slot binds its own generation, both seams agree, a ledger miss names slot and id

Three tests close the gap between #735's acceptance criteria and the tree.
The plan records the mapping, and records that #712's default-value
criterion depends on #857: plan.variable discards its default today, so
there is nothing in a document to envelope.

Refs #712, #735, #857.

Claude-Session: https://claude.ai/code/session_01SqQVFZz7AdiLZkGoAcUqRW

---------

Co-authored-by: David Noble <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON reload turns every authored integer into a float64, and the codecs disagree

1 participant