Repository navigation
devlore-test is broken: its primary artifact is empty, its streams are inverted, and its own tests cannot detect either #738
Description
Feature: #534 — Engine and combinator coverage
devlore-test run documents three output streams. None of the three descriptions matches what the stream
writes, two of them are effectively inverted, and the artifact a reader most wants — the execution trace — is
never written at all.
What the help text promises
cmd/devlore-test/devloretest/commands.go:69-73:
summary JSON test result (passed, node_count, failures) <script>.summary.json
graph The graph document from the software under test <script>.graph.yaml
receipt Full serialized execution graph <script>.receipt.<format>
What each stream actually writes
| Stream | Documented | Actual | Evidence |
|---|---|---|---|
summary |
passed, node_count, failures |
passed, unit_count, expectation_count, failures |
runner.go:42-45 — there is no node_count field |
graph |
"The graph document" | each t.run execution result, as JSON |
runner.go:295, test_context.go:854-863 |
receipt |
"Full serialized execution graph" | the graph document | commands.go:164 — writeReceipt(..., runner.Graph()), whose parameter is graph *op.Graph |
So the graph document comes out of the stream named receipt; the stream named graph emits run results that
are documented nowhere; and no stream writes a receipt.
Consequences
A silently empty artifact. test_context.go:854 skips a nil result, so <script>.graph.yaml is 0 bytes
for any script whose units return nothing. Measured: a script whose last unit is file.remove writes an empty
.graph.yaml, while test_hello.star writes 83 bytes because shell.exec returns a result. A reader
reasonably concludes the graph failed to serialize.
The existing test cannot catch it. cli_test.go:189 asserts only that the three artifacts exist. A
0-byte file exists.
No execution trace is persisted. GraphExecutor.Trace() is never called anywhere in cmd/devlore-test.
devloretest/trace.go is a Starlark step tracer behind --trace (script positions), unrelated to the op
trace. So receipts, results, and resource state transitions are discarded at the end of every run.
That last point has a practical cost: a plan whose producer creates a resource and whose consumer claims it
transitions that entry out of Pending during the run, and there is no artifact in which that is observable.
The receipt stream carries the graph with its PLAN-TIME catalog, which is legitimately empty whenever a
promise defers resource creation to execution.
Suggested shape
- Name the streams for what they carry, or make them carry what they are named.
- Document the run-result stream, which is currently undocumented.
- Add a trace stream, or route the executor's
Trace()into the receipt stream so "receipt" means one. - Assert artifact CONTENT in
cli_test.go, not merely existence.
Acceptance
- Every stream's help text matches what it writes
- The graph document is obtainable from a stream a reader would look in
-
t.runresults are documented wherever they are emitted - An execution trace is persisted, or its absence is a stated decision
-
cli_test.gofails if an artifact is empty when it should not be
Activity
[-]devlore-test output streams: all three descriptions are wrong, and no trace is written[/-][+]devlore-test is broken: its primary artifact is empty, its streams are inverted, and its own tests cannot detect either[/+]David-Noble-at-work commentedon Aug 29, 2026
Escalating: this is not a documentation defect
Filed as a help-text mismatch. It is worse than that — the tool's primary artifact is unusable for ordinary
plans, and nothing in the repository can notice.
<script>.graph.yamlis empty for essentially every planTwo runs of a two-node write-then-remove plan, one ordering by promise and one by path:
summary.jsongraph.yamlreceipt.yamlThe stream writes each
t.runresult and skips nil (test_context.go:854), so any plan whose units returnnothing produces an empty file.
file.write_textandfile.removeboth return nothing. The 83 bytestest_hello.starproduces are an accident ofshell.exechappening to return a value.A user looking for the graph document finds an empty file named
graph.yaml, and the actual graph in a filenamed
receipt.yaml.A test can pass having asserted nothing
{"passed":true,"unit_count":2,"expectation_count":0,"failures":[]}That run removed a file that was never created, under
on_missing=ignore, and reported success with zeroexpectations evaluated.
passed: truewithexpectation_count: 0should not be a quiet outcome.The harness's own tests cannot catch any of this
cli_test.go:189asserts the three artifacts EXIST. A 0-byte file exists. Every defect above survives thesuite untouched, which is why they are all still here.
Also still true, from the original report
GraphExecutor.Trace()is never called incmd/devlore-test, so receipts,results, and resource state transitions are discarded at the end of every run.
devloretest/trace.gois a Starlark step tracer behind--trace, unrelated to the op trace.summary's documentednode_countfield does not exist; the struct hasunit_count.Acceptance, revised
t.runresults are documented wherever they are emittedcli_test.goasserts artifact CONTENT, and fails on an unexpectedly empty artifactOne output convention for every app: result to stdout, documents to t…
Merge develop: the output convention, and the devlore-test rewrite (#740
Record the resumed state of the any-slot plan: develop merged, the re…
A value in an any slot carries its type, a resource slot records the …