Repository navigation
One output convention for every app: result to stdout, documents to the store - #744
Merged
Merged
Conversation
added 8 commits
August 28, 2026 23:51
Every binary presents the same command-line surface: a result to stdout, JSON by default; narration to stderr; documents -- definitions and traces -- to the execution store. One shared flag set decides how the result is rendered and where the store lives. The mechanism already existed. cmd/internal/cli/output.go binds four flags and BuildPipeline composes a result.Pipeline over pkg/result and pkg/sink. What never existed was a statement that using it is mandatory, or a way to point at the execution store. One command in forty-six calls AddOutputFlags. Reserved: --filter, --jq, --output/-o, --store, --template. Renderings: csv, json, none, table, template, yaml. No --output <file>, because stdout already is one. No boolean --json, because a boolean cannot express a third format. Settled against aws, az, docker, gcloud, and kubectl, with the survey recorded as a table rather than asserted. The --output versus --format split is 3-2; the case against the chosen name is written into the decision rather than omitted. --output none turns the result off, which `> /dev/null` cannot do -- redirection discards a rendered result, and a config file has no shell. The status companion records what this measured: 12 hand-rolled output flags, 13 fmt.Print calls in lore, 7 direct os.Stdout writes in writ, and devlore-test's three streams wired to the wrong artifacts (#738). It also records that adoption is decaying rather than lagging: extract-output- package.md counted two call sites in March, and writ snapshot was removed taking one with it, unrecorded.
Phase 1 of the output convention. make check passes.
The flags:
- --format becomes --output with the -o short form, matching aws, az,
and kubectl. It retires by deletion, not by alias.
- --store names the execution store's root.
- The set binds to PersistentFlags, so one call on a program's root
covers every subcommand -- the registration model all four in-scope
programs will use.
- `none` joins the formatters: a rendering that produces no result at
all. Not `> /dev/null`, which renders and then discards; this never
renders, and it is reachable from config and env where no shell exists
to redirect.
The store:
StoreHome() is the one root everything derives from -- the graphs
directory, the traces directory, the run index, and the fsroot anchor the
writers open. Relocating leaves while the anchor stayed at the state home
was the first implementation, and the round-trip test caught it:
mkdirat ../../002/traces/sha256-5eaf00d... : path escapes from parent
It also caught IndexPath() never consulting the store root, which would
have left a relocated store's traces in one place and the index recording
them in another. Its doc comment already claimed "at the store root" and
had been wrong before this change.
TestStoreRoot_RelocatesTheWholeStore pins all of it, using different
directories for the state home and the store root so anything consulting
the wrong one is caught rather than coinciding: the trace lands under the
store root, the index is under the store root, LoadLatestTrace finds the
trace by its graph checksum, and the index carries the entry.
Docs: the spec settles one table formatter for the suite, with lore's
hand-rolled runSearch table named as Phase 4's first conversion rather
than as an argument for a second one.
Nothing consumes opts.Store yet; no root calls SetStoreRoot. That is
Phases 2 through 4, where each program adopts the set.
#740) Phase 2 of the output convention. make check passes. devlore-test now obeys the law. Before and after: result <script>.summary.json -> stdout, json by default narration stderr -> stderr, unchanged definition <script>.receipt.yaml -> the store, keyed by checksum trace DISCARDED -> the store, under its definition t.run values the stream named "graph" -> the summary's results field That closes all four defects in #738. The graph document was being written to the stream named "receipt" while the stream named "graph" emitted t.run return values and wrote nothing when a unit returned nil -- which is why <script>.graph.yaml was empty for every file-based plan. The execution trace was one line away the whole time: executor.Trace() sat unused beside executor.Run in test_context.go. Persisting it is what makes a run's resource-state transitions observable at all. Retired by deletion, not aliased: --output stream=dest routing, --receipt-format, the outputFlags Value type, openDest, writeSummary, writeReceipt, and WithWriter. This supersedes the 2026-08-20 routing ruling, which sent all three payloads to files and kept stdout empty. That ruling predates the three-stream split and treated one category where there are two: the summary is a result, the definition and trace are documents. TestCLI_DefaultsToArtifactFiles pinned the old behavior and is replaced by TestCLI_ResultToStdoutStoreForDocuments, which asserts all of it -- result on stdout, narration on stderr, documents in the store, working directory untouched. Tests now assert artifact CONTENT, which #738 records as the reason these defects survived: the old suite checked only that the three files existed, and a 0-byte file exists. One stale assertion cost four false failures while doing this: assertValidSummary scanned for the first line starting with "{" and parsed that line alone -- correct for the old compact summary, wrong against indented json. The output was right; the test was not. Docs: section 14 recorded star as having "no result flags at all". It has a second copy of the cli package instead (#743), whose renderTable is the candidate to promote into pkg/result for Phase 4 rather than a thing to rewrite. Root internal/ holding cmd-only code is #742.
Phase 3a of the output convention. make check passes. writ registers the common set on its root, so every subcommand accepts every flag. --json is deleted rather than aliased: a boolean cannot express a third format, which is why writ status and writ verify could not emit yaml. verify computes; the command renders. verify.Execute returned an error and printed reports itself, choosing json or human text from cfg.JSON. It now returns []Report and the command emits them through the pipeline, which is where rendering belongs. The reports are emitted whether or not the signing policy rejected -- a rejection is the answer to the question, not a reason to withhold it. Two consequences worth stating plainly: - writ verify now emits JSON by default. presentReport was its human renderer and is deleted with the responsibility it carried. The friendly output returns as `-o table` once the shared TableFormatter lands (#741, #743). - writ status is bridged, not converted. cfg.JSONOutput now reads outputOptions.Format == "json" instead of a retired boolean, so its 22 fmt.Print calls still render themselves until phase 3b. Phase 3 is split. The status report is a human table, and converting it needs the TableFormatter currently scheduled in phase 4 -- so 3b (status, the four dry-run SerializeGraphs dumps, and migrate's own --format) waits on that. The plan records the split. Also corrected: four flag descriptions read "Promise <noun>" where they meant "Output <noun>", from a global replace that caught help text (#739). Three had shipped into the generated CLI reference under docs/cli/lore/. My own count of writ's stdout writes was wrong when phase 3 was written: I grepped the literal os.Stdout and missed every fmt.Print, which writes to stdout just as surely. The real figure is 30 call sites, not seven -- 22 of them in status/report.go.
…ent (#740) make check passes. One delimited formatter, two presets. csv and tsv differ in two attributes at once -- a spreadsheet wants a comma and a header row; `awk '{print $2}'` wants a tab and no header row to skip -- so each gets a name rather than a flag. gcloud reached the same shape: its `value` format is documented as "CSV with no heading and <TAB> separator instead of <COMMA>". The header inference, csv tag overrides, and RFC 4180 quoting are shared, untouched. The template formatter is gone. Reshaping a value is the filter stage's job, and --jq does it for every format; a template is the answer tools reach for when they have no query language. Of the five surveyed, aws, az, and gcloud have a query language and no templates; kubectl and docker have templates instead. Nobody has both, because they solve one problem. FormatterByName takes one argument now. The spec gains section 8, which draws the pipeline: two stages, one flag each. It also records the ruling that a format needing an argument carries it in the value -- `--output template=<body>` -- rather than in a sidecar flag. A sidecar gives the format stage two inputs and a mutual-exclusion rule to enforce; NAME=ARGUMENT makes the conflict impossible by construction instead. kubectl ships that exact form (-o go-template=, -o jsonpath=, -o custom-columns=). Presets stay named: csv and tsv are two values a user can guess, where csv=tab is one value with an argument to look up. Not built yet, and recorded as phase 3c: the NAME=ARGUMENT parse itself, template=<body>, and `value` -- the raw rendering that makes --jq complete, since a jq-built string has no other format that prints it as written.
#740) Phase 3c. make check passes. --output takes NAME or NAME=ARGUMENT, split on the FIRST '=' so an argument containing '=' survives. A format needing an argument carries it in the value rather than in a sidecar flag: a sidecar gives the format stage two inputs and a mutual-exclusion rule to enforce, where this makes the conflict impossible by construction. kubectl ships the same form (-o go-template=, -o jsonpath=, -o custom-columns=). Three malformed specs now error, each naming the format: a missing argument where one is required, an empty one after '=', and one given to a format that takes none. `value` renders raw -- no quoting, no document syntax, no header. It is what completes the filter stage: --jq can build any line, and every other format then imposes a syntax on it. json quotes and escapes, csv adds a header, yaml applies scalar rules. aws ships this as `text` and gcloud as `value`, and like gcloud's it expects a projection first: devlore-test run --jq '.unit_count' -o value → 1 Applied to a whole nested struct it prints every field, pointer addresses included. That is the same constraint gcloud documents by requiring a projection for csv and value, not a defect to paper over. template returns in the shape the spec describes: reached only as template=<body>, parsed once at construction, with no flag to pair it with. The previous implementation was removed two commits ago for being the sidecar form. Also fixed, found by running the binary rather than the tests: --store rejected a relative path with "not an absolute path on this platform". OpenTree requires absolute paths deliberately -- a drive-relative path on Windows anchors to whichever drive the process is standing on (#392) -- so SetStoreRoot absolutizes at the seam and returns an error, keeping the invariant true for every later reader instead of at each one.
…le (#740, #741) make check passes. DelimitedFormatter carries three attributes -- Separator, SuppressHeadings, Raw -- and ships three presets, which is gcloud's shape: its csv and value are one renderer differing in separator, heading, and quoting, with value documented as "CSV with no heading and <TAB> separator instead of <COMMA>". csv , heading RFC 4180 quoting spreadsheets tsv tab no heading RFC 4180 quoting cut -f2, awk value tab no heading raw text --jq already built Quoting is what separates the last from the middle. A machine-parseable format must quote a field containing its own delimiter or the row loses a boundary; a raw renderer must not, or text a caller composed comes back wearing quotes it did not write. The formatter now accepts every shape rather than only rows-with-fields. A non-sequence is one row -- `--jq '.count'` produces exactly that, and refusing it made the filter stage incomplete. A slice of scalars is one column with no header, since a scalar has no field to name. A byte slice is one value rather than a sequence of numbers. Two tests asserted refusals that are now supported shapes, and are rewritten as Accepts with the reason recorded rather than deleted. That folding fixed tsv as a side effect: `-o tsv` on a scalar said "expected slice or array, got int64" before, because value was a separate implementation rather than a preset of the same one. The redundancy was in the implementation, not the name. It also simplified what it touched. With scalars no longer an error case, csvHeadersFromElements and csvRowFromElement always returned nil errors; lint caught it, and both lost their error returns along with the handling at three call sites. TableFormatter shares that column inference -- HasHeaders, csv:"name" overrides, csv:"-" skips, the map-key union -- so one value renders the same columns as table, csv, or tsv, and only the presentation differs. A test pins it, because divergent column logic is how one rendering ends up showing a field the other hides. Alignment is text/tabwriter's, which measures in RUNES. That is half of #741 fixed at the shared formatter rather than at each call site: a hand-rolled %-30s counts bytes, so a row with a multi-byte character is short by the difference and every column after it shifts. The test uses a 7-rune, 21-byte name and asserts the next column starts at the same rune offset on both rows. The set is csv, json, none, table, template=BODY, tsv, value, yaml. Files renamed: csv.go and its tests hold three presets now, not one format.
make check, make docs, and make test-scenario all pass. The command help strings documented flags this branch retired. docs/cli is gitignored -- generated by `make docs`, never committed -- so the staleness lived in the Go source, and regenerating only proved the fix. #739's title says the corruption "shipped into the docs"; it did not, and that claim is corrected on the issue. A fourth #739 site: lore inspect's long description read "Promise is JSON by default ... Use --format", in prose rather than in a flag string, so a grep aimed at flag descriptions never saw it. Same class of miss as counting writ's stdout writes by grepping os.Stdout and missing 24 fmt.Print calls: a grep for one syntactic form undercounts a semantic problem. Section 15 claimed no root registers the common set. Two of four do. It now also states what it had been carrying silently: writ status is bridged, not converted, so `writ status -o yaml` produces the human report rather than yaml, until its report goes through the pipeline. Plan and status checkboxes were stale -- phases 1, 2, and 3c were complete with every box unticked. Reporting progress in conversation while the tracked document says nothing is done is the failure that rule exists to prevent. extract-output-package.md is corrected. It was marked complete while describing an internal/output package that was never created; the code went to pkg/result and pkg/sink. It also holds the last measurement of adoption -- two call sites, one of which vanished with writ snapshot and went unrecorded -- which is the evidence behind the enforcement invariants. Also written down, having been noticed and left: the jq-versus-JMESPath divergence now sits in section 16 with the rest, and section 7 states that `value` expects a projection, as gcloud requires one for its csv and value. Two test defects, both invisible to make check: - scenario_integration_test.go invoked `writ status --json`, a flag deleted several commits ago. make check does not run scenario tests, so it would have gone red on CI rather than locally. - TestCLI_BadOutputPath passed for the wrong reason: after --output became a rendering, "graph=/no/such/dir/out.txt" parses as an unknown format named "graph" and exits non-zero without reaching a filesystem. It is now TestCLI_UnwritableStore, against the flag that takes a path.
added 3 commits
August 30, 2026 15:06
Both Windows test jobs failed on PR #744: --- FAIL: TestCLI_UnwritableStore cli_test.go:290: an unwritable store exited 0, want non-zero The test asserted unwritability with "--store /no/such/dir/store". That path is unwritable only because Unix denies a non-root user writes under /. Windows CI runs as administrator and filepath.Abs resolves the same string to C:\no\such\dir\store, which MkdirAll simply creates. Exit 0, and the test was right to complain. The store root now goes under a regular file, which MkdirAll cannot descend through anywhere: ENOTDIR on Unix, ERROR_DIRECTORY on Windows. Verified against the built binary rather than inferred -- it exits 1 with "writing the definition to the store: open tree .../blocker/store: create: openat .../blocker: not a directory", so the assertion on "store" matches the message and not merely the path that was passed in. Renamed to TestCLI_UncreatableStore: "unwritable" was never what it asserted. The store root cannot be created. This is the second time this test passed or failed for a reason other than the one it names -- it previously asserted a bad --output destination that parsed as an unknown format, exiting non-zero without reaching a filesystem at all. The doc comment now records both, so neither form comes back.
status was still "draft" with four phases complete, and updated still read
2026-08-28. Stale plan metadata is what makes the documentation system
untrustworthy: a reader cannot tell a plan that has not started from one
that is most of the way done.
The epic body carried two harder errors, corrected on the issue itself:
- it cited 9-command-line-interface.md; the spec is 10-.
- it listed --template as a reserved flag and template as a plain
rendering. --template was deliberately removed. A format that needs an
argument carries it in the value, split on the first "=", which is why
there is no sidecar flag and no mutual-exclusion rule. The rendering
list also omitted tsv and value.
pkg/result/selection.go:19 is the authority on the format names and says
so; cmd/internal/cli/output.go:93-99 registers the four flags. Both were
checked against the code rather than restated from the epic.
The epic's progress narrative was refreshed at the same time. It still
said "one command out of forty-six calls AddOutputFlags" and listed
devlore-test and writ's flags as remaining, both of which have landed.
…740) tsv was added on an awk/cut rationale that does not survive inspection. Those tools have no quote awareness, so quoting a field that contains a tab does not stop the row from splitting -- cut -f2 returns "a and b" rather than a and b. The quoting helps only a caller running a real parser, and that caller is better served by csv, whose library support is universal. That leaves nothing a quoted tab format does better than both of its neighbours. gcloud and aws each ship one raw delimited format -- value and text -- and no tsv. The naming is why: csv and tsv promise that a parser can round-trip the output, and a format that does not quote cannot keep that promise. Neither tool claimed the name. Ours could have, since ours quoted, but earning the name and earning a slot are different questions. So the pair now splits by consumer rather than by separator. csv quotes, so a parser round-trips it. value does not, so a shell reads exactly what was composed, which is what makes --jq complete. DelimitedFormatter keeps all three attributes; Raw is what separates the two survivors. Also fixes the --output help, which never listed `table`. The rendering has been accepted by FormatterByName since the TableFormatter landed earlier in this branch, so it worked and was undiscoverable. The text now names what each rendering is for rather than listing them flat: Output rendering: json, yaml, table, csv, value (raw, pairs with --jq), template=BODY, or none The reasoning is written into 10-command-line-interface.md §8 rather than only the outcome, because "why is there no tsv?" is a question that will be asked again. The plan, the status document, and the epic body carry it too. BREAKING: --output tsv is no longer accepted. Use value for raw tab- separated output, or csv where the consumer parses.
Closed
3 tasks
added 2 commits
August 30, 2026 16:18
Four tracking items were still unchecked while their work was done: the --store implementation, the --format to --output rename with the none renderer, devlore-test, and writ's flags. A status document that lags the code is worse than none, because it is read as authoritative. writ's single item is split, since only half of it landed. Its flags are done -- the boolean --json retired by deletion on status and verify, verify.Execute returning []Report for the command to emit, the common set on the root. Its renderings are not: 30 stdout call sites remain. Leaving one checkbox to cover both would have had to be either wrong or unchecked, and unchecked would have hidden the flag work. star gets its own item. It was buried inside the lore line, which made a separate app's adoption invisible. The "current cost" measurement keeps its 7-direct-os.Stdout-writes figure as recorded, with the correction beside it: that number came from grepping the literal os.Stdout, which misses every fmt.Print, and those reach stdout just the same. The real figure is 30. The wrong number is left visible because how it was miscounted is the useful part. The plan's Current State section is date-stamped rather than rewritten. Its adoption figures are the baseline that justifies Phase 5, and updating them as work lands would erase the evidence that adoption decays when it is not enforced.
…aml (#736) Picks up #736, which landed on develop 67 seconds after this branch's last CI run was created -- so every green result on this PR so far, including the 12/12 on cece6c9, tested a merge ref that did not contain it. No files are shared. #736 touches .goreleaser.yaml and docs/plans/release-path.md; this branch touches pkg/result, cmd/internal/cli, cmd/devlore-test, cmd/writ, and its own docs. The one coupling is verify-ldflags, which greps .goreleaser.yaml for Version, Commit, and BuildDate against pkg/application: #736's new star build block carries all three, and the check is a whole-file grep, so an added block cannot break it. This merge exists so CI tests the combination rather than the assertion that it is safe.
David-Noble-at-work
pushed a commit
that referenced
this pull request
Aug 31, 2026
) This branch was cut from f3c8769, a develop from before #744. Bringing it current picks up the whole output convention, and one part of it matters to #712 directly: #738 is closed, so devlore-test now persists an execution trace. That is what a test needs to observe a resource's Pending to Active transition, which this plan's Blocked-on section named as a blocker. No overlap is expected. #744 changed pkg/result, cmd/internal/cli, cmd/devlore-test, cmd/writ, and docs; this branch changes pkg/op and its own plan. The tree here is RED and stays that way: three executor tests fail on $resource ... has no catalog to name it, pending the encodeResource fix ruled in decision 8.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the spec added by this branch:
docs/architecture/10-command-line-interface.md.Every binary presents the same surface. A result goes to stdout, JSON by default. Narration goes to stderr.
Documents -- definitions and traces -- go to the execution store. One shared flag set decides how the result
is rendered and where the store lives, and no command invents its own.
Closes #738. Closes #739. Advances #740. Partially addresses #741.
Breaking, and deliberate
Greenfield, so a flag retires by deletion rather than by alias -- an alias is the compatibility shim the
governing principle forbids, and it doubles the surface every later change must consider.
writ status --json,writ verify --json-o jsondevlore-test --output summary=PATH(x3 streams)--storedevlore-test --receipt-format-o json/-o yaml<script>.summary.json,.graph.yaml,.receipt.yamlin the cwdwrit verify's human report by default-o table; JSON is the default-o tsv-o valuefor raw,-o csvwhere the consumer parsesThe flag set
--filter,--jq,--output/-o,--store. Renderings:csv,json,none,table,template=BODY,value,yaml.csvandvalueare the delimited pair, and they split by consumer rather than by separator:csvquotes,so a parser round-trips it;
valuedoes not, so a shell reads exactly what was composed, which is what makes--jqcomplete. A quoted tab rendering (tsv) was carried on this branch and dropped before merge --cutand
awk -F'\t'have no quote awareness, so quoting a field containing a tab does not stop the rowsplitting; it yields
"aandb"rather thanaandb. Quoting helps only a caller running a real parser,and that caller is better served by
csv. gcloud and aws each ship one raw delimited format and notsv,and the naming is why:
tsvpromises round-tripping an unquoted format cannot keep.Settled against
aws,az,docker,gcloud, andkubectl, with the survey recorded as a table in thespec rather than asserted. The
--outputvs--formatsplit is 3-2; the case against the chosen name iswritten into the decision rather than omitted.
A format needing an argument carries it in the value --
--output template=<body>-- rather than in a sidecarflag. A sidecar gives the format stage two inputs and a mutual-exclusion rule to enforce;
NAME=ARGUMENTmakes the conflict impossible by construction. kubectl ships that exact form.
What #738 actually was
The stream named
receiptwrote the graph document. The stream namedgraphemittedt.runreturn valuesand wrote nothing when a unit returned nil -- which is why
<script>.graph.yamlwas empty for everyfile-based plan. No execution trace was persisted at all:
executor.Trace()sat unused besideexecutor.Run. Andcli_test.goasserted only that the three artifacts existed, which a 0-byte filesatisfies. All four are fixed, and the tests now assert content.
Not done, and tracked
Ten tasks remain on #740, all adoption rather than mechanism:
writ status's 22fmt.Printcalls, four dry-runSerializeGraphs(os.Stdout, ...)dumps, andmigrate'sown
--format.writ statusis bridged, not converted --cfg.JSONOutputreadsoutputOptions.Format == "json", sowrit status -o yamlsilently produces the human report. Section 15of the spec says so.
loreandstardo not register the common set.cmd/star/cliduplicates eighteen exported names fromcmd/internal/cliand now contradicts it (cmd/star/cli duplicates cmd/internal/cli, and the two now disagree on the flag surface #743).lore'srunSearchstill slices descriptions by byte count. The alignment half isfixed at the shared formatter, which measures in runes.
os.Stdoutwritefrom a command package. Both are greppable and both are red today.
A test that passed for the wrong reason, twice
TestCLI_UnwritableStoreasserted a bad--outputdestination until--outputbecame a rendering; left asit was,
graph=/no/such/dir/out.txtparsed as an unknown format and exited non-zero without touching afilesystem. Rewritten to use
--store /no/such/dir/store, it then failed on both Windows runners: that pathis unwritable only because Unix denies a non-root user writes under
/, while Windows CI runs asadministrator and simply creates
C:\no\such\dir\store.It now puts the store root under a regular file, which
MkdirAllcannot descend through on any platform, andit is named
TestCLI_UncreatableStorebecause "unwritable" was never what it asserted. Verified against thebuilt binary rather than inferred: it exits 1 with
writing the definition to the store: ... not a directory, so the assertion matches the message and not merely the path passed in.Verification
make check,make docs, andmake test-scenarioall pass, includingTestWritDeployScenario_Deploy.CI is green on all twelve jobs across darwin, linux, and windows on both architectures.
docs/cliis gitignored here, so no reference pages appear in this diff -- but it is published:docs-publish.yamlregenerates it on every push todevelopand auto-merges the result intodevlore.noblefactor.com. The corrected help strings go live when this merges, with no human in the path.That is worth knowing when reviewing the four changed strings: they are published prose, and they get no
diff here and no approval there.
make checkdoes not run scenario tests. That gap hid a brokenwrit status --jsoninvocation inscenario_integration_test.gothrough several commits on this branch; it is fixed here, and it is theargument for the enforcement tests above.