Skip to content

Every presentation is a presentation of the JSON: formatting rules, list, and stage 1 - #747

Merged
David-Noble-at-work merged 5 commits into
developfrom
feature/740-formatting-rules
Aug 31, 2026
Merged

David-Noble-at-work merged 5 commits into
developfrom
feature/740-formatting-rules

Conversation

@David-Noble-at-work

@David-Noble-at-work David-Noble-at-work commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Writes down the rules that take a Go value to JSON to a presentation, then fixes the four places the code
disagreed with them. The rules were written first and judged the code; three of the four are defects found
that way rather than reported.

Advances #740. Spec: docs/architecture/10-command-line-interface.md §8.

Closes #748. Closes #749. Closes #750. Closes #751. Closes #752. Closes #753. Closes #754. Closes #755.

Eight defects, all found by writing the rules first and then rendering a real result through every formatter
and reading the bytes -- not by review.

Defect Commit
#750 field names differed by unrelated flag 1
#751 a nested cell rendered as Go's map[runs:3] 1
#752 a single-object result had no correct rendering 1
#748 a table cell's newline split its row, silently 2
#749 --jq calling tostring panicked the process 2
#753 --store was registered and read nowhere in writ 3
#754 --output was accepted unvalidated on every writ command but verify 3
#755 help text did not wrap; every line width was maintained by hand 4

#753 and #754 are one defect wearing two faces: a flag registered on a root that no leaf consumes.
AddOutputFlags now wires the two flags whose meaning does not depend on a command rendering anything, so
registering the set and honoring it cannot come apart.

One regression was introduced and fixed inside this PR: keeping json.Number through stage 1 made -o yaml
answer unit_count: "2" where -o json answered 2, because encoding/json special-cases the type and
yaml.v3 sees a string. The conformance suite passed it throughout -- it asserted the digits were present,
and they were, in quotes. A second test now asserts the type.

Stage 1 is real

normalize moved from inside the jq filter to the head of Pipeline.Emit, so the filter and the formatter
see the same data.

-o list            →  UnitCount : 1        ← before
--jq . -o list     →  unit_count : 1       ← before

One result, two vocabularies, decided by an unrelated flag. The json: tag is the name the Starlark surface
shows a customer, and it is now the name every rendering shows.

Numbers stay json.Number. Decoding to float64 rounds any integer past 2^53 — the defect #712 records
against the document codec. gojq needs int64 and float64 and gets them inside the jq filter, where the
conversion is gojq's requirement and not this pipeline's.

The defects the rules exposed

Was Now
Field names UnitCount, unless --jq was passed unit_count, always #750
A nested cell map[runs:3] — Go's notation, unparseable {"runs":3} #751
A single record, -o table expected slice or array, got map[string]interface {} a header and one row #752
A single record, -o csv the whole record in one cell a header and one row #752
A cell containing a newline, -o table the row split, silently \n, one record per line #748
--jq calling tostring on a number panic: invalid type: int64 (1) the expression runs #749

#748 — every shell result carries a stdout field, and stdout ends in a newline, so a split row was
the ordinary case. csv is deliberately untouched: RFC 4180 quoting already keeps the newline inside its
field where a parser reads it, and escaping there would corrupt a value that round-trips today.

#749 — gojq accepts nil, bool, int, float64, *big.Int, json.Number, string, []any, and
map[string]any. int64 is not among them, and jqUnwrapNumbers handed it exactly that from v.Int64().
The conversion was never needed — gojq handles json.Number natively across compare.go, encoder.go, and
a dozen sites in func.go — and it was lossy, since Int64() fails past int64 and the fallback rounds.
Deleted.

Types that render themselves are excluded from the compact-JSON rule: an error, a fmt.Stringer, an
encoding.TextMarshaler. A time.Time is a struct, and encoding it would replace the form it defines for
itself.

list

One field per line, keys aligned within a record, each record keeping its own keys. That last point is
why it exists: the delimited formats union keys across the stream and leave holes, so a heterogeneous stream
renders mostly holes.

name  : x
state : active

name     : y
runs     : 3
findings : ["a","b"]

gcloud ships list and flattened separately; one suffices here because the compact-JSON cell rule
already handles the nesting flattened exists to spread out.

The survey behind the rules

Measured against pwsh 7.5.4, not recalled. PowerShell's inline rendering of a nested value is adopted. Three
of its behaviors are rejected, and all three have one cause — it formats .NET objects by their properties
without normalizing first
:

Behavior Output Cause
Depth truncates at the third level @{c=}, the value silently gone the property walk stops
Type names leak System.Collections.Hashtable where an object gives @{k=v} different .NET types
Arrays are reflected over Length LongLength Rank SyncRoot ... an array is an object with properties

Stage 1 makes all three impossible here.

Its automatic table/list switch at five properties is rejected too: §10 already refuses TTY adaptation
because a pipeline that behaves differently when observed is unreproducible, and switching on property count
is that defect with a different trigger.

One divergence is deliberate. Format-Table takes its columns from the first object and blanks every key
the later ones introduce, dropping them with no report. csvHeadersFromMaps unions across every element. A
sparse table is a worse presentation than a dense one; silently dropping columns is a worse answer.

Tests

A conformance suite over one fixture carrying every property the rules turn on: json: tags that differ from
the Go names, a nested object, an array, an integer past 2^53, a bool, and an empty string. Every formatter
runs against it through a real pipeline, which is the only path that applies stage 1. A formatter that
disagrees with the others about a field's name, or about what a nested value looks like, fails there rather
than in a command months later.

What this does NOT fix: the suite

Each issue above describes writ, and writ is fixed. The class survives elsewhere, because a repair in
cmd/internal/cli reaches exactly the programs that route through cmd/internal/cli. Measured after these
commits:

App Help wraps (#755) --output validated, --store resolved (#753, #754)
writ yes yes — registered on the root, so every command
lore yes inspect only
devlore-test no yes — registered on the root
star no no
COLUMNS=70, longest flag line
  writ           70
  lore           70
  devlore-test  389   <- unwrapped
  star           65
  • devlore-test builds its root directly instead of through cli.NewRootCmd, so it gets AddOutputFlags
    but not the help wrapping.
  • star binds its own cmd/star/cli.AddOutputFlags. A fix to the shared package cannot reach a second copy
    of it — the concrete cost of cmd/star/cli duplicates cmd/internal/cli, and the two now disagree on the flag surface #743, and the argument for deleting the duplicate rather than maintaining it.
  • lore registers the set on inspect rather than its root, so every other lore command lacks the flags
    entirely — the epic's "1 of 46 commands", still true.

Tracked by #743 and by tasks 35–37 of the plan. Recorded in §15 so a reviewer does not read these fixes as
suite-wide, and so the next person fixing something in cmd/internal/cli knows what it will and will not
reach.

Verification

make check passes. make docs regenerates clean. Every formatter, against one real result:

What #749 was blocking — gcloud's text format, needing no format of ours:

--jq '[paths(scalars) as $p | "\($p|map(tostring)|join(".")): \(getpath($p))"]' -o value

results[0].command: echo "Hello World!"
results[0].exit: 0

That is §8's rule working as written: a command needing a shape the set does not have reshapes with --jq
and renders with value.

-o json    {"command": "echo \"Hello World!\"", "exit": 0, ...}
-o yaml    command: echo "Hello World!"
-o table   COMMAND              EXIT  STDERR  STDOUT
-o list    command : echo "Hello World!"
-o csv     command,exit,stderr,stdout
-o value   echo "Hello World!"	0		Hello World!

Breaking

table, csv, list, and value now name fields by their json: tags rather than by their Go
identifiers, and a nested value renders as compact JSON rather than through fmt.Sprint.

docs/cli is gitignored here, so no reference pages appear in this diff — but docs-publish.yaml
regenerates it on every push to develop and auto-merges the result. The --output help text changes go
live when this merges, with no human in the path.

Writes down the rules that take a Go value to JSON to a presentation, then
fixes the four places the code disagreed with them. The rules were written
first and judged the code; three of these are defects found that way rather
than reported.

STAGE 1 IS REAL. normalize moved from inside the jq filter to the head of
Pipeline.Emit, so the filter and the formatter see the same data. Before
this, `-o list` named a field UnitCount and `--jq . -o list` named the same
field unit_count -- one result, two vocabularies, decided by an unrelated
flag. The json tag is the name the Starlark surface shows a customer, and
it is now the name every rendering shows.

Normalization keeps json.Number. Decoding to float64 rounds any integer
past 2^53, which is the defect #712 records against the document codec.
gojq needs int64 and float64 and gets them inside the jq filter, where the
conversion is gojq's requirement and not this pipeline's.

A NON-SCALAR CELL IS COMPACT JSON at any depth, never truncated. It went
through fmt.Sprint before, so a nested map rendered as Go's `map[runs:3]`
-- a notation that names the language rather than the data, and that
nothing downstream can read. Types that render themselves are excluded: an
error, a fmt.Stringer, an encoding.TextMarshaler. A time.Time is a struct,
and encoding it would replace the form it defines for itself.

A SINGLE RECORD IS ONE ROW. table refused an object outright and csv
rendered the whole record into one cell: two different wrong answers to the
shape a command produces whenever it reports on one thing.

LIST ADDED. One field per line, keys aligned within a record, each record
keeping its own keys. That last point is the reason it exists: the
delimited formats union keys across the stream and leave holes, so a
heterogeneous stream renders mostly holes. gcloud ships list and flattened
separately; one suffices here because the compact-JSON cell rule already
handles the nesting flattened exists to spread out.

The spec records the survey behind these, measured against pwsh 7.5.4
rather than recalled. PowerShell's inline rendering of a nested value is
adopted; three of its behaviors are rejected, and all three have one cause
-- it formats .NET objects by their properties without normalizing first.
Depth truncates at the third level and loses the value silently. Type names
leak, so a Hashtable renders as System.Collections.Hashtable where an
object renders as @{k=v}. Arrays get reflected over and print Length, Rank,
and SyncRoot. Stage 1 makes all three impossible here. Its automatic
table/list switch at five properties is rejected too: §10 already refuses
TTY adaptation because a pipeline that behaves differently when observed is
unreproducible, and switching on property count is that defect with a
different trigger.

One divergence is deliberate. Format-Table takes its columns from the first
object and blanks every key the later ones introduce, dropping them with no
report. csvHeadersFromMaps unions across every element. A sparse table is a
worse presentation than a dense one; silently dropping columns is a worse
ANSWER.

Tests are a conformance suite over one fixture carrying every property the
rules turn on: json tags that differ from the Go names, a nested object, an
array, an integer past 2^53, a bool, and an empty string. A formatter that
disagrees with the others about a field's name, or about what a nested
value looks like, fails there rather than in a command months later.

BREAKING: table, csv, list, and value now name fields by their json tags
rather than by their Go identifiers, and a nested value renders as compact
JSON rather than through fmt.Sprint.
@David-Noble-at-work David-Noble-at-work self-assigned this Aug 31, 2026
David Noble added 4 commits August 30, 2026 23:05
…ts own numbers (#748, #749)

Two defects found by rendering a real result through every formatter and
reading the bytes, rather than by review.

#748 -- A TABLE CELL CONTAINING A NEWLINE SPLITS ITS ROW, silently. Every
shell result carries a stdout field, and stdout ends in a newline, so this
is the ordinary case rather than an exotic one:

    COMMAND              EXIT  STDERR  STDOUT
    echo "Hello World!"  0             Hello World!
                                                    <- the record ended here

A table pads with spaces and has no quoting, so the cell's newline
terminates the line and tabwriter writes the row terminator onto the next
one. A reader sees one record as two lines. A tab is worse: tabwriter reads
it as a column break and shifts every following cell.

Control characters in a table cell now render as their two-character
escapes. csv is deliberately untouched: RFC 4180 quoting already keeps an
embedded newline inside its field where a parser reads it correctly, and
escaping there would corrupt a value that round-trips today. value is
untouched too -- raw is raw, which is the contract gcloud and aws state for
the same rendering.

#749 -- A --jq EXPRESSION THAT INSPECTS A TYPE PANICS THE PROCESS.

    panic: invalid type: int64 (1)
      gojq.TypeOf(...) type.go:29

gojq accepts nil, bool, int, float64, *big.Int, json.Number, string, []any,
and map[string]any. int64 is not among them, and jqUnwrapNumbers handed it
exactly that from v.Int64(). Any expression reaching TypeOf over an integer
crashed -- tostring, type, tonumber, the arithmetic paths.

The conversion was never needed. gojq handles json.Number natively and
everywhere: compare.go:67, encoder.go:66, and a dozen sites across func.go.
option.go:36 states the contract outright. The function's comment claimed it
kept == stable across integer and float; compare.go:67 puts json.Number in
the same case as int and float64, so that was already true without it.

It was also lossy. Int64() fails past int64 and the fallback is Float64(),
which rounds -- so an integer past 2^53 lost digits inside the filter, the
same defect #712 records against the document codec and the reason
normalize keeps json.Number for the presenters.

Deleted. Two tests asserted int64 and now assert json.Number: they had
pinned the contract that was crashing.

What the panic was blocking is gcloud's `text` format, which needs no format
of ours:

    --jq '[paths(scalars) as $p | "\($p|map(tostring)|join(".")): \(getpath($p))"]' -o value

    results[0].command: echo "Hello World!"
    results[0].exit: 0

That is §8's rule working as written -- a command needing a shape the set
does not have reshapes with --jq and renders with value.
#754)

REGISTERING THE COMMON SET NOW WIRES IT. AddOutputFlags binds --output and
--store to a PersistentPreRunE, so the two flags whose meaning does not
depend on a command rendering anything cannot come apart from their
registration.

  #754 -- writ accepted any --output value. FormatterByName is the only
  place the string is checked, reached from BuildPipeline, and writ reaches
  it once: runVerify. Everything else compared the value to the literal
  "json" and discarded it, so `writ status -o bogus` printed its report and
  exited 0. A typo was honored as a request for the default. Not only
  status -- every writ command but verify.

  #753 -- writ read --store nowhere. outputOptions is consumed in exactly
  two places: config.go:160 takes .Format, commands.go:302 hands the struct
  to BuildPipeline in runVerify. Meanwhile readback.go reads TracesDir()
  and GraphsDir() to fold runs, so `writ status --store <elsewhere>`
  reported on the DEFAULT store as though it had complied. Not the wrong
  shape -- the wrong data.

Both were one defect: a flag registered on a root that no leaf consumes.
Cobra advertised the whole set on every command's help while one command
honored it. A flag that is present and inert is worse than an absent one:
an absent flag errors, and the user tries something else.

The store selection is undone when the command tree finishes. Leaving it
set is harmless in a process that runs one command and exits, and
corrupting in a test binary that runs many -- the root is a package-level
value, so the first command passing --store would silently relocate every
command after it. The existing subcommand test passed "/tmp/store", which
was inert before this and leaked into four sibling tests the moment it
was not.

A NUMBER IS A NUMBER IN BOTH DOCUMENT FORMATS. Stage 1 keeps json.Number so
a presenter renders the digits it was given rather than a float64 rounded
past 2^53. encoding/json special-cases the type and emits it bare; yaml.v3
does not, and json.Number is a string type -- so `-o yaml` answered
unit_count: "2" where `-o json` answered 2. One value, two types, decided
by the rendering. This was a regression from 6bd5328, in this PR.

The conformance suite passed it throughout, because it asserted the digits
were PRESENT and they were: in quotes. A second test now asserts the type,
which is the assertion that would have caught it.

--output DOCUMENTS ITSELF. Each rendering closes with what it is for, and
the free text orders them by usefulness while the list stays alphabetical
-- the order the error message and §7 already used, and the one that helps
in everyday use. Lines are hand-wrapped to 88 so they fit 120 once pflag
adds its 32-column indent: FlagUsages calls FlagUsagesWrapped(0), and 0
means no wrapping, so a long line is the author's to prevent.

devlore-docs learns that a table cell holds neither a newline nor a pipe.

INTEGRATION COVERAGE. A two-operation fixture -- write a file, then remove
it, the removal consuming the write's promise so the graph has one real
edge -- rendered through every value of --output, each verified as the
thing it claims to be: json parses as JSON, csv as CSV, table has one
header and one row, none is empty, template renders. Plus the rejection
case, which is what #754 was.

The point is coverage of the SET. writ answered `-o yaml` with a human
dashboard for months because nothing asserted across the whole of it.
…755)

Help text did not wrap at all. Cobra's template calls FlagUsages, which is
FlagUsagesWrapped(0), and wrap() reads zero as "re-indent after existing
newlines, break nothing":

    func wrap(i, w int, s string) string {
        if w == 0 {
            return strings.Replace(s, "\n", "\n"+strings.Repeat(" ", i), -1)
        }

So every line's width was the author's to maintain by hand -- --output's
usage was wrapped to 88 columns by counting, a constant no test enforced
and one that was wrong on any terminal narrower than 120.

PFLAG'S OWN WRAPPING IS NOT THE FIX. wrapN indents every continuation to the
flag's description column: one indent level, no notion of structure. A
two-column usage collapses the moment it wraps, and the column was what made
it scannable:

    csv            quoted and parseable; when
    a spreadsheet or a data tool reads it

wrapUsageLine hangs continuations under the text they continue, finding that
column as the first run of two or more spaces after the line's first
non-space character. On a flag line that is pflag's description column; on
one of --output's renderings it is the sentence column, one level in:

    csv            quoted and parseable; when a
                   spreadsheet or a data tool reads
                   it

Below helpMinimumTextWidth the hang is abandoned and the line falls back to
its leading indent, because a sliver of text is worse than a lost column.
Widths are counted in runes, so a description carrying a multi-byte
character breaks where it looks like it should.

The width is COLUMNS when set and sane -- a user who exports it has said
what they want, and it is the only answer available when stdout is a pipe --
then the terminal, then a fallback. golang.org/x/term is already a
dependency through pkg/sink.

--output's prose becomes one logical line, joined across source lines, so
the wrapper reflows it rather than honoring a width guessed at authoring
time. The rendering list stays one line each, because those carry the column
the hanging indent exists for.

Cobra is not the obstacle here and this does not fight it: AddTemplateFunc
and SetUsageTemplate are the extension point, and this is what they are for.
What is primitive is pflag's renderer, which we were reaching through them.

Not addressed: a command's Long text, which cobra prints verbatim. It is
hand-authored prose with intentional shape -- examples, lists -- and
reflowing it would damage that.
…#740)

Three rulings and one measurement, written down so none has to be recalled.

§2 -- MEMBERSHIP IS ABOUT INFRASTRUCTURE, NOT ABOUT SHIPPING. An in-scope
program builds its command tree the way every other in-scope program does:
NewRootCmd for the root, AddOutputFlags on that root, cmd/internal/cli for
the exit codes and the store. devlore-test is subject to this whether or not
it is ever shipped, and whether it ships does not enter into it.

The reason is not symmetry. A program that assembles its own root inherits
nothing later. #755's help wrapping reached lore and writ and not
devlore-test, because devlore-test constructs a cobra.Command directly.
Nobody decided to exclude it; a construction choice made elsewhere did, and
measurement found it afterwards. Tracked as #757.

§13 -- TWO VERSION NUMBERS, ONE COMPUTED AND ONE DECLARED. An application
and the documents it writes version separately. The application version is
COMPUTED from the build -- git describe appends commit distance and a hash,
so it differs between two builds of identical source. A document format
version is DECLARED, written down by hand, and changes only when someone
changes it. Stamping a computed version into a document would claim a new
format on every build.

The hedge: the document version is a semver string whose value tracks the
application's for now. A reader seeing "1.0.0" attributes a document to a
release with no lookup table, which is the benefit of one number; and
because the value is a literal rather than a computed one, the two diverge
the day a format changes and the application does not, by editing one
constant. Nothing is decided in advance.

schema_version is a uint32 today, which forecloses that -- an integer cannot
express 1.2.0, so an additive change becomes a major bump or goes
unrecorded. Widening it moves the canonical form and every stored checksum,
so it belongs with the serialization work rather than a CLI branch. Tracked
with the rest of #758.

On the release of a major application version the store version is bumped
with it, so a document is attributable to a release without a lookup table.
Policy, not invariant: the two stay independently versionable, a format
bumps whenever it actually changes, and this is revisited regularly.

What it buys is #758's distinction. A document whose schema_version is
absent or below the floor was written by a RETIRED format; one at the
current version that fails to decode is DAMAGED. A trace carries no version
today, so the two are indistinguishable and both read as corruption.

§15 -- A FIX TO THE SHARED PACKAGE REACHES ONLY ITS CALLERS. Measured after
#753, #754, and #755 landed in cmd/internal/cli:

    COLUMNS=70, longest flag line
      writ           70
      lore           70
      devlore-test  389   <- unwrapped
      star           65

devlore-test builds its own root. star binds its own cmd/star/cli, which is
#743's cost made concrete: a defect fixed in the shared package is fixed
once per package. lore registers the set on inspect rather than its root, so
every other lore command lacks the flags entirely.

The general form: a repair in cmd/internal/cli reaches exactly the programs
that route through cmd/internal/cli. Registration on a root is what makes
one fix program-wide; a second copy of the package is what stops it being
suite-wide.

The plan gains Phase 4b for the three, so the coverage gap is work rather
than an observation.

Not settled here: whether to tag v1.0.0. §13 says there are no released CLI
contracts to preserve, release-path.md says we are not ready to release, and
this branch removes flags outright. Raised, not resolved.
@David-Noble-at-work
David-Noble-at-work merged commit 6a34aac into develop Aug 31, 2026
12 checks passed
@David-Noble-at-work
David-Noble-at-work deleted the feature/740-formatting-rules branch August 31, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment