Repository navigation
Every presentation is a presentation of the JSON: formatting rules, list, and stage 1 - #747
Merged
Merged
Conversation
Writes down the rules that take a Go value to JSON to a presentation, then fixes the four places the code disagreed with them. The rules were written first and judged the code; three of these are defects found that way rather than reported. STAGE 1 IS REAL. normalize moved from inside the jq filter to the head of Pipeline.Emit, so the filter and the formatter see the same data. Before this, `-o list` named a field UnitCount and `--jq . -o list` named the same field unit_count -- one result, two vocabularies, decided by an unrelated flag. The json tag is the name the Starlark surface shows a customer, and it is now the name every rendering shows. Normalization keeps json.Number. Decoding to float64 rounds any integer past 2^53, which is the defect #712 records against the document codec. gojq needs int64 and float64 and gets them inside the jq filter, where the conversion is gojq's requirement and not this pipeline's. A NON-SCALAR CELL IS COMPACT JSON at any depth, never truncated. It went through fmt.Sprint before, so a nested map rendered as Go's `map[runs:3]` -- a notation that names the language rather than the data, and that nothing downstream can read. Types that render themselves are excluded: an error, a fmt.Stringer, an encoding.TextMarshaler. A time.Time is a struct, and encoding it would replace the form it defines for itself. A SINGLE RECORD IS ONE ROW. table refused an object outright and csv rendered the whole record into one cell: two different wrong answers to the shape a command produces whenever it reports on one thing. LIST ADDED. One field per line, keys aligned within a record, each record keeping its own keys. That last point is the reason it exists: the delimited formats union keys across the stream and leave holes, so a heterogeneous stream renders mostly holes. gcloud ships list and flattened separately; one suffices here because the compact-JSON cell rule already handles the nesting flattened exists to spread out. The spec records the survey behind these, measured against pwsh 7.5.4 rather than recalled. PowerShell's inline rendering of a nested value is adopted; three of its behaviors are rejected, and all three have one cause -- it formats .NET objects by their properties without normalizing first. Depth truncates at the third level and loses the value silently. Type names leak, so a Hashtable renders as System.Collections.Hashtable where an object renders as @{k=v}. Arrays get reflected over and print Length, Rank, and SyncRoot. Stage 1 makes all three impossible here. Its automatic table/list switch at five properties is rejected too: §10 already refuses TTY adaptation because a pipeline that behaves differently when observed is unreproducible, and switching on property count is that defect with a different trigger. One divergence is deliberate. Format-Table takes its columns from the first object and blanks every key the later ones introduce, dropping them with no report. csvHeadersFromMaps unions across every element. A sparse table is a worse presentation than a dense one; silently dropping columns is a worse ANSWER. Tests are a conformance suite over one fixture carrying every property the rules turn on: json tags that differ from the Go names, a nested object, an array, an integer past 2^53, a bool, and an empty string. A formatter that disagrees with the others about a field's name, or about what a nested value looks like, fails there rather than in a command months later. BREAKING: table, csv, list, and value now name fields by their json tags rather than by their Go identifiers, and a nested value renders as compact JSON rather than through fmt.Sprint.
added 4 commits
August 30, 2026 23:05
…ts own numbers (#748, #749) Two defects found by rendering a real result through every formatter and reading the bytes, rather than by review. #748 -- A TABLE CELL CONTAINING A NEWLINE SPLITS ITS ROW, silently. Every shell result carries a stdout field, and stdout ends in a newline, so this is the ordinary case rather than an exotic one: COMMAND EXIT STDERR STDOUT echo "Hello World!" 0 Hello World! <- the record ended here A table pads with spaces and has no quoting, so the cell's newline terminates the line and tabwriter writes the row terminator onto the next one. A reader sees one record as two lines. A tab is worse: tabwriter reads it as a column break and shifts every following cell. Control characters in a table cell now render as their two-character escapes. csv is deliberately untouched: RFC 4180 quoting already keeps an embedded newline inside its field where a parser reads it correctly, and escaping there would corrupt a value that round-trips today. value is untouched too -- raw is raw, which is the contract gcloud and aws state for the same rendering. #749 -- A --jq EXPRESSION THAT INSPECTS A TYPE PANICS THE PROCESS. panic: invalid type: int64 (1) gojq.TypeOf(...) type.go:29 gojq accepts nil, bool, int, float64, *big.Int, json.Number, string, []any, and map[string]any. int64 is not among them, and jqUnwrapNumbers handed it exactly that from v.Int64(). Any expression reaching TypeOf over an integer crashed -- tostring, type, tonumber, the arithmetic paths. The conversion was never needed. gojq handles json.Number natively and everywhere: compare.go:67, encoder.go:66, and a dozen sites across func.go. option.go:36 states the contract outright. The function's comment claimed it kept == stable across integer and float; compare.go:67 puts json.Number in the same case as int and float64, so that was already true without it. It was also lossy. Int64() fails past int64 and the fallback is Float64(), which rounds -- so an integer past 2^53 lost digits inside the filter, the same defect #712 records against the document codec and the reason normalize keeps json.Number for the presenters. Deleted. Two tests asserted int64 and now assert json.Number: they had pinned the contract that was crashing. What the panic was blocking is gcloud's `text` format, which needs no format of ours: --jq '[paths(scalars) as $p | "\($p|map(tostring)|join(".")): \(getpath($p))"]' -o value results[0].command: echo "Hello World!" results[0].exit: 0 That is §8's rule working as written -- a command needing a shape the set does not have reshapes with --jq and renders with value.
#754) REGISTERING THE COMMON SET NOW WIRES IT. AddOutputFlags binds --output and --store to a PersistentPreRunE, so the two flags whose meaning does not depend on a command rendering anything cannot come apart from their registration. #754 -- writ accepted any --output value. FormatterByName is the only place the string is checked, reached from BuildPipeline, and writ reaches it once: runVerify. Everything else compared the value to the literal "json" and discarded it, so `writ status -o bogus` printed its report and exited 0. A typo was honored as a request for the default. Not only status -- every writ command but verify. #753 -- writ read --store nowhere. outputOptions is consumed in exactly two places: config.go:160 takes .Format, commands.go:302 hands the struct to BuildPipeline in runVerify. Meanwhile readback.go reads TracesDir() and GraphsDir() to fold runs, so `writ status --store <elsewhere>` reported on the DEFAULT store as though it had complied. Not the wrong shape -- the wrong data. Both were one defect: a flag registered on a root that no leaf consumes. Cobra advertised the whole set on every command's help while one command honored it. A flag that is present and inert is worse than an absent one: an absent flag errors, and the user tries something else. The store selection is undone when the command tree finishes. Leaving it set is harmless in a process that runs one command and exits, and corrupting in a test binary that runs many -- the root is a package-level value, so the first command passing --store would silently relocate every command after it. The existing subcommand test passed "/tmp/store", which was inert before this and leaked into four sibling tests the moment it was not. A NUMBER IS A NUMBER IN BOTH DOCUMENT FORMATS. Stage 1 keeps json.Number so a presenter renders the digits it was given rather than a float64 rounded past 2^53. encoding/json special-cases the type and emits it bare; yaml.v3 does not, and json.Number is a string type -- so `-o yaml` answered unit_count: "2" where `-o json` answered 2. One value, two types, decided by the rendering. This was a regression from 6bd5328, in this PR. The conformance suite passed it throughout, because it asserted the digits were PRESENT and they were: in quotes. A second test now asserts the type, which is the assertion that would have caught it. --output DOCUMENTS ITSELF. Each rendering closes with what it is for, and the free text orders them by usefulness while the list stays alphabetical -- the order the error message and §7 already used, and the one that helps in everyday use. Lines are hand-wrapped to 88 so they fit 120 once pflag adds its 32-column indent: FlagUsages calls FlagUsagesWrapped(0), and 0 means no wrapping, so a long line is the author's to prevent. devlore-docs learns that a table cell holds neither a newline nor a pipe. INTEGRATION COVERAGE. A two-operation fixture -- write a file, then remove it, the removal consuming the write's promise so the graph has one real edge -- rendered through every value of --output, each verified as the thing it claims to be: json parses as JSON, csv as CSV, table has one header and one row, none is empty, template renders. Plus the rejection case, which is what #754 was. The point is coverage of the SET. writ answered `-o yaml` with a human dashboard for months because nothing asserted across the whole of it.
…755) Help text did not wrap at all. Cobra's template calls FlagUsages, which is FlagUsagesWrapped(0), and wrap() reads zero as "re-indent after existing newlines, break nothing": func wrap(i, w int, s string) string { if w == 0 { return strings.Replace(s, "\n", "\n"+strings.Repeat(" ", i), -1) } So every line's width was the author's to maintain by hand -- --output's usage was wrapped to 88 columns by counting, a constant no test enforced and one that was wrong on any terminal narrower than 120. PFLAG'S OWN WRAPPING IS NOT THE FIX. wrapN indents every continuation to the flag's description column: one indent level, no notion of structure. A two-column usage collapses the moment it wraps, and the column was what made it scannable: csv quoted and parseable; when a spreadsheet or a data tool reads it wrapUsageLine hangs continuations under the text they continue, finding that column as the first run of two or more spaces after the line's first non-space character. On a flag line that is pflag's description column; on one of --output's renderings it is the sentence column, one level in: csv quoted and parseable; when a spreadsheet or a data tool reads it Below helpMinimumTextWidth the hang is abandoned and the line falls back to its leading indent, because a sliver of text is worse than a lost column. Widths are counted in runes, so a description carrying a multi-byte character breaks where it looks like it should. The width is COLUMNS when set and sane -- a user who exports it has said what they want, and it is the only answer available when stdout is a pipe -- then the terminal, then a fallback. golang.org/x/term is already a dependency through pkg/sink. --output's prose becomes one logical line, joined across source lines, so the wrapper reflows it rather than honoring a width guessed at authoring time. The rendering list stays one line each, because those carry the column the hanging indent exists for. Cobra is not the obstacle here and this does not fight it: AddTemplateFunc and SetUsageTemplate are the extension point, and this is what they are for. What is primitive is pflag's renderer, which we were reaching through them. Not addressed: a command's Long text, which cobra prints verbatim. It is hand-authored prose with intentional shape -- examples, lists -- and reflowing it would damage that.
…#740) Three rulings and one measurement, written down so none has to be recalled. §2 -- MEMBERSHIP IS ABOUT INFRASTRUCTURE, NOT ABOUT SHIPPING. An in-scope program builds its command tree the way every other in-scope program does: NewRootCmd for the root, AddOutputFlags on that root, cmd/internal/cli for the exit codes and the store. devlore-test is subject to this whether or not it is ever shipped, and whether it ships does not enter into it. The reason is not symmetry. A program that assembles its own root inherits nothing later. #755's help wrapping reached lore and writ and not devlore-test, because devlore-test constructs a cobra.Command directly. Nobody decided to exclude it; a construction choice made elsewhere did, and measurement found it afterwards. Tracked as #757. §13 -- TWO VERSION NUMBERS, ONE COMPUTED AND ONE DECLARED. An application and the documents it writes version separately. The application version is COMPUTED from the build -- git describe appends commit distance and a hash, so it differs between two builds of identical source. A document format version is DECLARED, written down by hand, and changes only when someone changes it. Stamping a computed version into a document would claim a new format on every build. The hedge: the document version is a semver string whose value tracks the application's for now. A reader seeing "1.0.0" attributes a document to a release with no lookup table, which is the benefit of one number; and because the value is a literal rather than a computed one, the two diverge the day a format changes and the application does not, by editing one constant. Nothing is decided in advance. schema_version is a uint32 today, which forecloses that -- an integer cannot express 1.2.0, so an additive change becomes a major bump or goes unrecorded. Widening it moves the canonical form and every stored checksum, so it belongs with the serialization work rather than a CLI branch. Tracked with the rest of #758. On the release of a major application version the store version is bumped with it, so a document is attributable to a release without a lookup table. Policy, not invariant: the two stay independently versionable, a format bumps whenever it actually changes, and this is revisited regularly. What it buys is #758's distinction. A document whose schema_version is absent or below the floor was written by a RETIRED format; one at the current version that fails to decode is DAMAGED. A trace carries no version today, so the two are indistinguishable and both read as corruption. §15 -- A FIX TO THE SHARED PACKAGE REACHES ONLY ITS CALLERS. Measured after #753, #754, and #755 landed in cmd/internal/cli: COLUMNS=70, longest flag line writ 70 lore 70 devlore-test 389 <- unwrapped star 65 devlore-test builds its own root. star binds its own cmd/star/cli, which is #743's cost made concrete: a defect fixed in the shared package is fixed once per package. lore registers the set on inspect rather than its root, so every other lore command lacks the flags entirely. The general form: a repair in cmd/internal/cli reaches exactly the programs that route through cmd/internal/cli. Registration on a root is what makes one fix program-wide; a second copy of the package is what stops it being suite-wide. The plan gains Phase 4b for the three, so the coverage gap is work rather than an observation. Not settled here: whether to tag v1.0.0. §13 says there are no released CLI contracts to preserve, release-path.md says we are not ready to release, and this branch removes flags outright. Raised, not resolved.
6 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writes down the rules that take a Go value to JSON to a presentation, then fixes the four places the code
disagreed with them. The rules were written first and judged the code; three of the four are defects found
that way rather than reported.
Advances #740. Spec:
docs/architecture/10-command-line-interface.md§8.Closes #748. Closes #749. Closes #750. Closes #751. Closes #752. Closes #753. Closes #754. Closes #755.
Eight defects, all found by writing the rules first and then rendering a real result through every formatter
and reading the bytes -- not by review.
map[runs:3]--jqcallingtostringpanicked the process--storewas registered and read nowhere in writ--outputwas accepted unvalidated on every writ command butverify#753 and #754 are one defect wearing two faces: a flag registered on a root that no leaf consumes.
AddOutputFlagsnow wires the two flags whose meaning does not depend on a command rendering anything, soregistering the set and honoring it cannot come apart.
One regression was introduced and fixed inside this PR: keeping
json.Numberthrough stage 1 made-o yamlanswer
unit_count: "2"where-o jsonanswered2, becauseencoding/jsonspecial-cases the type andyaml.v3sees a string. The conformance suite passed it throughout -- it asserted the digits were present,and they were, in quotes. A second test now asserts the type.
Stage 1 is real
normalizemoved from inside the jq filter to the head ofPipeline.Emit, so the filter and the formattersee the same data.
One result, two vocabularies, decided by an unrelated flag. The
json:tag is the name the Starlark surfaceshows a customer, and it is now the name every rendering shows.
Numbers stay
json.Number. Decoding tofloat64rounds any integer past 2^53 — the defect #712 recordsagainst the document codec.
gojqneeds int64 and float64 and gets them inside the jq filter, where theconversion is gojq's requirement and not this pipeline's.
The defects the rules exposed
UnitCount, unless--jqwas passedunit_count, alwaysmap[runs:3]— Go's notation, unparseable{"runs":3}-o tableexpected slice or array, got map[string]interface {}-o csv-o table\n, one record per line--jqcallingtostringon a numberpanic: invalid type: int64 (1)#748 — every shell result carries a
stdoutfield, andstdoutends in a newline, so a split row wasthe ordinary case.
csvis deliberately untouched: RFC 4180 quoting already keeps the newline inside itsfield where a parser reads it, and escaping there would corrupt a value that round-trips today.
#749 — gojq accepts
nil,bool,int,float64,*big.Int,json.Number,string,[]any, andmap[string]any.int64is not among them, andjqUnwrapNumbershanded it exactly that fromv.Int64().The conversion was never needed — gojq handles
json.Numbernatively acrosscompare.go,encoder.go, anda dozen sites in
func.go— and it was lossy, sinceInt64()fails past int64 and the fallback rounds.Deleted.
Types that render themselves are excluded from the compact-JSON rule: an
error, afmt.Stringer, anencoding.TextMarshaler. Atime.Timeis a struct, and encoding it would replace the form it defines foritself.
listOne field per line, keys aligned within a record, each record keeping its own keys. That last point is
why it exists: the delimited formats union keys across the stream and leave holes, so a heterogeneous stream
renders mostly holes.
gcloudshipslistandflattenedseparately; one suffices here because the compact-JSON cell rulealready handles the nesting
flattenedexists to spread out.The survey behind the rules
Measured against pwsh 7.5.4, not recalled. PowerShell's inline rendering of a nested value is adopted. Three
of its behaviors are rejected, and all three have one cause — it formats .NET objects by their properties
without normalizing first:
@{c=}, the value silently goneSystem.Collections.Hashtablewhere an object gives@{k=v}Length LongLength Rank SyncRoot ...Stage 1 makes all three impossible here.
Its automatic table/list switch at five properties is rejected too: §10 already refuses TTY adaptation
because a pipeline that behaves differently when observed is unreproducible, and switching on property count
is that defect with a different trigger.
One divergence is deliberate.
Format-Tabletakes its columns from the first object and blanks every keythe later ones introduce, dropping them with no report.
csvHeadersFromMapsunions across every element. Asparse table is a worse presentation than a dense one; silently dropping columns is a worse answer.
Tests
A conformance suite over one fixture carrying every property the rules turn on:
json:tags that differ fromthe Go names, a nested object, an array, an integer past 2^53, a bool, and an empty string. Every formatter
runs against it through a real pipeline, which is the only path that applies stage 1. A formatter that
disagrees with the others about a field's name, or about what a nested value looks like, fails there rather
than in a command months later.
What this does NOT fix: the suite
Each issue above describes
writ, andwritis fixed. The class survives elsewhere, because a repair incmd/internal/clireaches exactly the programs that route throughcmd/internal/cli. Measured after thesecommits:
--outputvalidated,--storeresolved (#753, #754)writloreinspectonlydevlore-teststardevlore-testbuilds its root directly instead of throughcli.NewRootCmd, so it getsAddOutputFlagsbut not the help wrapping.
starbinds its owncmd/star/cli.AddOutputFlags. A fix to the shared package cannot reach a second copyof it — the concrete cost of cmd/star/cli duplicates cmd/internal/cli, and the two now disagree on the flag surface #743, and the argument for deleting the duplicate rather than maintaining it.
loreregisters the set oninspectrather than its root, so every other lore command lacks the flagsentirely — the epic's "1 of 46 commands", still true.
Tracked by #743 and by tasks 35–37 of the plan. Recorded in §15 so a reviewer does not read these fixes as
suite-wide, and so the next person fixing something in
cmd/internal/cliknows what it will and will notreach.
Verification
make checkpasses.make docsregenerates clean. Every formatter, against one real result:What #749 was blocking — gcloud's
textformat, needing no format of ours:That is §8's rule working as written: a command needing a shape the set does not have reshapes with
--jqand renders with
value.Breaking
table,csv,list, andvaluenow name fields by theirjson:tags rather than by their Goidentifiers, and a nested value renders as compact JSON rather than through
fmt.Sprint.docs/cliis gitignored here, so no reference pages appear in this diff — butdocs-publish.yamlregenerates it on every push to
developand auto-merges the result. The--outputhelp text changes golive when this merges, with no human in the path.