Skip to content

Commit c93c164

Browse files
committed
feat: add markup_fmt coverage-only bench row
1 parent 3b8db7c commit c93c164

15 files changed

Lines changed: 483 additions & 106 deletions

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -436,6 +436,7 @@ BENCH_FILTER=zzz BENCH_LIMIT=10 deno task bench:deno:run
436436
- swc — parse-only TS/JS on both surfaces (its own AST dialect, so oxc-class payload disclosure; `decorators` must be enabled explicitly; goal axis `isModule`)
437437
- postcss — parse-only CSS (the parser behind prettier's CSS printer; the only kind available, since no Rust CSS parser exposes an AST to JS)
438438
- rsvelte's Svelte **parser** via its N-API addon — two `parse/svelte` rows: plain (mechanism-matched to tsv's default parse row; payload Svelte's own sparse-`loc` wire) and `skipExpressionLoc` (named for its option, since that reduction is not tsv's); the first third-party engine on that surface
439+
- markup_fmt (`markup-fmt-wasm`, Svelte only) — dprint's Svelte plugin, composed with the dprint and malva plugins for the embedded script, style and expressions; a **coverage-only** row, since it rejects valid real Svelte and a timed row would remove those files from every row's timed set. See ./docs/benchmarks.md §Coverage-only rows.
439440
- `rsvelte-fmt` (Svelte only) — a **coverage-only** row (accept rate, no timing: no in-process API, and a per-file subprocess row would rank process spawn, not format work); its end-to-end CLI numbers live in the separate hyperfine comparison on tsv.fuz.dev. See ./docs/benchmarks.md §Coverage-only rows.
440441

441442
Results: `benches/js/results/report.<runtime>.{json,md}` (committed; every row carries a `runtime` field) + the combined `report.{json,md}`. Publish to tsv.fuz.dev: `npm run update-benchmarks` in ../tsv.fuz.dev. See ./benches/js/CLAUDE.md.

‎benches/js/CLAUDE.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1103,7 +1103,7 @@ detector, not the raw cv). ⚠ A longer `BENCH_DURATION` is NOT the answer to an
11031103
drifting row's mean keeps moving with n, and at most sample counts the cleaner erases the
11041104
disclosure entirely — re-run the runtime, and read the raw fields.
11051105

1106-
**Null timing is not exclusive to a coverage-only report:** a coverage-only ROW (`rsvelte-fmt`)
1106+
**Null timing is not exclusive to a coverage-only report:** a coverage-only ROW (`rsvelte-fmt`, `markup-fmt-wasm`)
11071107
carries null stats inside an otherwise fully-timed perf report, identifiable by
11081108
`files_iterated: null` — timed on nothing, rather than on the group's intersection. A consumer
11091109
reading `entries[]` as speeds must skip a row with null `ops_per_second`, not treat it as a zero.
@@ -1209,7 +1209,11 @@ not drift — each surface registers its own subset):
12091209
there (`rows_missing_from_payload_tiers`).
12101210

12111211
All three WARN rather than throw: an absent row understates a table, where a stale
1212-
`SURFACE_DISCLOSURES` sentence asserts something false.
1212+
`SURFACE_DISCLOSURES` sentence asserts something false. One registry check at the same site
1213+
THROWS: a coverage-only row with no `COVERAGE_ONLY_REASONS` entry
1214+
(`coverage_only_rows_missing_reason`) — the report would otherwise publish an untimed name it
1215+
cannot explain, and failing at pre-flight costs nothing where failing at report time would throw
1216+
away a finished run.
12131217

12141218
A **fourth** row list in the same module is deliberately unchecked: the curated payload-matched
12151219
lines in `generate_summary_report` (`tsv` vs `oxc-parser`, and the rest). Its
@@ -1587,7 +1591,7 @@ benches/js/
15871591
├── format_config_probe.ts # Behavioral "did the pinned layout config LAND" check —
15881592
│ # one probe source + grading arm PER LANGUAGE, shared by prettier
15891593
│ # (the baseline) and every format impl (biome, oxfmt, dprint,
1590-
│ # malva); unit-tested by format_config_probe_test.ts
1594+
│ # malva, markup_fmt); unit-tested by format_config_probe_test.ts
15911595
├── gate_counts.ts # Pinned gate counts — see ../../docs/gate_counts.md
15921596
├── harvest_stamp.ts # Harvest freshness stamps (checkout ids + pins + view entry lists) + the HARVEST_STAMPS table
15931597
├── implementations.ts # Implementation registry (branches native FFI vs N-API by runtime)
@@ -1616,6 +1620,8 @@ benches/js/
16161620
│ # a binding still emitting `loc` can't be timed as the span-only
16171621
│ # default; unit-tested by locations_probe_test.ts
16181622
├── malva.ts # malva WASM wrapper (CSS only; dprint's CSS plugin, shared formatter host)
1623+
├── markup.ts # markup_fmt WASM wrapper (Svelte only; dprint's Svelte plugin composed with
1624+
│ # the dprint + malva plugins in one host context; COVERAGE-ONLY, never timed)
16191625
├── napi.ts # process.dlopen bindings (NapiImplementation — Node/Bun native)
16201626
├── oxc.ts # OXC native wrappers (oxc-parser + oxfmt)
16211627
├── oxc_wasm.ts # OXC WASM wrapper (oxc-parser via wasm32-wasi; per-runtime entry)

‎benches/js/bench_preflight.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ import {
8989
} from './lib/perf_omit.ts';
9090
import {
9191
type CoverageBySource,
92+
coverage_only_rows_missing_reason,
9293
rows_missing_from_comparisons,
9394
rows_missing_from_display_order,
9495
rows_missing_from_payload_tiers,
@@ -442,6 +443,22 @@ if (untiered_rows.length > 0) {
442443
);
443444
}
444445

446+
// Each coverage-only row's published reason (`report.ts` `COVERAGE_ONLY_REASONS`),
447+
// asked of every operation's cells. FATAL, unlike the three warnings above: the
448+
// report refuses an unexplained untimed name, and asking here fails the run before
449+
// anything is timed instead of after.
450+
const unexplained_rows = coverage_only_rows_missing_reason(
451+
OPERATIONS.flatMap((operation) => get_defined_cells(impls, operation, TASK_OPTIONS))
452+
);
453+
if (unexplained_rows.length > 0) {
454+
throw new Error(
455+
`${unexplained_rows.join(', ')} — coverage-only with no entry in report.ts ` +
456+
`COVERAGE_ONLY_REASONS, so the report could not say why ${
457+
unexplained_rows.length === 1 ? 'it is' : 'they are'
458+
} not timed`
459+
);
460+
}
461+
445462
//
446463
// Pre-flight state
447464
//

‎benches/js/lib/binary_sizes.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ const LABELS = {
7070
yuku_parser_napi: 'yuku-parser (napi)',
7171
yuku_parser_wasm: 'yuku-parser (wasm)',
7272
malva_wasm: 'malva (wasm)',
73+
markup_wasm: 'markup_fmt (wasm)',
7374
rsvelte_fmt_native: 'rsvelte-fmt (binary)',
7475
rsvelte_parse_napi: 'rsvelte compiler (napi)',
7576
swc_napi: 'swc (napi)'
@@ -392,6 +393,19 @@ export async function collect_binary_sizes(
392393
);
393394
}
394395

396+
// markup_fmt — dprint's Svelte (markup) plugin wasm, alone. Its row formats a
397+
// component with the dprint and malva plugins beside it (each sized above), so
398+
// formatting Svelte this way loads all three; this row is the markup layer.
399+
if (impls.markup) {
400+
await push_resolved(
401+
staged,
402+
LABELS.markup_wasm,
403+
'wasm',
404+
[`${node_modules}/dprint-plugin-markup`],
405+
'.wasm'
406+
);
407+
}
408+
395409
// rsvelte's N-API addon — the artifact behind the `rsvelte-parse` rows, and like
396410
// `rsvelte-fmt (binary)` NOT scope-matched to a tsv build: it carries the whole
397411
// compiler plus `svelte2tsx`, HMR diffing and a resolver, where the rows measure

‎benches/js/lib/dprint.ts‎

Lines changed: 26 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@
2222
* `ts,tsx,js,jsx,mjs,cjs,mts,cts` and rejects CSS/Svelte outright (verified), so
2323
* unlike `oxfmt`/`biome` this contributes no css or svelte row. dprint's CSS plugin
2424
* is a separate Wasm plugin with a row of its own over this same formatter host —
25-
* see `lib/malva.ts`; its HTML plugin stays unwired, since it does not format Svelte.
25+
* see `lib/malva.ts` — and its markup plugin backs the Svelte row, composed with this
26+
* plugin and malva — see `lib/markup.ts`.
2627
*/
2728

2829
import { readFile } from 'node:fs/promises';
@@ -35,6 +36,28 @@ import type { DprintVersions } from './versions.ts';
3536
// `init()` (see there) so a load-time crash can't escape the registry's skip.
3637
import type { Formatter } from '@dprint/formatter';
3738

39+
/**
40+
* dprint's GLOBAL config, shared by every dprint-hosted row (`dprint-wasm`,
41+
* `malva-wasm`, `markup-fmt-wasm`): the prettier/tsv layout targets — width 100,
42+
* tabs — so every format row does the same layout work. At dprint's defaults the
43+
* rows would wrap different amounts of code and the ratios would conflate config
44+
* with engine speed. See docs/benchmarks.md §Fairness caveats.
45+
*/
46+
export const DPRINT_GLOBAL_CONFIG = { lineWidth: 100, indentWidth: 2, useTabs: true } as const;
47+
48+
/**
49+
* `@dprint/typescript`'s plugin config — shared with `lib/markup.ts`, whose Svelte
50+
* row formats each embedded `<script>` and template expression through this same
51+
* plugin. `trailingCommas` fans out to the 12 per-construct keys
52+
* (`arguments.trailingCommas`, `arrayExpression.…`, …). `preferSingle` (not
53+
* `alwaysSingle`) is the faithful analogue of prettier's `singleQuote: true`,
54+
* which still switches quotes to avoid escaping.
55+
*/
56+
export const DPRINT_TYPESCRIPT_CONFIG = {
57+
quoteStyle: 'preferSingle',
58+
trailingCommas: 'never'
59+
} as const;
60+
3861
/**
3962
* dprint implementation using the `dprint-plugin-typescript` Wasm plugin.
4063
*
@@ -65,19 +88,8 @@ export class DprintImplementation extends BaseImplementation {
6588
const { createFromBuffer } = await import('@dprint/formatter');
6689
this._formatter = createFromBuffer(await readFile(getPath()));
6790

68-
// Match the prettier/tsv config — tabs, line width 100, single quotes, no
69-
// trailing commas — so every format row does the same layout work (at
70-
// dprint's defaults the rows wrap different amounts of code and the ratios
71-
// conflate config with engine speed). See docs/benchmarks.md §Fairness caveats.
72-
// `lineWidth`/`indentWidth`/`useTabs` are dprint GLOBAL config; the quote and
73-
// trailing-comma keys are plugin config (`trailingCommas` fans out to the 12
74-
// per-construct keys — `arguments.trailingCommas`, `arrayExpression.…`, …).
75-
// `preferSingle` (not `alwaysSingle`) is the faithful analogue of prettier's
76-
// `singleQuote: true`, which still switches quotes to avoid escaping.
77-
this._formatter.setConfig(
78-
{ lineWidth: 100, indentWidth: 2, useTabs: true },
79-
{ quoteStyle: 'preferSingle', trailingCommas: 'never' }
80-
);
91+
// the pinned layout targets — see `DPRINT_GLOBAL_CONFIG` / `DPRINT_TYPESCRIPT_CONFIG`
92+
this._formatter.setConfig(DPRINT_GLOBAL_CONFIG, DPRINT_TYPESCRIPT_CONFIG);
8193

8294
// Assert the config actually LANDED. dprint reports an unrecognized key as
8395
// a diagnostic rather than throwing (verified: a bogus key yields

‎benches/js/lib/format_config_probe.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
/**
22
* The shared "did the pinned layout config actually LAND" probe, run by every
3-
* formatter wrapper: the prettier baseline, biome, oxfmt, dprint and malva.
3+
* formatter wrapper: the prettier baseline, biome, oxfmt, dprint, malva and markup_fmt.
44
*
55
* Every formatter here is pinned to tsv's layout targets — width 100, tabs, single
66
* quotes, no trailing commas — so each row wraps and rewrites the same amount of
77
* code and a ratio measures the engine rather than the config (see
88
* docs/benchmarks.md §Fairness caveats). Two of the four timed format opponents
99
* have a channel that says so when a key stops being recognized, and two do not:
1010
* `lib/dprint.ts` and `lib/malva.ts` fail init on a non-empty
11-
* `getConfigDiagnostics()`, while biome's `applyConfiguration` and oxfmt's
12-
* per-call options bag each accept an unknown key SILENTLY and fall back to their
11+
* `getConfigDiagnostics()` (as does the coverage-only `lib/markup.ts`), while
12+
* biome's `applyConfiguration` and oxfmt's per-call options bag each accept an unknown key SILENTLY and fall back to their
1313
* own defaults — measured: biome drops to width 80 + double quotes + trailing
1414
* commas, oxfmt to spaces + double quotes + trailing commas. A renamed key there
1515
* would leave the row wrapping a different amount of code with nothing anywhere in

‎benches/js/lib/implementations.ts‎

Lines changed: 60 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ import { YukuImplementation } from './yuku.ts';
3333
import { BiomeImplementation } from './biome.ts';
3434
import { DprintImplementation } from './dprint.ts';
3535
import { MalvaImplementation } from './malva.ts';
36+
import { MarkupImplementation } from './markup.ts';
3637
import { PostcssImplementation } from './postcss.ts';
3738
import { RsvelteImplementation } from './rsvelte.ts';
3839
import { RsvelteParseImplementation } from './rsvelte_parse.ts';
@@ -126,6 +127,12 @@ export interface ImplementationSet {
126127
dprint: DprintImplementation | undefined;
127128
/** malva (via WASM; dprint's CSS plugin) - format only; undefined if not available */
128129
malva: MalvaImplementation | undefined;
130+
/**
131+
* markup_fmt (via WASM; dprint's Svelte plugin, composed with dprint's TS and CSS
132+
* plugins) - format only, Svelte; undefined if not available. Coverage-only: it
133+
* rejects valid real Svelte, so it is never timed (see `get_benchmark_tasks`).
134+
*/
135+
markup: MarkupImplementation | undefined;
129136
/** postcss - parse only, CSS; undefined if not available */
130137
postcss: PostcssImplementation | undefined;
131138
/**
@@ -195,16 +202,17 @@ export interface InitOptions {
195202
}
196203

197204
// Deliberately NOT surface-scoped: every impl initializes on every run, including
198-
// the format-only ones (biome, dprint, malva, rsvelte-fmt) on the parse-only
199-
// conformance surface, where they back no row. Gating init on `OPERATIONS` is the
200-
// obvious saving and it isn't worth taking — measured, the four cost ~233 ms total
201-
// (biome 187, dprint 23, malva 9, rsvelte-fmt 14) against a multi-minute run, and
202-
// two disclosures read the LIVE set rather than `complete`: `collect_binary_sizes`
203-
// gates each artifact on `impls.<key>`, so a skipped impl silently drops its row
204-
// from the published BINARY SIZES table (a catalog of what is on disk, which the
205-
// surface's operation list has no business thinning), and `get_alternative_versions`
206-
// feeds the report's `Versions:` line from the same set. The cost of the saving is a
207-
// thinner published report; the cost of not taking it is a fifth of a second.
205+
// the format-only ones (biome, dprint, malva, markup_fmt, rsvelte-fmt) on the
206+
// parse-only conformance surface, where they back no row. Gating init on
207+
// `OPERATIONS` is the obvious saving and it isn't worth taking — measured, the five
208+
// cost under 300 ms total (biome 187, markup_fmt ~50, dprint 23, rsvelte-fmt 14,
209+
// malva 9) against a multi-minute run, and two disclosures read the LIVE set rather
210+
// than `complete`: `collect_binary_sizes` gates each artifact on `impls.<key>`, so
211+
// a skipped impl silently drops its row from the published BINARY SIZES table (a
212+
// catalog of what is on disk, which the surface's operation list has no business
213+
// thinning), and `get_alternative_versions` feeds the report's `Versions:` line
214+
// from the same set. The cost of the saving is a
215+
// thinner published report; the cost of not taking it is under a third of a second.
208216

209217
/**
210218
* Initialize one REQUIRED implementation, rethrowing when it can't load.
@@ -364,6 +372,7 @@ export async function init_implementations(
364372
biome,
365373
dprint,
366374
malva,
375+
markup,
367376
rsvelte,
368377
rsvelte_parse,
369378
swc,
@@ -383,6 +392,12 @@ export async function init_implementations(
383392
const biome_impl = await optional(biome, 'biome', 'Biome (WASM)', 'Biome');
384393
const dprint_impl = await optional(dprint, 'dprint', 'dprint (WASM)', 'dprint');
385394
const malva_impl = await optional(malva, 'malva', 'malva (WASM, CSS)', 'malva');
395+
const markup_impl = await optional(
396+
markup,
397+
'markup',
398+
'markup_fmt (WASM, Svelte, coverage-only)',
399+
'markup_fmt'
400+
);
386401
const rsvelte_impl = await optional(
387402
rsvelte,
388403
'rsvelte',
@@ -428,6 +443,7 @@ export async function init_implementations(
428443
biome: biome_impl,
429444
dprint: dprint_impl,
430445
malva: malva_impl,
446+
markup: markup_impl,
431447
postcss: postcss_impl,
432448
rsvelte: rsvelte_impl,
433449
rsvelte_parse: rsvelte_parse_impl,
@@ -462,6 +478,7 @@ function construct_implementations(versions: AllVersions): ConstructedImplementa
462478
biome: new BiomeImplementation(versions.biome),
463479
dprint: new DprintImplementation(versions.dprint),
464480
malva: new MalvaImplementation(versions.malva),
481+
markup: new MarkupImplementation(versions.markup),
465482
postcss: new PostcssImplementation(versions.postcss),
466483
rsvelte: new RsvelteImplementation(versions.rsvelte),
467484
// A different package from rsvelte-fmt above — the N-API addon, which unlike
@@ -547,9 +564,13 @@ export interface BenchmarkTask {
547564
/** Whether this benchmark runs async */
548565
is_async: boolean;
549566
/**
550-
* Measure this impl's pre-flight coverage but never TIME it. Set for an impl
551-
* with no in-process API, where a timed row would be dominated by process
552-
* spawn rather than format work (`rsvelte-fmt` — see `lib/rsvelte.ts`).
567+
* Measure this impl's pre-flight coverage but never TIME it. Set where a timed
568+
* row would mislead: an impl with no in-process API, whose timed row would be
569+
* dominated by process spawn rather than format work (`rsvelte-fmt` — see
570+
* `lib/rsvelte.ts`), and an impl that rejects valid real code, whose timed row
571+
* would remove every file it rejects from every row's timed set
572+
* (`markup-fmt-wasm` — see `lib/markup.ts`). Each one's reason is published
573+
* beside its accept rate (`report.ts` `COVERAGE_ONLY_REASONS`).
553574
*
554575
* The bench honors this in four places, and all four are load-bearing: the
555576
* pre-flight (`bench_preflight.ts`) plans no timed row for the task, keeps it
@@ -877,7 +898,8 @@ export function get_benchmark_tasks(
877898
// No native peer exists to add here: none of the Rust CSS tools considered
878899
// exposes a parse call to JS (lightningcss hands a tree to its `visitor` only
879900
// mid-transform, biome's js-api exposes no parse, malva is a formatter, oxc has
880-
// no CSS parse binding). See lib/postcss.ts.
901+
// no CSS parse binding, `@swc/css` exposes only minify/transform). See
902+
// lib/postcss.ts.
881903
add(
882904
'postcss',
883905
impls.postcss?.supports_parse_language(language),
@@ -949,6 +971,24 @@ export function get_benchmark_tasks(
949971
impls.malva!.format(source, language)
950972
);
951973

974+
// markup_fmt (Svelte only) — dprint's Svelte plugin, composed with the
975+
// dprint and malva plugins for the embedded code (see lib/markup.ts) —
976+
// COVERAGE-ONLY. It rejects valid Svelte the real corpus holds (markup inside
977+
// a template literal in an attribute expression, `{…}` inside a quoted
978+
// `style`), a large share of the perf corpus's Svelte bytes. A timed row would
979+
// need a `PERF_OMITS` entry per file, and each would leave EVERY row's timed
980+
// set, so adding it would reshape every published Svelte number. Measured for
981+
// what it accepts instead; time it once upstream accepts the corpus. See
982+
// docs/benchmarks.md §Coverage-only rows.
983+
add(
984+
'markup',
985+
impls.markup?.supports_format_language(language),
986+
'markup-fmt-wasm',
987+
'markup',
988+
(source) => impls.markup!.format(source, language),
989+
{ coverage_only: true }
990+
);
991+
952992
// rsvelte-fmt (Svelte only) — COVERAGE-ONLY. It ships no in-process format
953993
// API in any package (the sibling N-API addon is the compiler), so this task
954994
// spawns a process per file: measured on ~5 KB of Svelte the spawn floor
@@ -1012,6 +1052,8 @@ export function get_defined_rows(
10121052
/** One cell this surface defines: a row in one language. */
10131053
export interface DefinedCell extends DefinedRow {
10141054
language: Language;
1055+
/** `BenchmarkTask.coverage_only` — measured for coverage, never timed. */
1056+
coverage_only: boolean;
10151057
}
10161058

10171059
/**
@@ -1028,7 +1070,8 @@ export function get_defined_cells(
10281070
get_benchmark_tasks(impls.complete, operation, language, options).map((task) => ({
10291071
name: task.name,
10301072
impl: task.impl,
1031-
language
1073+
language,
1074+
coverage_only: task.coverage_only === true
10321075
}))
10331076
);
10341077
}
@@ -1085,6 +1128,8 @@ export function get_alternative_versions(
10851128
dprint: impls.dprint?.versions.typescript,
10861129
// Same reasoning as dprint: the CSS plugin, not the shared host.
10871130
malva: impls.malva?.versions.malva,
1131+
// The markup plugin; its embedded TS and CSS are the dprint and malva versions above.
1132+
markup: impls.markup?.versions.markup,
10881133
postcss: impls.postcss?.versions.postcss,
10891134
rsvelte_fmt: impls.rsvelte?.versions.fmt,
10901135
// Two facts, both reported: the addon's own version, and the upstream Svelte

0 commit comments

Comments
 (0)