Add scaling and serve-loop benchmarks to the CodSpeed suite - #89
Conversation
Extend the benchmark suite merged in #88 with the scenarios needed to track the build-performance work for upstream issue mkdocs/mkdocs#3695: * full builds of the deterministic corpus at several sizes (10/50/200 pages by default, overridable via MKDOCS_BENCH_SIZES for local scaling studies) - a local run at 100/400/1600 pages measured 7.8/9.3/22.1 ms per page, i.e. clearly superlinear growth * a dirty rebuild with no modified files, approximating the 'mkdocs serve --dirty' inner loop * a single page rendered against the largest site's navigation, isolating the O(N^2) sitewide template-rendering term corpus.write_site() gains optional sections/pages_per_section parameters (defaults unchanged, existing benchmarks unaffected).
Merging this PR will degrade performance by 14.6%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | test_search_index_generate |
14.8 ms | 17.3 ms | -14.6% |
| 🆕 | test_dirty_rebuild |
N/A | 869.4 ms | N/A |
| 🆕 | test_full_build_scaling[10] |
N/A | 1.9 s | N/A |
| 🆕 | test_full_build_scaling[200] |
N/A | 29.1 s | N/A |
| 🆕 | test_full_build_scaling[50] |
N/A | 6.8 s | N/A |
| 🆕 | test_template_render_large_nav |
N/A | 58.8 ms | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/mkdocs-issue-4167-15dhse (b5f10dc) with main (bb9d39d)
|
The reported regression on CodSpeed's own report flags "Different runtime environments detected" on this comparison, which matches: this looks like CI-runner measurement noise rather than a real regression. No core code ( @shenxianpeng since acknowledging on CodSpeed requires dashboard access, could you acknowledge this one (or re-run to confirm it clears)? Happy to investigate further if it persists on a re-run. Generated by Claude Code |
1 participant
Follow-up to #88, adding the scenarios from (closed) #87 that the wizard-generated suite doesn't cover — the ones needed to track the build-performance work for upstream mkdocs/mkdocs#3695:
test_full_build_scaling[10|50|200]— full builds of the deterministic corpus at several sizes. Sizes are overridable viaMKDOCS_BENCH_SIZESfor local scaling studies; a local run at 100/400/1600 pages measured 7.8 / 9.3 / 22.1 ms per page — 4× the pages costs ~9.5× the time, the superlinear growth reported in Improve performance and authoring experience ofmkdocs servemkdocs/mkdocs#3695.test_dirty_rebuild— a rebuild with no modified files, approximating themkdocs serve --dirtyinner loop (still re-walks docs, rebuilds nav, recreates the Jinja env and re-runs all plugin events). Local wall time: ~66 ms for a 50-page site.test_template_render_large_nav— a single page rendered against the largest site's navigation, isolating the O(N²) sitewide template-rendering term (built-in themes iterate the full nav for every page).corpus.write_site()gains optionalsections/pages_per_sectionparameters; defaults are unchanged, so the existing 23 benchmarks and their CodSpeed history are unaffected. The full suite is now 28 benchmarks; ruff/isort clean;pytest benchmarkspasses in ~9 s un-instrumented.Related Issue
Groundwork for upstream mkdocs/mkdocs#3695; completes the consolidation of #87 into the #88 structure.
Checklist
MKDOCS_BENCH_SIZESdocs/about/release-notes.mdupdated (if applicable)🤖 Generated with Claude Code
https://claude.ai/code/session_018Hw7x9BxxpuKiLkzjLczp9
Generated by Claude Code