Skip to content

Add historical CDX intelligence views - #6

Merged
aleff-github merged 1 commit into
mainfrom
historical-intelligence-v0.5.0
Sep 20, 2026
Merged

aleff-github merged 1 commit into
mainfrom
historical-intelligence-v0.5.0

Conversation

@aleff-github

Copy link
Copy Markdown
Owner

Summary

Introduce capture-level historical intelligence while preserving the existing normalized-URL workflow as the default.

New analysis views

  • --history: first/last seen, capture count, status-code and MIME distributions per normalized URL
  • --params: historical prevalence per query-parameter name
  • --summary: one aggregate historical record per domain
  • TXT and JSONL output, with mode-specific filenames
  • matching View selector in the optional local web UI

CDX metadata

  • add a CaptureRecord model
  • request timestamp,statuscode,mimetype,original only for metadata workflows
  • preserve the URL-only contract of iter_original_urls()
  • reuse the existing auto/blocks/resume pagination strategy
  • historical analysis forces collapse=None so capture counts and first/last seen are not distorted by URL-key collapse

Compatibility and limits

  • normal wayparam -d ... behavior is unchanged
  • filters, normalization, date ranges, CDX filters, concurrency, proxy and rate limiting are reused
  • in analysis modes, --max-results is a global cap on accepted capture rows before aggregation

Tests and docs

  • parser and metadata iterator tests
  • historical aggregation and mocked HTTP runner tests
  • analysis file/stdout tests
  • local web UI historical streaming tests
  • README, MkDocs guide/reference, internals, API docs and manpage updates

No active target crawling/scanning is introduced; all data still comes from the Wayback CDX API.

@aleff-github
aleff-github force-pushed the historical-intelligence-v0.5.0 branch from 9c868a6 to 6118a0e Compare September 20, 2026 19:58
@aleff-github
aleff-github merged commit ea350f0 into main Sep 20, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant