Skip to content

Add multi-source overlap summary for v0.7.0 - #15

Merged
aleff-github merged 14 commits into
mainfrom
source-summary-v0.7.0
Sep 21, 2026
Merged

aleff-github merged 14 commits into
mainfrom
source-summary-v0.7.0

Conversation

@aleff-github

Copy link
Copy Markdown
Owner

Summary

Add provider-neutral archive coverage intelligence on top of the provenance foundation merged in #14.

New CLI

wayparam -d example.com \
  --source wayback,commoncrawl \
  --source-summary \
  --format jsonl

Summary schema

  • union_urls: normalized URLs seen by at least one selected source
  • source_counts: unique normalized URLs per source
  • exclusive_counts: normalized URLs seen only by one source
  • shared_urls: normalized URLs seen by at least two sources
  • overlap_urls: normalized URLs seen by every selected source
  • complete: false when collection stops early or a source fails

Design

The analysis reuses core.run() in provenance mode. It does not duplicate provider, pagination, filtering, retry or normalization logic. URL membership is stored as compact 128-bit fingerprints.

--max-results counts per-source evidence records before aggregation.

Compatibility

  • normal collection remains unchanged
  • requires at least two selected sources
  • cannot be combined with --provenance
  • no live-target requests

Includes tests and documentation.

Part of #13.

@aleff-github
aleff-github merged commit 7a20f41 into main Sep 21, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant