Skip to content

Add Common Crawl and multi-source archive providers - #9

Merged
aleff-github merged 1 commit into
mainfrom
multi-source-v0.6.0
Sep 20, 2026
Merged

aleff-github merged 1 commit into
mainfrom
multi-source-v0.6.0

Conversation

@aleff-github

Copy link
Copy Markdown
Owner

Summary

Add a provider layer for archive URL discovery while keeping Wayback as the default, backward-compatible source.

Sources

  • --source wayback remains the default
  • --source commoncrawl queries the Common Crawl CDXJ index
  • --source wayback,commoncrawl (or reverse order) queries both and deduplicates normalized URLs across providers in source priority order
  • JSONL provenance uses the provider that discovered a normalized URL first

Common Crawl

  • dynamically resolves latest from collinfo.json
  • supports explicit/repeatable or comma-separated --cc-index crawl IDs
  • walks ZipNum pages using showNumPages + page
  • adds --cc-page-size, --cc-rps, and --cc-filter
  • serializes Common Crawl requests per run and defaults to 1 request/sec
  • uses a descriptive wayparam User-Agent unless explicitly overridden
  • treats Common Crawl 404/no-capture responses as empty results
  • continues to later providers if one archive source fails

Compatibility

  • existing commands still use Wayback only
  • the normal normalization/filter/output pipeline is shared by every provider
  • historical intelligence remains Wayback-only for now; incompatible source combinations are rejected explicitly
  • optional web UI exposes Wayback, Common Crawl, or combined collection

Tests/docs

  • Common Crawl collection discovery, pagination, metadata parsing, 404 behavior and UA tests
  • cross-provider deduplication and partial-provider-failure tests
  • CLI/GUI source configuration tests
  • HTTP empty-status tests
  • README, MkDocs source guide, architecture/API/CLI docs and manpage updates

Common Crawl integration queries only the public archive index; it does not fetch WARC bodies or contact live targets.

@aleff-github
aleff-github merged commit 63a870a into main Sep 20, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant