Skip to content

Fix historical URL filtering and Common Crawl stream cleanup - #11

Merged
aleff-github merged 9 commits into
mainfrom
hotfix-v0.6.1
Sep 21, 2026
Merged

aleff-github merged 9 commits into
mainfrom
hotfix-v0.6.1

Conversation

@aleff-github

Copy link
Copy Markdown
Owner

Summary

Hotfixes two issues reproduced by the post-release 0.6.0 functional audit.

Historical Intelligence

Malformed archived URLs such as bracketed non-IP hosts can make urllib.parse.urlsplit raise ValueError. The filter layer now treats those unusable archived URLs as dropped input instead of aborting the whole historical analysis.

Common Crawl lifecycle

When a consumer stops early (for example because --max-results is reached), nested Common Crawl async iterators are now explicitly closed with contextlib.aclosing. This prevents unraisable async-generator shutdown errors such as aclose(): asynchronous generator is already running.

Regression coverage

  • malformed bracketed archived host is safely filtered
  • historical analysis skips malformed captures and continues
  • early Common Crawl provider close cascades to the nested stream

This PR intentionally contains no new features.

@aleff-github
aleff-github merged commit 8dd8e71 into main Sep 21, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant