⚓ WebHarbor

WebHarbor docks popular websites into local, stable, Docker-based mirrors with full auth, database, and multimodal image content. Environments evolve with agent capability.

Live websites are noisy: reCAPTCHA, geo-blocks, network flakiness, content drift. Their most useful features sit behind login walls that benchmarks can't touch. Existing offline web environments either freeze the web into toy synthetic sites or fall back to static traces with no real interaction, which limits large-scale RL training.

WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Code/CodeX) to mirror real sites into local Docker images that:

• Stable & reproducible — no network noise, no content drift, no geo-blocks
• Deep features unlocked — carts, checkouts, accounts, all fully testable
• Evolving — harder tasks drive richer mirrors; the environment grows with agents
• RL-ready — sub-second database resets between rollouts
• Community-driven — 32 sites today, scaling to 100+ together

One command to run all web environments:

docker run -p 8101:8101 -p 40000-40031:40000-40031 battalion7244/webharbor:latest

Then point your agent at http://localhost:40000 through http://localhost:40031 to explore 32 local mirrors of WebVoyager sites: Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, NVIDIA, UC Berkeley, B&H Photo, and AccuWeather.

For sub-second reset between rollouts, expose the control plane and call /reset/<site>:

curl -X POST http://localhost:8101/reset/amazon          # one site
curl -X POST http://localhost:8101/reset-all             # all sites in parallel

If you prefer to build the image yourself:

git clone https://github.com/aiming-lab/WebHarbor && cd WebHarbor
./scripts/fetch_assets.sh                          # pulls static assets from ChilleD/WebHarbor on HF
./scripts/build.sh                                 # docker build -t webharbor:dev .

This checkout registers 32 sites. NVIDIA remains at index 28, UC Berkeley remains at index 29, B&H Photo remains at index 30, and AccuWeather is appended at index 31. Build the image from this checkout to use this registry; publishing source does not update the published Docker image automatically.

websyn_start.sh, control_server.py, the Dockerfile EXPOSE line and every site's tasks.jsonl web URL agree on 32 sites and 40000-40031; scripts/check_site_registry.py (run by scripts/check_assets.sh) fails when they drift.

After preparing the candidate assets and building webharbor:dev, the local review deployment uses:

docker run -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48031:40000-40031 webharbor:dev

NVIDIA inherits the site contribution from @KaKituken (#55) and the verifier/rubric contribution from @DEM1TASSE (#58). This is file-level integration, not a claim that either PR was merged or that the NVIDIA review has passed.

.assets-revision pins the merged dataset commit fa1e8a5b9e8e5d0e42764cd658825f4dea088d8f from HF asset PR #92. Its complete asset tree matches the tested candidate commit f09e586eec8bf1bca0bc0881e08b77f3c2a5508e.

This revision adds bh_photo.tar.gz and preserves all 32 existing archives from c32018ca3b3d67e7b858b1b85fb101aea5090cd7 byte-for-byte, including Berkeley and NVIDIA. It carries 33 archives for 31 registered sites plus the unregistered Bandcamp and Drugs.com archives, which fetch_assets.sh ignores. A clean asset fetch downloaded and extracted all 31 registered sites successfully.

B&H's archive contains images and external cache. The Docker build validates its 508 declared assets and generates instance_seed/bh_photo.db from the tracked catalog. No manually prepared B&H database is required for a fresh build.

The earlier pin b7e605c0ec5fc47de85b09e7427162cc50e38980 is the squash-merge commit of HF dataset PR #85 on the dataset's main. It sits on top of PR #84 and PR #75, which added the first reviewed NVIDIA bundle.

PR #85 replaces five product images and adds three dedicated hero images (see "Image and verifier follow-up" below). Its archive passes validate_asset_archive.py (validated 37 managed members) and a clean-room extract in which all 36 images have distinct SHA-256 values, the seed database is byte-identical to the previous pin (2143c954def96cc921760ab2bea79fe119de3d73212d1b01daf6c61792c2b38d) and every products.image path resolves.

The revisions rejected in earlier rounds are kept here for the record: the older candidate archive from HF PR #38 (2707761e4041a492379ea227f09b3bd9ea838a02, sha256 89e0d0d21000bb94acaeaa329fd28a1264afa05f40834c0f3e3cee5c3a2ae9a1) passes the validator but carries a stale seed (the Jetson descriptions lost the kit/module identity text, the RTX 5060 Ti is named without 16GB and its recommended_psu_watts is 550 while the page's own source note says 600 W), and a revision with no nvidia.tar.gz cannot prepare this candidate at all.

HF PR #84 replaced twelve product images that were byte-identical duplicates of another SKU (RTX 5060/5060 Ti, RTX 5070/5070 Ti, RTX 4070 SUPER/4080 SUPER) or depicted something other than the named product. The follow-up round replaced the remaining mismatched or near-identical assets with official NVIDIA media, so static/images now holds 36 files with 36 distinct SHA-256 values and no within-page image reuse on /, the two series pages or any listing:

• geforce-rtx-5080.png and geforce-rtx-5090.png use NVIDIA's own per-SKU og renders instead of two crops of one mirror-bundle strip;
• geforce-rtx-5070.png, rtx-6000-ada.png and shield-tv-pro.png now carry official renders, which brings every product image into a 1.77–1.90 aspect range;
• static/images/heroes/*.jpg adds three dedicated hero images, so the home hero, the 50-series hero and the 40-series hero no longer reuse a product-card file;
• captions state what each file shows (dgx-b200, h200-tensor-core, and the two family assets that share one vendor artwork).

The same round closed the 20-task audit's findings in the graders and in the site: the T6 phrasing false negative (blocker), the unit-first spec-row phrasing, the bare-price and single-product evidence gates, the driver-series evidence scope, the newsletter topic check, the whole-catalog search scoring, the price-ordered series grid, the leaky sort options, the Email/Search accessible-name collisions, and the scroll-hint and footprint defects. The per-item disposition and evidence are in _wh_review_tools/pr107-audit/agent-{a,b,c}/summary.md, _wh_review_tools/orch/integration/logs/pr107-fixall/{00-issue-list,01-disposition,02-site-reverify,06-images}.md and review-reports/PR-107-FINAL-AUDIT.md.

Several pages render links to nvidia.com, marketplace.nvidia.com and store.nvidia.com as dated source references. They are labelled as leaving the local mirror, no route fetches them (0 external requests over 114 routes at 1440, 768, 390 and 320 px), and the site verifiers treat any navigation outside the mirror's loopback origin as a failure, so a run that follows one fails rather than silently grading against an unreachable page.

The commands below are the ones actually run for the review rounds; raw outputs live under /data/zhaoyang-user-projects/websyn/_wh_review_tools/ (pr107-fixes/ for the phase-1/phase-2 review, pr107-audit/ for the 20-task audit, pr107-fixall/ for the follow-up round).

# site suites (the driver suite is skipped unless its explicit input/output paths are set)
python3 -B sites/nvidia/tests/test_verifiers.py --seed <seed.db> --out <new-dir>   # 316 cases
WH_CONTAINER=<container> TEST_OUT=<outside-source-dir> python3 -B -m unittest discover -s sites/nvidia/tests
#   -> 44 tests when DRIVER_TEST_INPUTS is unset (two classes skipped),
#      73 tests when it is set (29 driver cases included)
DRIVER_TEST_INPUTS=<dir-with-initial/after.db> DRIVER_TEST_OUT=<new-dir> \
  python3 -B -m unittest discover -s sites/nvidia/tests -p 'test_driver_qualifier.py'   # 29 tests
python3 -B sites/nvidia/test_ui_contract.py --output <new-dir>                     # 15 tests
python3 -B sites/nvidia/tests/test_t7_verifier.py                                   # 23 tests (parser regressions)
# verifier CLI contract and the mechanical negative-sample matrix
WH_CONTAINER=<container> TEST_OUT=<dir> python3 -B -m unittest discover -s sites/nvidia/tests -p 'test_verifier_contract.py'   # 11 tests
./scripts/check_assets.sh                                                          # exit 0, 36 inventoried assets

The earlier draft of this section quoted "44 unit tests pass" for the site suites; 44 is the count when the driver regression class is skipped, and that class used to fail its own test_other_information_task_not_relaxed case in the PR head. Both are fixed and the current counts are the ones listed above. The GitHub PR description itself cannot be edited from this repository.

We have built 30 high-quality mirrors covering the WebVoyager benchmark. The next goal is 100+ sites, covering everything in Online-Mind2Web. We are inviting the community to build this together.

There are two ways to join the author list:

Use a coding agent to build a new mirror (frontend + backend + database + tasks). Contributing one website qualifies you for consideration on the final paper's author list.

1. Browse the Contribution Track Sheet and pick an unclaimed site.
2. Submit the Contribution Request Form to claim it. We lock the site to prevent duplicate work.
3. Follow the Website Contribution Guide and CONTRIBUTING.md to build and open a PR.

Review submitted mirrors for visual fidelity, functional correctness, and task grounding. Reviewing 5 environments earns a spot on the author list.

1. Browse open Pull Requests.
2. Check whether the submitted environment supports its proposed tasks, and whether those tasks are meaningful and challenging.
3. Follow the Review Pipeline for systematic verification.

Any other improvement — bug fixes, UI polish, data enrichment, task suggestions, or even feedback, qualifies for the paper's acknowledgement section.

WebHarbor is initiated by UNC-Chapel Hill and Microsoft, with contributions from the broader community. If you have any questions, please contact us via webharborcomm at gmail dot com or zhaoyang at cs dot unc dot edu.

@misc{webharbor2026,
  title        = {WebHarbor: Docking Real Websites for Evolving GUI Agent Environments},
  author       = {{WebHarbor Team and Contributors}},
  year         = {2026},
  url          = {https://aiming-lab.github.io/webharbor.github.io},
  note         = {Project website.}
}