Skip to content

Add Overture Maps building data integration - #886

Merged
louis-e merged 8 commits into
mainfrom
feature/overture-maps-integration
Apr 5, 2026
Merged

louis-e merged 8 commits into
mainfrom
feature/overture-maps-integration

Conversation

@louis-e

@louis-e louis-e commented Apr 5, 2026 •

Copy link
Copy Markdown
Owner

Complement OSM building data with Overture Maps ML-derived footprints to fill coverage gaps in areas with sparse OSM mapping (e.g., rural Africa, parts of Asia).

Key implementation details:

  • STAC collections.parquet catalog for fast partition discovery (1 req instead of 512) with geographic pre-filtering
  • Sparse HTTP Range requests: only download matching Parquet row groups (~4-5 MB each) instead of full partition files (~580 MB each)
  • Source-based dedup: skip buildings whose primary source is OSM
  • Spatial dedup: grid-accelerated centroid-in-bbox check against existing OSM buildings
  • Extract height, min_height, num_floors, roof_shape, roof_material, roof_orientation, facade_color, roof_color, building class/subtype
  • Smart height filtering: omit low Overture heights so buildings use the pipeline's default, keeping visual consistency with OSM buildings

Complement OSM building data with Overture Maps ML-derived footprints
to fill coverage gaps in areas with sparse OSM mapping (e.g., rural
Africa, parts of Asia).

Key implementation details:
- STAC collections.parquet catalog for fast partition discovery (1 req
  instead of 512) with geographic pre-filtering
- Sparse HTTP Range requests: only download matching Parquet row groups
  (~4-5 MB each) instead of full partition files (~580 MB each)
- Source-based dedup: skip buildings whose primary source is OSM
- Spatial dedup: grid-accelerated centroid-in-bbox check against
  existing OSM buildings
- Extract height, min_height, num_floors, roof_shape, roof_material,
  roof_orientation, facade_color, roof_color, building class/subtype
- Smart height filtering: omit low Overture heights so buildings use
  the pipeline's default, keeping visual consistency with OSM buildings
- New --overture CLI flag (default on) with GUI toggle

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Copilot AI review requested due to automatic review settings April 5, 2026 21:12
Resolve conflicts: keep both rotation (from main) and
no_overture (from this branch) in args.rs and gui.rs.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Integrates Overture Maps GeoParquet building footprints as a supplemental data source to improve building coverage where OpenStreetMap is sparse, and merges/deduplicates them into the existing OSM-based generation pipeline.

Changes:

  • Added src/overture.rs implementing STAC partition discovery, sparse HTTP Range downloads of Parquet row groups, OSM-source filtering, and conversion to ProcessedElements.
  • Wired Overture fetching + OSM-dedup into both CLI and GUI generation flows, gated by a new CLI arg (--no-overture).
  • Added new dependencies (parquet, bytes) and updated the lockfile accordingly.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
src/overture.rs New Overture integration module: fetch, parse, filter, dedup, and convert buildings into existing element format.
src/main.rs CLI: fetches Overture buildings (unless disabled) and merges them after OSM parsing.
src/gui.rs GUI: fetches Overture buildings (currently hardcoded enabled) and merges them after OSM parsing.
src/args.rs Adds --no-overture CLI switch to disable supplementation.
Cargo.toml Adds bytes and parquet dependencies required for sparse Parquet reading.
Cargo.lock Locks new transitive dependencies introduced by Parquet/Bytes.
Comments suppressed due to low confidence (1)

src/gui.rs:907

  • GUI generation hardcodes no_overture: false, and there’s no other in-repo reference that lets the UI set/override this option. Since the PR description calls out a GUI toggle, this looks incomplete: please plumb a GUI setting through to Args.no_overture so users can disable Overture supplementation from the GUI.
            // Create generation options
            let generation_options = GenerationOptions {
                path: generation_path.clone(),
                format: world_format,
                level_name,
                spawn_point: mc_spawn_point,
            };

            // Create an Args instance with the chosen bounding box
            // Note: path is used for Java-specific features like spawn point update
            let args: Args = Args {
                bbox,
                file: None,
                save_json_file: None,
                path: Some(if world_format == WorldFormat::JavaAnvil {
                    generation_path
                } else {
                    world_path
                }),
                bedrock: world_format == WorldFormat::BedrockMcWorld,
                downloader: "requests".to_string(),
                scale: world_scale,
                ground_level,
                terrain: terrain_enabled,

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/overture.rs
Comment thread src/overture.rs Outdated
Comment thread src/overture.rs
Comment thread src/args.rs Outdated
Comment thread src/overture.rs
Comment thread src/overture.rs
- Track non-OSM building count for early-stop limit instead of total
  count (prevents premature partition processing halt in OSM-heavy areas)
- Reject rows with missing id field instead of falling back to "unknown"
  (prevents ProcessedWay ID collisions from hash convergence)
- Reject invalid WKB byte order values in parse_wkb_polygon (only 0/1
  are valid per WKB spec; other values indicate corrupt geometry)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/overture.rs Outdated
Comment thread src/overture.rs
Comment thread src/gui.rs
Comment thread src/overture.rs Outdated
- Replace vertex-in-bbox containment check with polygon bbox overlap
  test in building_to_processed_way. The old check dropped buildings
  that straddled the bbox boundary (edges crossing but all vertices
  outside). The new check matches the standard overlap test used by
  parse_overture_row, correctly including buildings whose footprint
  overlaps the target area regardless of vertex positions.
- Fix GUI progress bar regression: Overture fetch emitted progress
  value 4.0 after OSM parsing had already reached 14.0, causing the
  progress indicator to jump backwards. Now emits 14.5, which slots
  correctly between parse completion (14.0) and elevation fetch (14.0).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/overture.rs
Comment thread src/args.rs Outdated
Comment thread src/gui.rs
Comment thread src/overture.rs Outdated
Comment thread src/overture.rs
- Remove --overture/--no-overture CLI flag and conditionals — Overture
  Maps data is now always fetched as a supplemental building source.
  The error boundary in fetch_overture_buildings already degrades
  gracefully to an empty Vec on any failure.
- Clip Overture building polygons to xzbbox via clip_way_to_bbox,
  matching how OSM ways are clipped in osm_parser. Previously Overture
  buildings straddling the bbox boundary could produce out-of-bounds
  XZ coordinates.
- Fix panic in process_partition_file: add length check on the Parquet
  tail bytes returned by fetch_range before slicing. A truncated HTTP
  206 response would have caused an unrecoverable index-out-of-bounds
  panic that bypassed the Result error boundary.
- Remove redundant Bytes allocation in list_partition_files: pass
  response.bytes()? directly to SerializedFileReader instead of
  round-tripping through Vec<u8>.
- Filter out OSM-sourced buildings at collection time rather than at
  the final conversion step, reducing peak memory usage.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 8 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/overture.rs
Comment thread src/overture.rs
Comment thread src/overture.rs
Comment thread src/overture.rs
Comment thread src/overture.rs
Comment thread src/overture.rs
Comment thread src/main.rs
Comment thread src/gui.rs
- Replace DefaultHasher with FNV-1a in gers_id_to_u64. Rust does not
  guarantee DefaultHasher's algorithm across compiler versions, so
  Overture building IDs (and deterministic RNG seeded from them) could
  silently change on toolchain upgrades. FNV-1a is a well-known, stable
  hash with good distribution and zero dependencies.
- Guard fetch_range against length==0: the subtraction
  end = start + length - 1 would underflow to u64::MAX, producing an
  invalid HTTP Range header. Now returns an explicit error.
@louis-e
louis-e requested a review from Copilot April 5, 2026 22:36
@louis-e

louis-e commented Apr 5, 2026

Copy link
Copy Markdown
Owner Author

retrigger-benchmark

@github-actions

github-actions Bot commented Apr 5, 2026

Copy link
Copy Markdown

⏱️ Benchmark run finished in 0m 36s
🧠 Peak memory usage: 1660 MB

📈 Compared against baseline: 30s
🧮 Delta: 6s
🔢 Commit: 7b24d6a

⚠️ This PR worsens generation time.

📅 Last benchmark: 2026-04-05 22:39:18 UTC

You can retrigger the benchmark by commenting retrigger-benchmark.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/overture.rs
Comment thread src/main.rs
Comment thread src/gui.rs
louis-e added 2 commits April 6, 2026 01:13
When the user draws a new rectangle, edits/resizes the existing one,
deletes it, navigates via city search, or types new coordinates in the
bbox input field, the rotation angle is now reset to 0. This prevents
stale rotation values from carrying over to a completely different area.

Rotation reset flows through two funnel points:
- displayBboxInfoText: all map-originated bbox changes (draw, edit,
  delete, search, paste)
- handleBboxInput: manual text input of new coordinates
The local_boost cap controls how many stair layers are placed before
the flat ridge section at the peak. At 0.75, many small-medium houses
had a noticeably wide flat strip at the top, missing just one more
stair layer. Bumping to 0.85 narrows the ridge for most building
sizes while the wall_cap (60% of wall height) still prevents
disproportionately tall roofs. Applied to both gabled and hipped roofs.
@louis-e
louis-e merged commit eb975a5 into main Apr 5, 2026
2 checks passed
@louis-e
louis-e deleted the feature/overture-maps-integration branch April 5, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants