drewda released this 21 Sep 17:01

v0.5.1

47648cc

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified

Learn about vigilant mode.

* Add --keep-compressed to store skadi tiles as .hgt.gz (#47)

Skadi tiles arrive gzipped and were always inflated on write, so what landed
on disk was several times larger than what came over the wire: a full planet
is about 1.6 TB inflated against roughly 350-500 GB compressed. Valhalla
reads .hgt.gz natively, so keeping them packed is an option rather than a
conversion step.

Three pieces, as the issue set out:

- _download stores the gzip stream verbatim instead of inflating it.
- tile_path carries the .gz suffix. Note the URL is built from tile_path
  plus a suffix, so the compressed local name has to drop the suffix the
  uncompressed one adds; otherwise the request becomes .hgt.gz.gz.
- tile_exists accepts either form, so switching the flag on or off does not
  re-download a cache that is already complete. A .gz has no predictable
  size, so the gzip magic bytes stand in for the HGT_SIZE check that guards
  the inflated form -- that still rejects an empty file, a truncated
  download or a stored error body.

--keep-compressed is rejected for --format=geotiff, where it is meaningless,
rather than silently ignored.

Verified against the real bucket: N37W123 stores as 6,581,259 bytes versus
25,934,402 inflated (3.9x), the stored file carries the gzip magic and
inflates to exactly HGT_SIZE, and the request URL is correct in both modes.
Cross-mode caching verified both ways, and a truncated .gz is re-downloaded.

Co-Authored-By: Claude Opus 5 <[email protected]>

* Review fixes: validate --format via argparse; fix a test precedence bug

- --keep-compressed was rejected for any non-skadi format, but the message
  named GeoTIFF, so `--format=bogus --keep-compressed` reported a GeoTIFF
  problem for a format the user never passed, and the hand-rolled
  unknown-format branch became unreachable. --format now declares its
  choices, so argparse rejects an unknown one first; the flag's message
  names the format actually given, and the unreachable branch is removed.
- `assert not list(...) == written` parses as `not (x == y)`, so it held
  whenever the two lists merely differed -- including when both a .hgt and a
  .hgt.gz were written, which is the regression it was meant to catch.
  Replaced with an explicit empty-list check on both the compressed and the
  default path, and confirmed by mutation that writing both forms now fails.

Co-Authored-By: Claude Opus 5 <[email protected]>

* Review fixes: do not decode stored Content-Encoding; detect truncated .gz

The first of these reverts a change I made in this PR on faulty reasoning.

- download(compressed=True) set decode_content=True, on the argument that
  Accept-Encoding: identity means there is nothing to decode. That conflates
  two different things: a transfer encoding the client negotiated, and an
  encoding stored with the object. S3 returns the latter from object
  metadata whatever the request asked for, so a mirror or store that labels
  a .gz object `Content-Encoding: gzip` had its body inflated into the file
  meant to hold it compressed. Verified end to end: the result carries no
  gzip magic, so the tile is re-downloaded on every run forever and Valhalla
  cannot read it. Now covered by a test that serves exactly that header.

- gzip_tile_exists checked only the magic bytes, which survive any
  truncation, so a half-written .hgt.gz was reported complete and, lacking
  the HGT_SIZE guard the inflated form has, would never be repaired. It now
  compares gzip's ISIZE trailer against HGT_SIZE, giving the same guarantee.
  The test that claimed to cover this wrote a single non-gzip byte, which is
  an error body rather than a truncation and was already covered elsewhere;
  it now truncates a real stream.

- tile_exists fell through to the other form whenever the target form was
  not intact, so a truncated .hgt beside a valid .hgt.gz was reported as
  found and silently kept -- a regression of an existing guard. The form the
  run would write is now checked first and must be intact; the other form is
  a fallback only when the target is absent.

- Switching --keep-compressed off cannot produce inflated tiles from a
  compressed cache: the tiles are found, so nothing is written. That was
  presented as a pure win. The run now reports how many tiles were satisfied
  by the other form and that the cache is not converted, and the README says
  so.

Test fixtures now stand in for real tiles: the completeness check compares
ISIZE against HGT_SIZE, so a gzip of a few kilobytes is no longer a valid
stand-in.

Co-Authored-By: Claude Opus 5 <[email protected]>

---------

Co-authored-by: Claude Opus 5 <[email protected]>

drewda released this 20 Sep 15:32

v0.5.0

9924dd2

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified

Learn about vigilant mode.

Modernize packaging, restore test coverage, and support macOS/Linux/W…