fix: decode non-ASCII response header values as ISO-8859-1 - #434
Merged
Merged
Conversation
Pijukatel
pushed a commit
that referenced
this pull request
Jul 1, 2026
What we're solving: response header values were decoded byte-for-byte as ISO-8859-1 (b as char), which garbled UTF-8 header values such as Content-Disposition: attachment; filename="naïve.pdf" into mojibake (#479). That decode was introduced deliberately in #434 to stop non-ASCII header bytes from crashing the Node bindings / emptying the Python ones (#430), so a naive switch to UTF-8 would regress those. How: added a shared decode_header_value helper in the core crate (re-exported via impit::utils) that decodes the bytes as UTF-8 when they are valid UTF-8 and otherwise falls back to the byte-preserving ISO-8859-1 decode. Both the Node and Python bindings now call it. This fixes the common UTF-8 case, keeps #434's genuine ISO-8859-1 values intact, and never emits U+FFFD replacement characters, so #430's non-crash / non-empty guarantee holds. Alternatives considered: the issue's suggested String::from_utf8_lossy was rejected because it turns invalid-UTF-8 latin-1 bytes (e.g. a lone 0xE4) into replacement characters, reintroducing the corruption #434 fixed. Exposing raw header bytes for signature/HMAC callers is left as a separate follow-up. Note: final review is still in progress; the full workspace build and JS/Py suites must run in CI as the pinned github.com/apify/h2 git dependency is not reachable from this environment. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01VrUiE5CzcJ9TiRTqvqb1JE
Pijukatel
added a commit
that referenced
this pull request
Jul 7, 2026
…le Response.headers (#492) Fixes #479 (and revisits the decision made in #434). ## Context Response header values were decoded as ISO-8859-1 (`b as char`, introduced in #434 to stop non-ASCII header bytes from crashing Node / emptying Python, #430). That corrupts the common case of **UTF-8** header values (e.g. `Content-Disposition: filename="naïve.pdf"`) into mojibake (#479). The two positions genuinely conflict: a byte sequence can't be decoded as both ISO-8859-1 and UTF-8, and the "right" answer differs by ecosystem. ## How Each binding follows the reference client it emulates: - **Python** — `Response.headers` is now an **httpx-style `Headers` object** (adapted from httpx and kept in Python, like the existing `Cookies` class). It provides case-insensitive `str` access and a `.raw` property returning the exact wire bytes as `list[tuple[bytes, bytes]]`. The `Headers` object owns decoding — ascii → utf-8 → iso-8859-1, chosen once over the whole header set, exactly as httpx does — so UTF-8 header values decode correctly (fixes #479) while non-UTF-8 values fall back to ISO-8859-1 (keeps #434/#430). The Rust side just hands `Headers` the raw wire bytes. - **JavaScript** — keeps **strict ISO-8859-1** decoding (Fetch semantics). Because that mapping is a bijection, the string form stays byte-recoverable: `Buffer.from(value, 'latin1')` reproduces the exact bytes (and `.toString('utf8')` recovers a UTF-8 value). No raw-header accessor is added — it isn't part of the Fetch interface impit implements, and the latin-1 round-trip already covers the need. Header decoding now lives in exactly one place per binding (the Python `Headers` class; Node's inline latin-1). The former shared `decode_header_value` core helper is removed; body-charset detection reads the (ASCII, per RFC 9110) content-type via `String::from_utf8_lossy`. ### Net effect on #479 - **Python**: fully fixed — UTF-8 header values decode correctly, and `response.headers.raw` exposes exact bytes for signature/HMAC use. - **JavaScript**: string values remain ISO-8859-1 **by design** (Fetch parity), and are byte-recoverable via `Buffer.from(value, 'latin1')`. > Note: `response.headers` on the Python side is a read view — each access rebuilds it from the > response, so in-place mutations don't persist (documented on the accessor). ## Consistency with ecosystem Each binding matches the reference client it implements. ### Python — matches `httpx` (which impit-python implements) impit-python advertises the httpx interface ("drop-in replacement for `httpx.AsyncClient`"). This PR mirrors httpx's `Headers` directly: - httpx `Headers.encoding` tries `ascii`, then `utf-8`, then falls back to `iso-8859-1`: [`httpx/_models.py` @ v0.28.1](https://github.com/encode/httpx/blob/0.28.1/httpx/_models.py#L125-L145) — *"Header encoding is mandated as ascii, but we allow fallbacks to utf-8 or iso-8859-1."* - httpx exposes `Headers.raw: list[tuple[bytes, bytes]]`: [same file, `raw` property](https://github.com/encode/httpx/blob/0.28.1/httpx/_models.py#L152-L156). impit's `Response.headers` is that same `Headers` type with the same `.raw`. ### JavaScript — matches the Fetch API / undici (which impit-node implements) impit-node is "API-compatible with the Fetch API `Response`". In Fetch, header values are a [byte sequence](https://fetch.spec.whatwg.org/#concept-header-value) exposed to JS as a [`ByteString`](https://fetch.spec.whatwg.org/#headers-class), i.e. via [isomorphic decode](https://webidl.spec.whatwg.org/#idl-ByteString) — each byte `0x00–0xFF` maps to the code point of equal value (ISO-8859-1). This PR keeps impit-node on that behavior: - undici (Node's `fetch`) does the same: [nodejs/undici#1560](nodejs/undici#1560), [#1317](nodejs/undici#1317) (Latin-1 `ByteString`s). - Node's core `http` parser decodes header values as `latin1`/`binary` ([nodejs/node#17390](nodejs/node#17390), [#58240](nodejs/node#58240)); **axios** inherits this via its Node and XHR/Fetch adapters. Fetch has no raw-header accessor, and ISO-8859-1 is a bijection, so a JS raw accessor would be redundant surface — hence none is added. ## Tests - **Python:** unit tests for the `Headers` class (`.raw` byte-exactness, encoding selection ascii/utf-8/iso-8859-1, case-insensitive access, comma-joined duplicates, construction from bytes-pairs and str mappings); an integration test over a raw socket asserting a UTF-8 header decodes as UTF-8 (`encoding == 'utf-8'`) with exact `.raw` bytes; `response_test` uses `.headers.raw`. - **JavaScript:** existing latin-1 regression test kept; a new test asserts a UTF-8 header value round-trips via `Buffer.from(value, 'latin1').toString('utf8')`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #430
HeaderValue::to_str()rejects any non-ASCII bytes, which caused Node bindings to crash and Python bindings to silently return empty strings when servers sent headers likelast-modified: Dienstag, 31. März 2026.Replaced with a per-byte ISO-8859-1 decode (
b as char), which maps bytes 0x00–0xFF directly to Unicode codepoints U+0000–U+00FF. This matches theobs-textallowance in RFC 9110 §5.5.