Skip to content

fix: decode non-ASCII response header values as ISO-8859-1 - #434

Merged
barjin merged 2 commits into
masterfrom
fix/non-ascii-header-values
Apr 13, 2026
Merged

barjin merged 2 commits into
masterfrom
fix/non-ascii-header-values

Conversation

@barjin

@barjin barjin commented Apr 13, 2026

Copy link
Copy Markdown
Member

Fixes #430

HeaderValue::to_str() rejects any non-ASCII bytes, which caused Node bindings to crash and Python bindings to silently return empty strings when servers sent headers like last-modified: Dienstag, 31. März 2026.

Replaced with a per-byte ISO-8859-1 decode (b as char), which maps bytes 0x00–0xFF directly to Unicode codepoints U+0000–U+00FF. This matches the obs-text allowance in RFC 9110 §5.5.

@github-actions github-actions Bot added this to the 138th sprint - Tooling team milestone Apr 13, 2026
@github-actions github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Apr 13, 2026
@barjin
barjin merged commit f67eb04 into master Apr 13, 2026
47 of 48 checks passed
@barjin
barjin deleted the fix/non-ascii-header-values branch April 13, 2026 12:32
Pijukatel pushed a commit that referenced this pull request Jul 1, 2026
What we're solving: response header values were decoded byte-for-byte as
ISO-8859-1 (b as char), which garbled UTF-8 header values such as
Content-Disposition: attachment; filename="naïve.pdf" into mojibake (#479).
That decode was introduced deliberately in #434 to stop non-ASCII header
bytes from crashing the Node bindings / emptying the Python ones (#430), so a
naive switch to UTF-8 would regress those.

How: added a shared decode_header_value helper in the core crate
(re-exported via impit::utils) that decodes the bytes as UTF-8 when they are
valid UTF-8 and otherwise falls back to the byte-preserving ISO-8859-1
decode. Both the Node and Python bindings now call it. This fixes the common
UTF-8 case, keeps #434's genuine ISO-8859-1 values intact, and never emits
U+FFFD replacement characters, so #430's non-crash / non-empty guarantee
holds.

Alternatives considered: the issue's suggested String::from_utf8_lossy was
rejected because it turns invalid-UTF-8 latin-1 bytes (e.g. a lone 0xE4) into
replacement characters, reintroducing the corruption #434 fixed. Exposing raw
header bytes for signature/HMAC callers is left as a separate follow-up.

Note: final review is still in progress; the full workspace build and JS/Py
suites must run in CI as the pinned github.com/apify/h2 git dependency is not
reachable from this environment.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VrUiE5CzcJ9TiRTqvqb1JE
Pijukatel added a commit that referenced this pull request Jul 7, 2026
…le Response.headers (#492)

Fixes #479 (and revisits the decision made in #434).

## Context

Response header values were decoded as ISO-8859-1 (`b as char`,
introduced in #434 to stop
non-ASCII header bytes from crashing Node / emptying Python, #430). That
corrupts the common case
of **UTF-8** header values (e.g. `Content-Disposition:
filename="naïve.pdf"`) into mojibake
(#479). The two positions genuinely conflict: a byte sequence can't be
decoded as both ISO-8859-1
and UTF-8, and the "right" answer differs by ecosystem.

## How

Each binding follows the reference client it emulates:

- **Python** — `Response.headers` is now an **httpx-style `Headers`
object** (adapted from httpx and
kept in Python, like the existing `Cookies` class). It provides
case-insensitive `str` access and
a `.raw` property returning the exact wire bytes as `list[tuple[bytes,
bytes]]`. The `Headers`
object owns decoding — ascii → utf-8 → iso-8859-1, chosen once over the
whole header set, exactly
as httpx does — so UTF-8 header values decode correctly (fixes #479)
while non-UTF-8 values fall
back to ISO-8859-1 (keeps #434/#430). The Rust side just hands `Headers`
the raw wire bytes.
- **JavaScript** — keeps **strict ISO-8859-1** decoding (Fetch
semantics). Because that mapping is a
bijection, the string form stays byte-recoverable: `Buffer.from(value,
'latin1')` reproduces the
exact bytes (and `.toString('utf8')` recovers a UTF-8 value). No
raw-header accessor is added —
it isn't part of the Fetch interface impit implements, and the latin-1
round-trip already covers
  the need.

Header decoding now lives in exactly one place per binding (the Python
`Headers` class; Node's
inline latin-1). The former shared `decode_header_value` core helper is
removed; body-charset
detection reads the (ASCII, per RFC 9110) content-type via
`String::from_utf8_lossy`.

### Net effect on #479
- **Python**: fully fixed — UTF-8 header values decode correctly, and
`response.headers.raw` exposes
  exact bytes for signature/HMAC use.
- **JavaScript**: string values remain ISO-8859-1 **by design** (Fetch
parity), and are
  byte-recoverable via `Buffer.from(value, 'latin1')`.

> Note: `response.headers` on the Python side is a read view — each
access rebuilds it from the
> response, so in-place mutations don't persist (documented on the
accessor).

## Consistency with ecosystem

Each binding matches the reference client it implements.

### Python — matches `httpx` (which impit-python implements)

impit-python advertises the httpx interface ("drop-in replacement for
`httpx.AsyncClient`"). This
PR mirrors httpx's `Headers` directly:

- httpx `Headers.encoding` tries `ascii`, then `utf-8`, then falls back
to `iso-8859-1`:
[`httpx/_models.py` @
v0.28.1](https://github.com/encode/httpx/blob/0.28.1/httpx/_models.py#L125-L145)
— *"Header encoding is mandated as ascii, but we allow fallbacks to
utf-8 or iso-8859-1."*
- httpx exposes `Headers.raw: list[tuple[bytes, bytes]]`:
[same file, `raw`
property](https://github.com/encode/httpx/blob/0.28.1/httpx/_models.py#L152-L156).
impit's `Response.headers` is that same `Headers` type with the same
`.raw`.

### JavaScript — matches the Fetch API / undici (which impit-node
implements)

impit-node is "API-compatible with the Fetch API `Response`". In Fetch,
header values are a
[byte sequence](https://fetch.spec.whatwg.org/#concept-header-value)
exposed to JS as a
[`ByteString`](https://fetch.spec.whatwg.org/#headers-class), i.e. via
[isomorphic decode](https://webidl.spec.whatwg.org/#idl-ByteString) —
each byte `0x00–0xFF` maps to
the code point of equal value (ISO-8859-1). This PR keeps impit-node on
that behavior:

- undici (Node's `fetch`) does the same:
[nodejs/undici#1560](nodejs/undici#1560),
[#1317](nodejs/undici#1317) (Latin-1
`ByteString`s).
- Node's core `http` parser decodes header values as `latin1`/`binary`
  ([nodejs/node#17390](nodejs/node#17390),
[#58240](nodejs/node#58240)); **axios**
inherits this via its Node and
  XHR/Fetch adapters.

Fetch has no raw-header accessor, and ISO-8859-1 is a bijection, so a JS
raw accessor would be
redundant surface — hence none is added.

## Tests

- **Python:** unit tests for the `Headers` class (`.raw` byte-exactness,
encoding selection
ascii/utf-8/iso-8859-1, case-insensitive access, comma-joined
duplicates, construction from
bytes-pairs and str mappings); an integration test over a raw socket
asserting a UTF-8 header
decodes as UTF-8 (`encoding == 'utf-8'`) with exact `.raw` bytes;
`response_test` uses
  `.headers.raw`.
- **JavaScript:** existing latin-1 regression test kept; a new test
asserts a UTF-8 header value
  round-trips via `Buffer.from(value, 'latin1').toString('utf8')`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crashes on non-ASCII header values

2 participants