fix(fetch): ByteString checks & conversion in Headers - #1560
Merged
Merged
Conversation
ronag
approved these changes
Jul 18, 2022
Codecov Report
@@ Coverage Diff @@
## main #1560 +/- ##
=======================================
Coverage 94.91% 94.91%
=======================================
Files 50 50
Lines 4702 4702
=======================================
Hits 4463 4463
Misses 239 239
Continue to review full report at Codecov.
|
ken-matsui
pushed a commit
to ken-matsui/cabin-website
that referenced
this pull request
Aug 19, 2022
Bumps undici from 5.8.0 to 5.9.1. Release notes Sourced from undici's releases. v5.9.1 What's Changed fix: don't timeout while waiting for client to send request (#1604) Fix array headers by @mateonunez in nodejs/undici#1598 fix(fetch): implement fully read body algorithm by @KhafraDev in nodejs/undici#1597 fix: add support for integrity option to Fetch by @jelmervdl in nodejs/undici#1596 fix(File): respect typed array byteOffset and byteLength by @mrbbot in nodejs/undici#1601 New Contributors @mateonunez made their first contribution in nodejs/undici#1598 @jelmervdl made their first contribution in nodejs/undici#1596 @mrbbot made their first contribution in nodejs/undici#1601 Full Changelog: nodejs/[email protected] v5.8.2⚠️ Security Release⚠️ CRLF Injection in Nodejs ‘undici’ via Content-Type GHSA-f772-66g8-q5h3 CVE-2022-35948 undici.request vulnerable to SSRF using absolute URL on pathname GHSA-8qr4-xgw6-wmr3 CVE-2022-35949 What's Changed docs: mock different endpoints in a single file by @ritvik130 in nodejs/undici#1589 feat(webidl): better error message for ByteString converter by @KhafraDev in nodejs/undici#1591 New Contributors @ritvik130 made their first contribution in nodejs/undici#1589 Full Changelog: nodejs/[email protected] v5.8.1 What's Changed Do not decode the body while we are following a redirect by @mcollina in nodejs/undici#1554 docs: Fix spelling/grammar in "Mocking Request" by @meyfa in nodejs/undici#1555 fix(MockInterceptor): callback options.headers w/ fetch by @KhafraDev in nodejs/undici#1559 fix(fetch): ByteString checks & conversion in Headers by @KhafraDev in nodejs/undici#1560 test: update client certificates with ones that expires in 100 years by @jodevsa in nodejs/undici#1566 fix: x-www-form-urlencoded parser keep the BOM by @cola119 in nodejs/undici#1563 fix: prioritise error events over timeouts by @jodevsa in nodejs/undici#1551 fix: add isErrorLike by @KhafraDev in nodejs/undici#1570 fix(types): add missing pool stats by @SkeLLLa in nodejs/undici#1573 fix: fetch a long base64 url will crash and nothing happens (close: #1574) by @ahaoboy in nodejs/undici#1575 fix: follow signal.reason in Request by @LiviaMedeiros in nodejs/undici#1580 docs: Fix DiagnosticsChannel sidebar link by @trentm in nodejs/undici#1582 fix: make mock headers case-insensitive by @cola119 in nodejs/undici#1585 New Contributors @meyfa made their first contribution in nodejs/undici#1555 @cola119 made their first contribution in nodejs/undici#1563 @SkeLLLa made their first contribution in nodejs/undici#1573 ... (truncated) Commits 5890e16 5.9.1 ecae314 fix: don't timeout while waiting for client to send request (#1604) fa9fd90 fix(File): respect typed array byteOffset and byteLength (#1601) ae6f554 fix: add support for integrity option to Fetch (#1596) deed628 fix(fetch): implement fully read body algorithm (#1597) 0d1419c Fix array headers (#1598) 52d1ce5 Bumped v5.8.2 66165d6 Merge pull request from GHSA-f772-66g8-q5h3 124f7eb Merge pull request from GHSA-8qr4-xgw6-wmr3 aef314c feat(webidl): better error message for ByteString converter (#1591) Additional commits viewable in compare view Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase. Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: @dependabot rebase will rebase this PR @dependabot recreate will recreate this PR, overwriting any edits that have been made to it @dependabot merge will merge this PR after your CI passes on it @dependabot squash and merge will squash and merge this PR after your CI passes on it @dependabot cancel merge will cancel a previously requested merge and block automerging @dependabot reopen will reopen this PR if it is closed @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
metcoder95
pushed a commit
to metcoder95/undici
that referenced
this pull request
Dec 26, 2022
crysmags
pushed a commit
to crysmags/undici
that referenced
this pull request
Feb 27, 2024
ken-matsui
pushed a commit
to cabinpkg/cabin
that referenced
this pull request
May 30, 2026
Bumps undici from 5.8.0 to 5.9.1. Release notes Sourced from undici's releases. v5.9.1 What's Changed fix: don't timeout while waiting for client to send request (#1604) Fix array headers by @mateonunez in nodejs/undici#1598 fix(fetch): implement fully read body algorithm by @KhafraDev in nodejs/undici#1597 fix: add support for integrity option to Fetch by @jelmervdl in nodejs/undici#1596 fix(File): respect typed array byteOffset and byteLength by @mrbbot in nodejs/undici#1601 New Contributors @mateonunez made their first contribution in nodejs/undici#1598 @jelmervdl made their first contribution in nodejs/undici#1596 @mrbbot made their first contribution in nodejs/undici#1601 Full Changelog: nodejs/[email protected] v5.8.2⚠️ Security Release⚠️ CRLF Injection in Nodejs ‘undici’ via Content-Type GHSA-f772-66g8-q5h3 CVE-2022-35948 undici.request vulnerable to SSRF using absolute URL on pathname GHSA-8qr4-xgw6-wmr3 CVE-2022-35949 What's Changed docs: mock different endpoints in a single file by @ritvik130 in nodejs/undici#1589 feat(webidl): better error message for ByteString converter by @KhafraDev in nodejs/undici#1591 New Contributors @ritvik130 made their first contribution in nodejs/undici#1589 Full Changelog: nodejs/[email protected] v5.8.1 What's Changed Do not decode the body while we are following a redirect by @mcollina in nodejs/undici#1554 docs: Fix spelling/grammar in "Mocking Request" by @meyfa in nodejs/undici#1555 fix(MockInterceptor): callback options.headers w/ fetch by @KhafraDev in nodejs/undici#1559 fix(fetch): ByteString checks & conversion in Headers by @KhafraDev in nodejs/undici#1560 test: update client certificates with ones that expires in 100 years by @jodevsa in nodejs/undici#1566 fix: x-www-form-urlencoded parser keep the BOM by @cola119 in nodejs/undici#1563 fix: prioritise error events over timeouts by @jodevsa in nodejs/undici#1551 fix: add isErrorLike by @KhafraDev in nodejs/undici#1570 fix(types): add missing pool stats by @SkeLLLa in nodejs/undici#1573 fix: fetch a long base64 url will crash and nothing happens (close: #1574) by @ahaoboy in nodejs/undici#1575 fix: follow signal.reason in Request by @LiviaMedeiros in nodejs/undici#1580 docs: Fix DiagnosticsChannel sidebar link by @trentm in nodejs/undici#1582 fix: make mock headers case-insensitive by @cola119 in nodejs/undici#1585 New Contributors @meyfa made their first contribution in nodejs/undici#1555 @cola119 made their first contribution in nodejs/undici#1563 @SkeLLLa made their first contribution in nodejs/undici#1573 ... (truncated) Commits 5890e16 5.9.1 ecae314 fix: don't timeout while waiting for client to send request (#1604) fa9fd90 fix(File): respect typed array byteOffset and byteLength (#1601) ae6f554 fix: add support for integrity option to Fetch (#1596) deed628 fix(fetch): implement fully read body algorithm (#1597) 0d1419c Fix array headers (#1598) 52d1ce5 Bumped v5.8.2 66165d6 Merge pull request from GHSA-f772-66g8-q5h3 124f7eb Merge pull request from GHSA-8qr4-xgw6-wmr3 aef314c feat(webidl): better error message for ByteString converter (#1591) Additional commits viewable in compare view Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase. Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: @dependabot rebase will rebase this PR @dependabot recreate will recreate this PR, overwriting any edits that have been made to it @dependabot merge will merge this PR after your CI passes on it @dependabot squash and merge will squash and merge this PR after your CI passes on it @dependabot cancel merge will cancel a previously requested merge and block automerging @dependabot reopen will reopen this PR if it is closed @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Pijukatel
added a commit
to apify/impit
that referenced
this pull request
Jul 7, 2026
…le Response.headers (#492) Fixes #479 (and revisits the decision made in #434). ## Context Response header values were decoded as ISO-8859-1 (`b as char`, introduced in #434 to stop non-ASCII header bytes from crashing Node / emptying Python, #430). That corrupts the common case of **UTF-8** header values (e.g. `Content-Disposition: filename="naïve.pdf"`) into mojibake (#479). The two positions genuinely conflict: a byte sequence can't be decoded as both ISO-8859-1 and UTF-8, and the "right" answer differs by ecosystem. ## How Each binding follows the reference client it emulates: - **Python** — `Response.headers` is now an **httpx-style `Headers` object** (adapted from httpx and kept in Python, like the existing `Cookies` class). It provides case-insensitive `str` access and a `.raw` property returning the exact wire bytes as `list[tuple[bytes, bytes]]`. The `Headers` object owns decoding — ascii → utf-8 → iso-8859-1, chosen once over the whole header set, exactly as httpx does — so UTF-8 header values decode correctly (fixes #479) while non-UTF-8 values fall back to ISO-8859-1 (keeps #434/#430). The Rust side just hands `Headers` the raw wire bytes. - **JavaScript** — keeps **strict ISO-8859-1** decoding (Fetch semantics). Because that mapping is a bijection, the string form stays byte-recoverable: `Buffer.from(value, 'latin1')` reproduces the exact bytes (and `.toString('utf8')` recovers a UTF-8 value). No raw-header accessor is added — it isn't part of the Fetch interface impit implements, and the latin-1 round-trip already covers the need. Header decoding now lives in exactly one place per binding (the Python `Headers` class; Node's inline latin-1). The former shared `decode_header_value` core helper is removed; body-charset detection reads the (ASCII, per RFC 9110) content-type via `String::from_utf8_lossy`. ### Net effect on #479 - **Python**: fully fixed — UTF-8 header values decode correctly, and `response.headers.raw` exposes exact bytes for signature/HMAC use. - **JavaScript**: string values remain ISO-8859-1 **by design** (Fetch parity), and are byte-recoverable via `Buffer.from(value, 'latin1')`. > Note: `response.headers` on the Python side is a read view — each access rebuilds it from the > response, so in-place mutations don't persist (documented on the accessor). ## Consistency with ecosystem Each binding matches the reference client it implements. ### Python — matches `httpx` (which impit-python implements) impit-python advertises the httpx interface ("drop-in replacement for `httpx.AsyncClient`"). This PR mirrors httpx's `Headers` directly: - httpx `Headers.encoding` tries `ascii`, then `utf-8`, then falls back to `iso-8859-1`: [`httpx/_models.py` @ v0.28.1](https://github.com/encode/httpx/blob/0.28.1/httpx/_models.py#L125-L145) — *"Header encoding is mandated as ascii, but we allow fallbacks to utf-8 or iso-8859-1."* - httpx exposes `Headers.raw: list[tuple[bytes, bytes]]`: [same file, `raw` property](https://github.com/encode/httpx/blob/0.28.1/httpx/_models.py#L152-L156). impit's `Response.headers` is that same `Headers` type with the same `.raw`. ### JavaScript — matches the Fetch API / undici (which impit-node implements) impit-node is "API-compatible with the Fetch API `Response`". In Fetch, header values are a [byte sequence](https://fetch.spec.whatwg.org/#concept-header-value) exposed to JS as a [`ByteString`](https://fetch.spec.whatwg.org/#headers-class), i.e. via [isomorphic decode](https://webidl.spec.whatwg.org/#idl-ByteString) — each byte `0x00–0xFF` maps to the code point of equal value (ISO-8859-1). This PR keeps impit-node on that behavior: - undici (Node's `fetch`) does the same: [nodejs/undici#1560](nodejs/undici#1560), [#1317](nodejs/undici#1317) (Latin-1 `ByteString`s). - Node's core `http` parser decodes header values as `latin1`/`binary` ([nodejs/node#17390](nodejs/node#17390), [#58240](nodejs/node#58240)); **axios** inherits this via its Node and XHR/Fetch adapters. Fetch has no raw-header accessor, and ISO-8859-1 is a bijection, so a JS raw accessor would be redundant surface — hence none is added. ## Tests - **Python:** unit tests for the `Headers` class (`.raw` byte-exactness, encoding selection ascii/utf-8/iso-8859-1, case-insensitive access, comma-joined duplicates, construction from bytes-pairs and str mappings); an integration test over a raw socket asserting a UTF-8 header decodes as UTF-8 (`encoding == 'utf-8'`) with exact `.raw` bytes; `response_test` uses `.headers.raw`. - **JavaScript:** existing latin-1 regression test kept; a new test asserts a UTF-8 header value round-trips via `Buffer.from(value, 'latin1').toString('utf8')`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1317
Properly converts headers to Latin1. Also fixes some inconsistencies with the regex check in webidl.