Skip to content

fix(fetch): ByteString checks & conversion in Headers - #1560

Merged
ronag merged 1 commit into
nodejs:mainfrom
KhafraDev:bytestring-headers
Jul 19, 2022
Merged

ronag merged 1 commit into
nodejs:mainfrom
KhafraDev:bytestring-headers

Conversation

@KhafraDev

Copy link
Copy Markdown
Member

Fixes #1317

Properly converts headers to Latin1. Also fixes some inconsistencies with the regex check in webidl.

@codecov-commenter

codecov-commenter commented Jul 18, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1560 (f4bec9e) into main (0ab421f) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1560   +/-   ##
=======================================
  Coverage   94.91%   94.91%           
=======================================
  Files          50       50           
  Lines        4702     4702           
=======================================
  Hits         4463     4463           
  Misses        239      239           
Impacted Files Coverage Δ
lib/fetch/index.js 81.58% <100.00%> (ø)
lib/fetch/webidl.js 98.29% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ab421f...f4bec9e. Read the comment docs.

@ronag
ronag merged commit 76f6627 into nodejs:main Jul 19, 2022
@KhafraDev
KhafraDev deleted the bytestring-headers branch July 19, 2022 14:45
ken-matsui pushed a commit to ken-matsui/cabin-website that referenced this pull request Aug 19, 2022
Bumps undici from 5.8.0 to 5.9.1.

Release notes
Sourced from undici's releases.

v5.9.1
What's Changed

fix: don't timeout while waiting for client to send request (#1604)
Fix array headers by @​mateonunez in nodejs/undici#1598
fix(fetch): implement fully read body algorithm by @​KhafraDev in nodejs/undici#1597
fix: add support for integrity option to Fetch by @​jelmervdl in nodejs/undici#1596
fix(File): respect typed array byteOffset and byteLength by @​mrbbot in nodejs/undici#1601

New Contributors

@​mateonunez made their first contribution in nodejs/undici#1598
@​jelmervdl made their first contribution in nodejs/undici#1596
@​mrbbot made their first contribution in nodejs/undici#1601

Full Changelog: nodejs/[email protected]
v5.8.2
⚠️ Security Release ⚠️

CRLF Injection in Nodejs ‘undici’ via Content-Type GHSA-f772-66g8-q5h3 CVE-2022-35948
undici.request vulnerable to SSRF using absolute URL on pathname GHSA-8qr4-xgw6-wmr3 CVE-2022-35949

What's Changed

docs: mock different endpoints in a single file by @​ritvik130 in nodejs/undici#1589
feat(webidl): better error message for ByteString converter by @​KhafraDev in nodejs/undici#1591

New Contributors

@​ritvik130 made their first contribution in nodejs/undici#1589

Full Changelog: nodejs/[email protected]
v5.8.1
What's Changed

Do not decode the body while we are following a redirect by @​mcollina in nodejs/undici#1554
docs: Fix spelling/grammar in "Mocking Request" by @​meyfa in nodejs/undici#1555
fix(MockInterceptor): callback options.headers w/ fetch by @​KhafraDev in nodejs/undici#1559
fix(fetch): ByteString checks & conversion in Headers by @​KhafraDev in nodejs/undici#1560
test: update client certificates with ones that expires in 100 years by @​jodevsa in nodejs/undici#1566
fix: x-www-form-urlencoded parser keep the BOM by @​cola119 in nodejs/undici#1563
fix: prioritise error events over timeouts by @​jodevsa in nodejs/undici#1551
fix: add isErrorLike by @​KhafraDev in nodejs/undici#1570
fix(types): add missing pool stats by @​SkeLLLa in nodejs/undici#1573
fix: fetch a long base64 url will crash and nothing happens (close: #1574) by @​ahaoboy in nodejs/undici#1575
fix: follow signal.reason in Request by @​LiviaMedeiros in nodejs/undici#1580
docs: Fix DiagnosticsChannel sidebar link by @​trentm in nodejs/undici#1582
fix: make mock headers case-insensitive by @​cola119 in nodejs/undici#1585

New Contributors

@​meyfa made their first contribution in nodejs/undici#1555
@​cola119 made their first contribution in nodejs/undici#1563
@​SkeLLLa made their first contribution in nodejs/undici#1573



... (truncated)


Commits

5890e16 5.9.1
ecae314 fix: don't timeout while waiting for client to send request (#1604)
fa9fd90 fix(File): respect typed array byteOffset and byteLength (#1601)
ae6f554 fix: add support for integrity option to Fetch (#1596)
deed628 fix(fetch): implement fully read body algorithm (#1597)
0d1419c Fix array headers (#1598)
52d1ce5 Bumped v5.8.2
66165d6 Merge pull request from GHSA-f772-66g8-q5h3
124f7eb Merge pull request from GHSA-8qr4-xgw6-wmr3
aef314c feat(webidl): better error message for ByteString converter (#1591)
Additional commits viewable in compare view




Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the Security Alerts page.

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
metcoder95 pushed a commit to metcoder95/undici that referenced this pull request Dec 26, 2022
crysmags pushed a commit to crysmags/undici that referenced this pull request Feb 27, 2024
ken-matsui pushed a commit to cabinpkg/cabin that referenced this pull request May 30, 2026
Bumps undici from 5.8.0 to 5.9.1.

Release notes
Sourced from undici's releases.

v5.9.1
What's Changed

fix: don't timeout while waiting for client to send request (#1604)
Fix array headers by @​mateonunez in nodejs/undici#1598
fix(fetch): implement fully read body algorithm by @​KhafraDev in nodejs/undici#1597
fix: add support for integrity option to Fetch by @​jelmervdl in nodejs/undici#1596
fix(File): respect typed array byteOffset and byteLength by @​mrbbot in nodejs/undici#1601

New Contributors

@​mateonunez made their first contribution in nodejs/undici#1598
@​jelmervdl made their first contribution in nodejs/undici#1596
@​mrbbot made their first contribution in nodejs/undici#1601

Full Changelog: nodejs/[email protected]
v5.8.2
⚠️ Security Release ⚠️

CRLF Injection in Nodejs ‘undici’ via Content-Type GHSA-f772-66g8-q5h3 CVE-2022-35948
undici.request vulnerable to SSRF using absolute URL on pathname GHSA-8qr4-xgw6-wmr3 CVE-2022-35949

What's Changed

docs: mock different endpoints in a single file by @​ritvik130 in nodejs/undici#1589
feat(webidl): better error message for ByteString converter by @​KhafraDev in nodejs/undici#1591

New Contributors

@​ritvik130 made their first contribution in nodejs/undici#1589

Full Changelog: nodejs/[email protected]
v5.8.1
What's Changed

Do not decode the body while we are following a redirect by @​mcollina in nodejs/undici#1554
docs: Fix spelling/grammar in "Mocking Request" by @​meyfa in nodejs/undici#1555
fix(MockInterceptor): callback options.headers w/ fetch by @​KhafraDev in nodejs/undici#1559
fix(fetch): ByteString checks & conversion in Headers by @​KhafraDev in nodejs/undici#1560
test: update client certificates with ones that expires in 100 years by @​jodevsa in nodejs/undici#1566
fix: x-www-form-urlencoded parser keep the BOM by @​cola119 in nodejs/undici#1563
fix: prioritise error events over timeouts by @​jodevsa in nodejs/undici#1551
fix: add isErrorLike by @​KhafraDev in nodejs/undici#1570
fix(types): add missing pool stats by @​SkeLLLa in nodejs/undici#1573
fix: fetch a long base64 url will crash and nothing happens (close: #1574) by @​ahaoboy in nodejs/undici#1575
fix: follow signal.reason in Request by @​LiviaMedeiros in nodejs/undici#1580
docs: Fix DiagnosticsChannel sidebar link by @​trentm in nodejs/undici#1582
fix: make mock headers case-insensitive by @​cola119 in nodejs/undici#1585

New Contributors

@​meyfa made their first contribution in nodejs/undici#1555
@​cola119 made their first contribution in nodejs/undici#1563
@​SkeLLLa made their first contribution in nodejs/undici#1573



... (truncated)


Commits

5890e16 5.9.1
ecae314 fix: don't timeout while waiting for client to send request (#1604)
fa9fd90 fix(File): respect typed array byteOffset and byteLength (#1601)
ae6f554 fix: add support for integrity option to Fetch (#1596)
deed628 fix(fetch): implement fully read body algorithm (#1597)
0d1419c Fix array headers (#1598)
52d1ce5 Bumped v5.8.2
66165d6 Merge pull request from GHSA-f772-66g8-q5h3
124f7eb Merge pull request from GHSA-8qr4-xgw6-wmr3
aef314c feat(webidl): better error message for ByteString converter (#1591)
Additional commits viewable in compare view




Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the Security Alerts page.

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Pijukatel added a commit to apify/impit that referenced this pull request Jul 7, 2026
…le Response.headers (#492)

Fixes #479 (and revisits the decision made in #434).

## Context

Response header values were decoded as ISO-8859-1 (`b as char`,
introduced in #434 to stop
non-ASCII header bytes from crashing Node / emptying Python, #430). That
corrupts the common case
of **UTF-8** header values (e.g. `Content-Disposition:
filename="naïve.pdf"`) into mojibake
(#479). The two positions genuinely conflict: a byte sequence can't be
decoded as both ISO-8859-1
and UTF-8, and the "right" answer differs by ecosystem.

## How

Each binding follows the reference client it emulates:

- **Python** — `Response.headers` is now an **httpx-style `Headers`
object** (adapted from httpx and
kept in Python, like the existing `Cookies` class). It provides
case-insensitive `str` access and
a `.raw` property returning the exact wire bytes as `list[tuple[bytes,
bytes]]`. The `Headers`
object owns decoding — ascii → utf-8 → iso-8859-1, chosen once over the
whole header set, exactly
as httpx does — so UTF-8 header values decode correctly (fixes #479)
while non-UTF-8 values fall
back to ISO-8859-1 (keeps #434/#430). The Rust side just hands `Headers`
the raw wire bytes.
- **JavaScript** — keeps **strict ISO-8859-1** decoding (Fetch
semantics). Because that mapping is a
bijection, the string form stays byte-recoverable: `Buffer.from(value,
'latin1')` reproduces the
exact bytes (and `.toString('utf8')` recovers a UTF-8 value). No
raw-header accessor is added —
it isn't part of the Fetch interface impit implements, and the latin-1
round-trip already covers
  the need.

Header decoding now lives in exactly one place per binding (the Python
`Headers` class; Node's
inline latin-1). The former shared `decode_header_value` core helper is
removed; body-charset
detection reads the (ASCII, per RFC 9110) content-type via
`String::from_utf8_lossy`.

### Net effect on #479
- **Python**: fully fixed — UTF-8 header values decode correctly, and
`response.headers.raw` exposes
  exact bytes for signature/HMAC use.
- **JavaScript**: string values remain ISO-8859-1 **by design** (Fetch
parity), and are
  byte-recoverable via `Buffer.from(value, 'latin1')`.

> Note: `response.headers` on the Python side is a read view — each
access rebuilds it from the
> response, so in-place mutations don't persist (documented on the
accessor).

## Consistency with ecosystem

Each binding matches the reference client it implements.

### Python — matches `httpx` (which impit-python implements)

impit-python advertises the httpx interface ("drop-in replacement for
`httpx.AsyncClient`"). This
PR mirrors httpx's `Headers` directly:

- httpx `Headers.encoding` tries `ascii`, then `utf-8`, then falls back
to `iso-8859-1`:
[`httpx/_models.py` @
v0.28.1](https://github.com/encode/httpx/blob/0.28.1/httpx/_models.py#L125-L145)
— *"Header encoding is mandated as ascii, but we allow fallbacks to
utf-8 or iso-8859-1."*
- httpx exposes `Headers.raw: list[tuple[bytes, bytes]]`:
[same file, `raw`
property](https://github.com/encode/httpx/blob/0.28.1/httpx/_models.py#L152-L156).
impit's `Response.headers` is that same `Headers` type with the same
`.raw`.

### JavaScript — matches the Fetch API / undici (which impit-node
implements)

impit-node is "API-compatible with the Fetch API `Response`". In Fetch,
header values are a
[byte sequence](https://fetch.spec.whatwg.org/#concept-header-value)
exposed to JS as a
[`ByteString`](https://fetch.spec.whatwg.org/#headers-class), i.e. via
[isomorphic decode](https://webidl.spec.whatwg.org/#idl-ByteString) —
each byte `0x00–0xFF` maps to
the code point of equal value (ISO-8859-1). This PR keeps impit-node on
that behavior:

- undici (Node's `fetch`) does the same:
[nodejs/undici#1560](nodejs/undici#1560),
[#1317](nodejs/undici#1317) (Latin-1
`ByteString`s).
- Node's core `http` parser decodes header values as `latin1`/`binary`
  ([nodejs/node#17390](nodejs/node#17390),
[#58240](nodejs/node#58240)); **axios**
inherits this via its Node and
  XHR/Fetch adapters.

Fetch has no raw-header accessor, and ISO-8859-1 is a bijection, so a JS
raw accessor would be
redundant surface — hence none is added.

## Tests

- **Python:** unit tests for the `Headers` class (`.raw` byte-exactness,
encoding selection
ascii/utf-8/iso-8859-1, case-insensitive access, comma-joined
duplicates, construction from
bytes-pairs and str mappings); an integration test over a raw socket
asserting a UTF-8 header
decodes as UTF-8 (`encoding == 'utf-8'`) with exact `.raw` bytes;
`response_test` uses
  `.headers.raw`.
- **JavaScript:** existing latin-1 regression test kept; a new test
asserts a UTF-8 header value
  round-trips via `Buffer.from(value, 'latin1').toString('utf8')`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fetch fails on ByteString header with non-ASCII chars

3 participants