Repository navigation
Lambda extension stops polling after a 300s invocation, hanging every later invocation on that execution environment #24218
Description
Which SDK are you using?
@sentry/aws-serverless
SDK Version
10.36.0 (Lambda layer SentryNodeServerlessSDKv10:49). Also present on develop (10.73.0,
layer :88) — the extension source is unchanged, so this is not fixed by upgrading.
Framework Version
AWS Lambda, nodejs22.x
Steps to Reproduce
No AWS account needed — the official runtime image runs external extensions.
cd packages/aws-serverless && yarn build:extension
mkdir -p /tmp/repro/task /tmp/repro/opt/extensions /tmp/repro/opt/sentry-extension
cat > /tmp/repro/task/index.js <<'EOF'
exports.handler = async event => {
await new Promise(r => setTimeout(r, Number(event.sleepMs || 0)));
return { ok: true };
};
EOF
cp build/lambda-extension/sentry-extension /tmp/repro/opt/extensions/
cp build/lambda-extension/index.mjs /tmp/repro/opt/sentry-extension/
# AWS_LAMBDA_FUNCTION_TIMEOUT matters: the emulator defaults to 300s, the same number
# under test, and a function timeout at 300s masks the bug entirely.
docker run -d --name repro -p 9000:8080 \
-e AWS_LAMBDA_FUNCTION_TIMEOUT=900 \
-v /tmp/repro/task:/var/task:ro -v /tmp/repro/opt:/opt:ro \
public.ecr.aws/lambda/nodejs:22 index.handler
URL=http://localhost:9000/2015-03-31/functions/function/invocations
curl -s -XPOST $URL -d '{"sleepMs":310000}' # {"ok":true} after ~310s
curl -s --max-time 60 -XPOST $URL -d '{"sleepMs":100}' # never returns
docker logs reproExpected Result
Both invocations return.
Actual Result
The second invocation's handler finishes in ~110ms and the response never comes:
handler: done
INVOKE RTDONE(status: success, produced bytes: 0, duration: 107.806000ms)
<- nothing after this
External agent sentry-extension ... registered appears exactly once, so both invocations ran
on the same execution environment — the first one poisoned it. On real Lambda this ends as
Status: timeout with the handler long finished, and PostRuntimeExtensionsDuration equal to
the full function timeout.
Production numbers from one SQS worker (600s timeout, 155,083 invocations over 7 days):
| statistic | value |
|---|---|
PostRuntimeExtensionsDuration p50 / p90 / p99.9 |
0 / 0 / 0.04 ms |
| maximum | 599,949.55 ms |
37 invocations hit Status: timeout in that window, against 33 that ran past 300s and did not
— about one poisoned environment each, which is what the mechanism predicts.
Additional Context
Mechanism. /event/next both acknowledges the previous event and waits for the next one,
so the poll issued after each event stays open for the whole of the following invocation.
Node's fetch applies undici's headersTimeout, default 300,000 ms — measured against a
server that accepts the connection and never sends headers:
rejected after 301.0 s -> UND_ERR_HEADERS_TIMEOUT. Frozen time does not count against it
(undici's clock is tick-driven, fastNow += TICK_MS), so only a genuinely long invocation
triggers this, which is why the cliff sits exactly at 300s.
The rejection then escapes the loop, which has no try/catch:
while (true) { await extension.next(); }
main().catch(err => { DEBUG_BUILD && debug.error('Error in Lambda Extension', err); });Why it is silent. Lambda ends an invocation only once the runtime and every registered
extension have asked for the next event, so the long invocation itself completes normally and
later ones hang with nothing logged. And debug is only enabled from Sentry.init, which this
separate process never calls — debug.error here cannot print under any option or env var.
Scope. The extension is enabled by default for layer users, and useLayerExtension: false
does not help: the layer ships the binary in /opt/extensions/, which Lambda starts regardless
of SDK options. Reproduced identically with the published SentryNodeServerlessSDKv10:88 layer,
and does not reproduce with a patched extension.
Metadata
Metadata
Assignees
Projects
- Status
Activity
linear-code commentedon Sep 8, 2026
JS-3635
fix(aws-serverless): Keep the Lambda extension polling past 300s invo…
LuccaRebelloToledo commentedon Sep 8, 2026
Small triage note: this is a bug report, but it was filed through the CLI, which skips the
bug.ymlform — so it never picked up theBuglabel the template applies and was auto-labelledFeature/Waiting for: Product Owner. Could someone with triage rights relabel it? I don't have the permission to do it myself.A fix is up in #24219.
fix(aws-serverless): Keep the Lambda extension polling past 300s invo…
fix(aws-serverless): Keep the Lambda extension polling past 300s invo…
fix(aws-serverless): Keep the Lambda extension polling past 300s invo…
LuccaRebelloToledo commentedon Sep 8, 2026
Following up with the controls, since "the second invocation hangs" on its own does not
establish that 300s is the trigger. Same harness throughout —
public.ecr.aws/lambda/nodejs:22,AWS_LAMBDA_FUNCTION_TIMEOUT=900, second invocation always 100ms:develop(yarn build:extension)developdevelopSentryNodeServerlessSDKv10:88Every run logged exactly one
External agent sentry-extension ... registeredagainst twoINVOKE STARTs, so none of these came from a reset execution environment.So it takes both conditions — the unpatched extension and an invocation that crosses the
threshold — and the threshold is bracketed at (290s, 310s]. That lines up with the two
independent measurements already in the report: undici's
headersTimeoutdefault timing out at301.0s in isolation, and the shortest poisoning invocation observed in production at 301,095ms.
msonnb commentedon Sep 9, 2026
Hey @LuccaRebelloToledo, thanks for reporting this and opening a fix! I'm gonna take a look.
fix(aws-serverless): Never let the extension stop polling or exit mid…
LuccaRebelloToledo commentedon Sep 17, 2026
Heads-up on scope before you spend more time on #24219, @msonnb.
It started as the polling fix described here and grew while I was verifying it. It's now 24 files: the 300s poll cliff, plus SHUTDOWN-only registration (the two are load-bearing on each other — the exit policy is only safe once the extension is out of the invocation gate), the shutdown drain, and a separate pre-existing bug where every envelope over 32KiB was answered 500 and dropped because the tunnel couldn't read a gzipped envelope header.
The extension also went from one file to ten with tests mirroring them, which is most of the line count.
I know you offered a follow-up for the drain and I kept it here — reasoning is on that review thread. Happy to split any of it out, the gzip fix especially. Say the word and I'll carve it up rather than have you review it as one piece.
msonnb commentedon Sep 18, 2026
@LuccaRebelloToledo thanks for the update. Since this grew significantly in scope, it might take some time to review since we're pretty busy right now preparing the next major of the SDK.
fix(aws-serverless): Keep Lambda extension polling past 300s (#24811)
msonnb commentedon Sep 29, 2026
the fix will be released in the next versions of v10 and v11
github-actions commentedon Oct 1, 2026
A PR closing this issue has just been released 🚀
This issue was referenced by PR #24811, which was included in the 11.2.0 release.