Repository navigation
Conversation
e6d97b7 to
21c8742
Compare
|
Discussed in the maintainers call (without looking closely at the implementation);
Can chat more tomorrow 😄 |
The output of an exec only exists for whoever is attached while it runs: nothing reaches the container's logging driver, so there is no post-hoc way to know what an exec printed, and no durable way to tell which exec a given line belonged to. ExecCreateRequest gains two optional fields: - CaptureLogs tees the exec process's stdout and stderr into the container's logging driver. Capture is opt-in per exec: the daemon itself runs execs (healthcheck probes) whose unconditional capture would flood the logs of every healthchecked container. - Labels holds user-defined metadata for the exec instance, reported by exec inspect and stamped on captured log messages as per-message attributes (alongside exec_id), so log readers can attribute output without holding any client-side state. Signed-off-by: Nicolas De Loof <[email protected]>
When an exec is created with CaptureLogs, its stdout and stderr are copied into the container's logging driver, mirroring the container's own startLogging: a logger.Copier is attached to the exec's stream pipes in InitializeStdio, before the process output starts flowing, so no early output is missed. Each captured message is stamped with the exec's identity — an exec_id attribute plus the exec's Labels, sorted for deterministic order — so log readers can attribute the output without holding any state. CloseStreams waits for the copier to drain, guaranteeing every captured line reached the driver before the exec is reported as terminated. The driver is shared with the container's stdio copier; drivers already accept concurrent Log calls from the stdout and stderr copy goroutines, the capture copier follows the same contract. Create-time validation rejects CaptureLogs when the container's log driver is "none": there is nothing to capture into, and silently accepting would let the caller believe output is being captured. For API versions below 1.56 the router rejects CaptureLogs rather than dropping it — a client asking for capture must not believe it happened when it didn't — while Labels, pure metadata, are dropped like ConsoleSize is for pre-1.42 clients. Exec inspect reports the labels. Healthcheck probes are execs too; they do not set CaptureLogs, so their periodic runs keep out of the container's logs. Signed-off-by: Nicolas De Loof <[email protected]>
Both file-backed drivers have a per-entry attributes slot in their storage format — json-file's "attrs" object and local's protobuf Attrs — but the write path only ever fed them the static logger-level attributes computed once from log-opts, and dropped the per-message Message.Attrs. The read paths already decode entry attributes back into Message.Attrs, so persisting them completes an existing round-trip rather than introducing a format change. Per-message attributes are merged over the static ones. Messages without attributes — the overwhelming majority — keep the zero-cost static path: json-file only remarshals when a message actually carries attributes, and local builds a fresh slice only in that case (appending to the shared static slice would mutate its backing array across messages). This gives captured exec output its attribution: the exec_id and label attributes stamped by the exec log copier survive storage and surface through the log endpoints when details are requested. Signed-off-by: Nicolas De Loof <[email protected]>
exec_create, exec_start, exec_detach and exec_die events only carried the execID (and exitCode for die). With execs now able to declare labels, attach them to those events so subscribers can attribute exec activity — which tool ran what — without correlating execIDs through inspect, whose records are in-memory only and pruned after exit. Reserved event keys and container labels keep precedence over exec labels: LogContainerEventWithAttributes already copies container labels over caller-supplied attributes, and the reserved keys are merged last. Signed-off-by: Nicolas De Loof <[email protected]>
Success-case coverage for CaptureLogs: an exec created with capture and labels prints on both streams; the lines are then retrieved through the container logs endpoint, on the right streams, stamped with the exec's identity when details are requested. Also asserts exec inspect reports the declared labels. Error cases (log driver "none", pre-1.55 API rejection) are covered by unit tests. Signed-off-by: Nicolas De Loof <[email protected]>
21c8742 to
6069bc8
Compare
|
Splitting the streams is already possible client-side with what's in this PR: every captured line is stamped with an For first-class server-side splitting I'd suggest a follow-up rather than growing this PR: either a filter on the container logs endpoint (by exec ID / attributes) or a dedicated |
|
Oh! Forgot about this one. I think the suggestion was to use the streams for this; moby/api/pkg/stdcopy/stdcopy.go Lines 14 to 19 in 4c6e571 So add 2 extra streams for exec stdout/stderr; const (
Stdin StdType = 0
Stdout StdType = 1
Stderr StdType = 2
Systemerr StdType = 3
ExecStdout StdType = 4
ExecStderr StdType = 5
)Then the log endpoint could pass which streams to return; Or perhaps we should make that more generic for newer API versions; The log could still have an exec_id attribute, but that's more of an extra in that case. |
Instead of blending captured exec output into the container's own
stdout/stderr, record it under dedicated "exec-stdout"/"exec-stderr"
stream sources so log readers can serve them separately:
GET /containers/{id}/logs gains `exec-stdout` and `exec-stderr` query
parameters (API v1.56) to request them, and the multiplexed stream
format identifies those frames with the new stdcopy stream types
ExecStdout (4) and ExecStderr (5).
StdCopy demultiplexes the new types onto the regular stdout/stderr
writers, so existing consumers keep working; callers that need to tell
exec output apart from the container's own output can demultiplex on
the frame headers. The exec_id / label per-message attributes are
unchanged.
Suggested-by: Sebastiaan van Stijn <[email protected]>
Signed-off-by: Nicolas De Loof <[email protected]>
Replace the all-or-nothing CaptureLogs boolean with CaptureStdout and CaptureStderr, so an exec can capture only one of its output streams (e.g. record stderr diagnostics while leaving a verbose stdout out of the container's logs). The booleans mirror the AttachStdout / AttachStderr fields of the same request, and the exec-stdout / exec-stderr read side of the logs endpoint. Capture pipes are only opened for the requested streams: an unread pipe would block the stream broadcaster. CaptureLogs never shipped in a release, so this is a plain rename, not an API break. Signed-off-by: Nicolas De Loof <[email protected]>
Summary
The output of an exec exists only for whoever is attached while it runs: nothing reaches the container's logging driver, so there is no post-hoc way to know what an exec printed, nor to tell which exec a given line belonged to.
This implements the proposal from the issue above, in five self-contained commits:
POST /containers/{id}/execacceptsCaptureStdout/CaptureStderr(opt-in tee of the exec's stdout and/or stderr into the container's logging driver, mirroring theAttachStdout/AttachStderrfields) andLabels(user-defined metadata, reported by exec inspect). Both gated on API v1.56: capture requests are rejected for older clients (a client asking for capture must not believe it happened when it didn't),Labelsare dropped likeConsoleSizeis for pre-1.42. Captured output is recorded on dedicatedexec-stdout/exec-stderrlog streams (stdcopy stream types 4 and 5), returned byGET /containers/{id}/logsonly when requested through the newexec-stdout/exec-stderrquery parameters.startLogging: alogger.Copierattached to the exec's stream pipes inInitializeStdio, before output starts flowing. Each message is stamped withexec_idand the exec's labels as per-message attributes.CloseStreamsdrains the copier so every captured line reached the driver before the exec is reported terminated. Capture is rejected at create when the container's log driver isnone. Healthcheck probes are execs too and do not opt in, keeping their periodic runs out of the logs.json-fileandlocalhave per-entry attribute slots in their storage formats, but the write path only ever fed them the static logger-level attributes; per-messageMessage.Attrsare now merged over them (the read paths already decode them back, completing an existing round-trip). Messages without attributes keep the zero-cost static path.exec_create/exec_start/exec_detach/exec_dieevents, so subscribers can attribute exec activity without correlating in-memory execIDs.details; error cases are unit-tested.Concrete consumer: Docker Compose lifecycle hooks (
post_start/pre_stop) are execs; with this,docker compose logs --hooksbecomes a stateless filter over the engine's own log stream (docker/compose#14088 / docker/compose#14091 carry that discussion).Release notes
Created with: Claude Code
A picture of a cute animal
🦭