Skip to content

daemon: opt-in capture of exec output through the container's logging driver - #53407

Open
ndeloof wants to merge 7 commits into
moby:masterfrom
ndeloof:exec-log-capture
Open

ndeloof wants to merge 7 commits into
moby:masterfrom
ndeloof:exec-log-capture

Conversation

@ndeloof

@ndeloof ndeloof commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The output of an exec exists only for whoever is attached while it runs: nothing reaches the container's logging driver, so there is no post-hoc way to know what an exec printed, nor to tell which exec a given line belonged to.

This implements the proposal from the issue above, in five self-contained commits:

  • api: POST /containers/{id}/exec accepts CaptureStdout / CaptureStderr (opt-in tee of the exec's stdout and/or stderr into the container's logging driver, mirroring the AttachStdout / AttachStderr fields) and Labels (user-defined metadata, reported by exec inspect). Both gated on API v1.56: capture requests are rejected for older clients (a client asking for capture must not believe it happened when it didn't), Labels are dropped like ConsoleSize is for pre-1.42. Captured output is recorded on dedicated exec-stdout / exec-stderr log streams (stdcopy stream types 4 and 5), returned by GET /containers/{id}/logs only when requested through the new exec-stdout / exec-stderr query parameters.
  • daemon: the capture mirrors the container's own startLogging: a logger.Copier attached to the exec's stream pipes in InitializeStdio, before output starts flowing. Each message is stamped with exec_id and the exec's labels as per-message attributes. CloseStreams drains the copier so every captured line reached the driver before the exec is reported terminated. Capture is rejected at create when the container's log driver is none. Healthcheck probes are execs too and do not opt in, keeping their periodic runs out of the logs.
  • daemon/logger: json-file and local have per-entry attribute slots in their storage formats, but the write path only ever fed them the static logger-level attributes; per-message Message.Attrs are now merged over them (the read paths already decode them back, completing an existing round-trip). Messages without attributes keep the zero-cost static path.
  • daemon: exec labels are attached to the exec_create/exec_start/exec_detach/exec_die events, so subscribers can attribute exec activity without correlating in-memory execIDs.
  • integration: one success-case test — captured lines retrieved through the container logs endpoint on the right streams, stamped with the exec's identity under details; error cases are unit-tested.

Concrete consumer: Docker Compose lifecycle hooks (post_start/pre_stop) are execs; with this, docker compose logs --hooks becomes a stateless filter over the engine's own log stream (docker/compose#14088 / docker/compose#14091 carry that discussion).

Release notes

Add `CaptureStdout`/`CaptureStderr` and `Labels` to exec create: opt in to capturing an exec's output through the container's logging driver, on dedicated log streams with per-message attribution.

Created with: Claude Code

A picture of a cute animal

🦭

@thaJeztah

Copy link
Copy Markdown
Member

Discussed in the maintainers call (without looking closely at the implementation);

  • Overall it seems OK to combine it into the container logs, and same lifecycle
  • We discussed options to split the streams out (i.e., being able to get only the logs for the exec, or container, or vice-versa)

Can chat more tomorrow 😄

The output of an exec only exists for whoever is attached while it
runs: nothing reaches the container's logging driver, so there is no
post-hoc way to know what an exec printed, and no durable way to tell
which exec a given line belonged to.

ExecCreateRequest gains two optional fields:

- CaptureLogs tees the exec process's stdout and stderr into the
  container's logging driver. Capture is opt-in per exec: the daemon
  itself runs execs (healthcheck probes) whose unconditional capture
  would flood the logs of every healthchecked container.
- Labels holds user-defined metadata for the exec instance, reported
  by exec inspect and stamped on captured log messages as per-message
  attributes (alongside exec_id), so log readers can attribute output
  without holding any client-side state.

Signed-off-by: Nicolas De Loof <[email protected]>
When an exec is created with CaptureLogs, its stdout and stderr are
copied into the container's logging driver, mirroring the container's
own startLogging: a logger.Copier is attached to the exec's stream
pipes in InitializeStdio, before the process output starts flowing, so
no early output is missed. Each captured message is stamped with the
exec's identity — an exec_id attribute plus the exec's Labels, sorted
for deterministic order — so log readers can attribute the output
without holding any state. CloseStreams waits for the copier to drain,
guaranteeing every captured line reached the driver before the exec is
reported as terminated.

The driver is shared with the container's stdio copier; drivers already
accept concurrent Log calls from the stdout and stderr copy goroutines,
the capture copier follows the same contract.

Create-time validation rejects CaptureLogs when the container's log
driver is "none": there is nothing to capture into, and silently
accepting would let the caller believe output is being captured.

For API versions below 1.56 the router rejects CaptureLogs rather than
dropping it — a client asking for capture must not believe it happened
when it didn't — while Labels, pure metadata, are dropped like
ConsoleSize is for pre-1.42 clients. Exec inspect reports the labels.

Healthcheck probes are execs too; they do not set CaptureLogs, so
their periodic runs keep out of the container's logs.

Signed-off-by: Nicolas De Loof <[email protected]>
Both file-backed drivers have a per-entry attributes slot in their
storage format — json-file's "attrs" object and local's protobuf Attrs
— but the write path only ever fed them the static logger-level
attributes computed once from log-opts, and dropped the per-message
Message.Attrs. The read paths already decode entry attributes back
into Message.Attrs, so persisting them completes an existing
round-trip rather than introducing a format change.

Per-message attributes are merged over the static ones. Messages
without attributes — the overwhelming majority — keep the zero-cost
static path: json-file only remarshals when a message actually carries
attributes, and local builds a fresh slice only in that case (appending
to the shared static slice would mutate its backing array across
messages).

This gives captured exec output its attribution: the exec_id and label
attributes stamped by the exec log copier survive storage and surface
through the log endpoints when details are requested.

Signed-off-by: Nicolas De Loof <[email protected]>
exec_create, exec_start, exec_detach and exec_die events only carried
the execID (and exitCode for die). With execs now able to declare
labels, attach them to those events so subscribers can attribute exec
activity — which tool ran what — without correlating execIDs through
inspect, whose records are in-memory only and pruned after exit.

Reserved event keys and container labels keep precedence over exec
labels: LogContainerEventWithAttributes already copies container labels
over caller-supplied attributes, and the reserved keys are merged last.

Signed-off-by: Nicolas De Loof <[email protected]>
Success-case coverage for CaptureLogs: an exec created with capture and
labels prints on both streams; the lines are then retrieved through the
container logs endpoint, on the right streams, stamped with the exec's
identity when details are requested. Also asserts exec inspect reports
the declared labels.

Error cases (log driver "none", pre-1.55 API rejection) are covered by
unit tests.

Signed-off-by: Nicolas De Loof <[email protected]>
@ndeloof

ndeloof commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Splitting the streams is already possible client-side with what's in this PR: every captured line is stamped with an exec_id per-message attribute (plus any user-provided exec labels), and both json-file and local persist those attributes per message. With GET /containers/{id}/logs?details=1 each line comes back prefixed with its attributes (exec_id=<id>,mylabel=value <line>), so a client can keep only the lines carrying a given exec_id (exec-only) or drop every line that has an exec_id attribute (container-only). The structured JSON log format (format=json) exposes the same data as an Attrs field per message, which makes that filtering trivial for tooling. TestExecCaptureLogs covers the exec_id/labels round-trip through the logs endpoint.

For first-class server-side splitting I'd suggest a follow-up rather than growing this PR: either a filter on the container logs endpoint (by exec ID / attributes) or a dedicated GET /exec/{id}/logs endpoint — both can be built on the same per-message attributes without changing the storage format.

@thaJeztah

Copy link
Copy Markdown
Member

Oh! Forgot about this one.

I think the suggestion was to use the streams for this;

const (
Stdin StdType = 0 // Stdin represents standard input stream. It is present for completeness and should NOT be used. When reading the stream with [StdCopy] it is output on [Stdout].
Stdout StdType = 1 // Stdout represents standard output stream.
Stderr StdType = 2 // Stderr represents standard error steam.
Systemerr StdType = 3 // Systemerr represents errors originating from the system. When reading the stream with [StdCopy] it is returned as an error.
)

So add 2 extra streams for exec stdout/stderr;

const (
    Stdin      StdType = 0
    Stdout     StdType = 1
    Stderr     StdType = 2
    Systemerr  StdType = 3

    ExecStdout StdType = 4
    ExecStderr StdType = 5
)

Then the log endpoint could pass which streams to return;

?stdout=1&stderr=1&exec_stdout=1&exec_stderr=1

Or perhaps we should make that more generic for newer API versions;

?streams=stdout,stderr,exec-stdout,exec-stderr

The log could still have an exec_id attribute, but that's more of an extra in that case.

Instead of blending captured exec output into the container's own
stdout/stderr, record it under dedicated "exec-stdout"/"exec-stderr"
stream sources so log readers can serve them separately:
GET /containers/{id}/logs gains `exec-stdout` and `exec-stderr` query
parameters (API v1.56) to request them, and the multiplexed stream
format identifies those frames with the new stdcopy stream types
ExecStdout (4) and ExecStderr (5).

StdCopy demultiplexes the new types onto the regular stdout/stderr
writers, so existing consumers keep working; callers that need to tell
exec output apart from the container's own output can demultiplex on
the frame headers. The exec_id / label per-message attributes are
unchanged.

Suggested-by: Sebastiaan van Stijn <[email protected]>
Signed-off-by: Nicolas De Loof <[email protected]>
@vvoland vvoland added this to the 29.9.0 milestone Sep 3, 2026
Replace the all-or-nothing CaptureLogs boolean with CaptureStdout and
CaptureStderr, so an exec can capture only one of its output streams
(e.g. record stderr diagnostics while leaving a verbose stdout out of
the container's logs). The booleans mirror the AttachStdout /
AttachStderr fields of the same request, and the exec-stdout /
exec-stderr read side of the logs endpoint.

Capture pipes are only opened for the requested streams: an unread
pipe would block the stream broadcaster.

CaptureLogs never shipped in a release, so this is a plain rename, not
an API break.

Signed-off-by: Nicolas De Loof <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

Proposal: opt-in capture of exec output through the container's logging driver (follow-up to #32047)

3 participants