Secret files not mounted in /run/secrets

Description

I am able to mount a file as a secret in /run/secrets via Dockerfile only, e.g. this example.

But when I try to use docker compose (V2) on top, nothing is mounted in /run/secrets.

Steps to reproduce the issue:

Working Dockerfile setup:

echo "SECRET CONTENT" > secret.txt

# syntax = docker/dockerfile:1.4
FROM debian:stable-slim
RUN --mount=type=secret,id=mysecret ls -la /run/secrets/

$ DOCKER_BUILDKIT=1 docker build --progress=plain --no-cache --secret id=mysecret,src=secret.txt .

#9 [stage-0 2/2] RUN --mount=type=secret,id=mysecret ls -la /run/secrets/
#9 0.332 total 12
#9 0.332 drwxr-xr-x 2 root root 4096 Sep  8 14:35 .
#9 0.332 drwxr-xr-x 1 root root 4096 Sep  8 14:35 ..
#9 0.332 -r-------- 1 root root   15 Sep  8 14:35 mysecret

Adding docker-compose.yml on top:

docker compose build --no-cache --progress=plain

#9 [stage-0 2/2] RUN --mount=type=secret,id=mysecret ls -la /run/secrets/
#9 0.373 ls: cannot access '/run/secrets/': No such file or directory
#9 ERROR: executor failed running [/bin/sh -c ls -la /run/secrets/]: exit code: 2

Describe the results you received:
/run/secrets/: No such file or directory

Describe the results you expected:
/run/secrets/mysecret found.

Additional information you deem important:

A lot of work on secrets support has been added lately, and to my understanding, this was added as in version 2.5.0 / #6358, correct me if I'm wrong. The docs/PR has not been too clear. I'm using the latest Docker Compose V2 version 2.10.2.

I have also tested with docker-compose V1.

$ DOCKER_BUILDKIT=1 docker-compose build --no-cache --progress=plain
#9 [stage-0 2/2] RUN --mount=type=secret,id=mysecret ls -la /run/secrets/
#9 0.387 ls: cannot access '/run/secrets/': No such file or directory

I'm not sure if compose V1 supports secrets. But if I make obvious syntax mistakes in docker-compose.yml (e.g. inconsistent secret names or a wrong filename), compose V1 understands the issue and provides meaningfull errors. V2 ignores these syntax errors. Maybe this could be a hint. Example:

services:
  app:
    build: .
    secrets:
      - mysecret_X
secrets:
  mysecret:
    file: secret.txt

$ DOCKER_BUILDKIT=1 docker-compose build --no-cache --progress=plain
ERROR: Service "app" uses an undefined secret "mysecret_X"

$ docker compose build --no-cache --progress=plain
#9 0.398 ls: cannot access '/run/secrets/': No such file or directory

Output of docker compose version:

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:46 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:00:51 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.10.2)
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 39
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 24.88GiB
 Name: LPCP2003
 ID: C3RB:XUID:KZWW:WYF5:757G:LNJW:S74X:HWYA:54FH:DMSN:3IJQ:NC2E
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details:

WSL2 with

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

Your service config is malformed for build-time secrets - it should be <svc>.build.secrets not <svc>.secrets.

Note that syntactically both are valid, but <svc>.build.secrets exposes secrets to RUN --mount=type=secret,... (docker compose build ...) while <svc>.secrets exposes secrets at runtime (docker compose up ...).

Here's an amended version of your example (need to use the expanded form of build with build.context):

services:
  app:
    build:
      context: .
      secrets:
        - mysecret
secrets:
  mysecret:
    file: secret.txt

By default, secrets are optional, which is why you don't get an error until trying to access it. Mark the secret as required to get a better error:

RUN --mount=type=secret,id=mysecret,required ls -la /run/secrets/

$ docker compose build
...
failed to solve: secret mysecret not found: not found

@milas

Your service config is malformed for build-time secrets - it should be <svc>.build.secrets not <svc>.secrets.

Note that syntactically both are valid, but <svc>.build.secrets exposes secrets to RUN --mount=type=secret,... (docker compose build ...) while <svc>.secrets exposes secrets at runtime (docker compose up ...).

Oh my! First of all, thanks @milas, you saved my week!

I hindsight, maybe the docs could improve.

I didn't realize that secrets could be specified in the build.secrets as well. I had the docs of <srv>.secrets at hand - how could I've known that I was looking at the wrong docs...
The description in <srv>.secrets and <srv>.build.secrets, are 100% identical. But your explanation reveals, that their scope and usage is very different.

One proposal would be to place some notes like

Note: Secrets defined in the service context are exposed to the service at runtime. To expose secrets at buildtime see <build.secrets>.

and vice versa in the other location.

Seems like I'm not the only one to get this wrong, here some examples: #6358 (comment), #9193

Absolutely agreed that we need some refinement on the spec/docs here! I've made an internal ticket for that (EN-229 for other Docker peeps) so that it gets to the right team 🙂

Your service config is malformed for build-time secrets - it should be <svc>.build.secrets not <svc>.secrets.

Oh my goodness, I could not figure out why my colleague's code was not working. Thank you so much. Some documentation on placing the secrets at each level would be helpful as the documentation around this is a bit sparse still.

My IDE using this docker-compose version 3.9 file marks services.foo.build.secrets (as opposed to one level up) as red/error, so I thought it was invalid as well until I tried it and it worked fine.

Should I be using a higher version of docker compose, or is there some metadata file that needs updating?

@jamisonbryant

Oh, well, I just spent quite some time on this because I had secrets at the same level as build instead of being a field under it... anyway:

Should I be using a higher version of docker compose, or is there some metadata file that needs updating?

@jamisonbryant version is deprecated so you should be able to remove it. Also, the newer docker compose command will ignore the version field. The version is only used when running the old docker-compose (with a dash) command.

I had a related problem.

Perhaps I missed it in the docs, but the two step process outlined in the manual could also mention that for build time secrets the secret also needs to be mounted in the RUN command where it's used.

I lost a day. Then thankfully found this thread :)

Are there any plans to update the Docker docs? I ran into the same pitfall that multiple people mentioned in the thread.

Are there any plans to update the Docker docs? I ran into the same pitfall that multiple people mentioned in the thread.

Exactly the same here.

Same here!

Navigation Menu

Repository navigation

Secret files not mounted in /run/secrets #9822

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Secret files not mounted in /run/secrets #9822

Description

Activity

milas commented on Sep 8, 2022

th0ger commented on Sep 8, 2022

th0ger commented on Sep 8, 2022

milas commented on Sep 8, 2022

tleerai commented on Jan 21, 2023

jamisonbryant commented on May 9, 2023

rafaelbiten commented on May 15, 2023

TBlackmore commented on Aug 25, 2023

trevor-evans commented on Jun 19, 2024

samcole8 commented on Jun 27, 2024

joshmurrayeu commented on Aug 21, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions