Repository navigation
[BUG] docker compose does not start newly built image when COMPOSE_BAKE is true #12774
Description
Description
Running docker compose up -d --build «container» in the presence of COMPOSE_BAKE=true does not re-create the container with the newly-built image. Removing COMPOSE_BAKE from the environment returns to the original behaviour.
Steps To Reproduce
Scenario
I am using an instance of a customised MariaDB container intended to be the back-end for a Gitea container. Although I haven't tested with different base images, I don't believe that the base image is relevant. MariaDB is simply where I noticed this problem and am able to demonstrate it easily.
Instrumentation
Instrumentation used to demonstrate problem:
-
Health-check script includes:
FORCE_CHANGE=1 echo "Forced change marker = $FORCE_CHANGE"
-
Dockerfile includes:
RUN date >/builddate.txtThis is positioned after the health-check script is copied into the image.
Runtime environment
$ echo $COMPOSE_BAKE
true
$Test 1
-
Show container not running:
$ docker top gitea_db Error response from daemon: No such container: gitea_db
-
Build container:
$ docker compose up -d --build gitea_db [+] Building 0.5s (11/11) FINISHED => [internal] load local bake definitions 0.0s => => reading from stdin 382B 0.0s => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.16kB 0.0s => [internal] load metadata for ghcr.io/linuxserver/mariadb:latest 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [1/4] FROM ghcr.io/linuxserver/mariadb:latest 0.0s => [internal] load build context 0.0s => => transferring context: 45B 0.0s => CACHED [2/4] RUN for CNF in /defaults/my.cnf /defaults/custom.cnf ; do [ -f ${CNF} ] && break ; done ; sed -i.bak -e "s/^thread_cache_siz 0.0s => CACHED [3/4] COPY iotstack_healthcheck.sh /usr/local/bin/iotstack_healthcheck.sh 0.0s => CACHED [4/4] RUN date >/builddate.txt 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:e66432205ec39196c2cac3dde2391bbe741af5906b26ba8508127ff3af49f56c 0.0s => => naming to docker.io/library/iotstack-gitea_db 0.0s => resolving provenance for metadata file 0.0s [+] Running 3/3 ✔ gitea_db Built 0.0s ✔ Network iotstack_nextcloud Created 0.1s ✔ Container gitea_db Started 0.6s $
-
Probe instrumentation:
$ docker exec gitea_db cat /builddate.txt Thu Apr 24 01:32:18 UTC 2025 $ docker exec gitea_db iotstack_healthcheck.sh Forced change marker = 1
This is the baseline.
-
Bump counter in health-check script:
FORCE_CHANGE=2 echo "Forced change marker = $FORCE_CHANGE"
-
Rebuild container:
$ docker compose up -d --build gitea_db [+] Building 0.5s (11/11) FINISHED => [internal] load local bake definitions 0.0s => => reading from stdin 382B 0.0s => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.16kB 0.0s => [internal] load metadata for ghcr.io/linuxserver/mariadb:latest 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [1/4] FROM ghcr.io/linuxserver/mariadb:latest 0.0s => [internal] load build context 0.0s => => transferring context: 1.08kB 0.0s => CACHED [2/4] RUN for CNF in /defaults/my.cnf /defaults/custom.cnf ; do [ -f ${CNF} ] && break ; done ; sed -i.bak -e "s/^thread_cache_siz 0.0s => CACHED [3/4] COPY iotstack_healthcheck.sh /usr/local/bin/iotstack_healthcheck.sh 0.0s => CACHED [4/4] RUN date >/builddate.txt 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:a59aa5c348501b84089dd38195edddb1167175750fc0f7b189f216e3b5faa7ac 0.0s => => naming to docker.io/library/iotstack-gitea_db 0.0s => resolving provenance for metadata file 0.0s [+] Running 2/2 ✔ gitea_db Built 0.0s ✔ Container gitea_db Running 0.0s
Note that container is marked "running" rather than being re-created.
-
Probe instrumentation:
$ docker exec gitea_db cat /builddate.txt Thu Apr 24 01:32:18 UTC 2025 $ docker exec gitea_db iotstack_healthcheck.sh Forced change marker = 1
No change from baseline so this is still the prior image that is running.
-
Send another explicit "up":
$ docker compose up -d gitea_db [+] Running 1/1 ✔ Container gitea_db Started 4.1s
Container "Started", as distinct from "Running" at the end of step 5.
-
Probe instrumentation:
$ docker exec gitea_db cat /builddate.txt Thu Apr 24 01:30:47 UTC 2025 $ docker exec gitea_db iotstack_healthcheck.sh Forced change marker = 2
Newly-built image now running.
Revised runtime environment
$ echo $COMPOSE_BAKE
$Test 2
-
Create initial conditions (steps 1..3 of Test 1):
$ docker top gitea_db Error response from daemon: No such container: gitea_db $ docker compose up -d --build gitea_db Compose can now delegate builds to bake for better performance. To do so, set COMPOSE_BAKE=true. [+] Building 0.9s (10/10) FINISHED docker:default => [gitea_db internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.16kB 0.0s => [gitea_db internal] load metadata for ghcr.io/linuxserver/mariadb:latest 0.0s => [gitea_db internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [gitea_db internal] load build context 0.0s => => transferring context: 1.08kB 0.0s => [gitea_db 1/4] FROM ghcr.io/linuxserver/mariadb:latest 0.0s => CACHED [gitea_db 2/4] RUN for CNF in /defaults/my.cnf /defaults/custom.cnf ; do [ -f ${CNF} ] && break ; done ; sed -i.bak -e "s/^thread_ 0.0s => [gitea_db 3/4] COPY iotstack_healthcheck.sh /usr/local/bin/iotstack_healthcheck.sh 0.1s => [gitea_db 4/4] RUN date >/builddate.txt 0.5s => [gitea_db] exporting to image 0.1s => => exporting layers 0.1s => => writing image sha256:3d783b11bca0626cb88cb08ad48f16f16db5a662e145bc94e68058b1d0cf0f17 0.0s => => naming to docker.io/library/iotstack-gitea_db 0.0s => [gitea_db] resolving provenance for metadata file 0.0s [+] Running 3/3 ✔ gitea_db Built 0.0s ✔ Network iotstack_nextcloud Created 0.1s ✔ Container gitea_db Started 0.5s $ docker exec gitea_db cat /builddate.txt Thu Apr 24 01:51:42 UTC 2025 $ docker exec gitea_db iotstack_healthcheck.sh Forced change marker = 3
-
Bump forced-change marker to 4 and rebuild:
$ docker compose up -d --build gitea_db Compose can now delegate builds to bake for better performance. To do so, set COMPOSE_BAKE=true. [+] Building 0.9s (10/10) FINISHED docker:default => [gitea_db internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.16kB 0.0s => [gitea_db internal] load metadata for ghcr.io/linuxserver/mariadb:latest 0.0s => [gitea_db internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [gitea_db 1/4] FROM ghcr.io/linuxserver/mariadb:latest 0.0s => [gitea_db internal] load build context 0.0s => => transferring context: 1.08kB 0.0s => CACHED [gitea_db 2/4] RUN for CNF in /defaults/my.cnf /defaults/custom.cnf ; do [ -f ${CNF} ] && break ; done ; sed -i.bak -e "s/^thread_ 0.0s => [gitea_db 3/4] COPY iotstack_healthcheck.sh /usr/local/bin/iotstack_healthcheck.sh 0.0s => [gitea_db 4/4] RUN date >/builddate.txt 0.5s => [gitea_db] exporting to image 0.1s => => exporting layers 0.1s => => writing image sha256:9bdb11df90af862a04e93887dc7ffc32f90271825a5eb5e54ac52c6b6ccdc979 0.0s => => naming to docker.io/library/iotstack-gitea_db 0.0s => [gitea_db] resolving provenance for metadata file 0.0s [+] Running 2/2 ✔ gitea_db Built 0.0s ✔ Container gitea_db Started 4.0s $ docker exec gitea_db cat /builddate.txt Thu Apr 24 01:52:29 UTC 2025 $ docker exec gitea_db iotstack_healthcheck.sh Forced change marker = 4
This time the
-d --buildresulted in the newly-built image being instantiated as the running container.
The only difference between tests 1 and 2 is COMPOSE_BAKE.
Compose Version
$ docker version -f "{{.Server.Version}}"
28.1.1
$ docker compose version
Docker Compose version v2.35.1
$ docker-compose version
Docker Compose version v2.35.1
Docker Environment
Client: Docker Engine - Community
Version: 28.1.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.35.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 8
Running: 8
Paused: 0
Stopped: 0
Images: 20
Server Version: 28.1.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: local
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.21-v8+
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.704GiB
Name: sec-dev
ID: f6406dbb-2fc1-4834-9606-5e826871aec6
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Anything else?
No response
Activity
ndeloof commentedon Apr 24, 2025
step 5 shows all build stepped resolved from CACHE, so image hasn't changed after the build.
can you pease try running
docker buildx bake -f compose.yamlin same conditions, to double check buildx detect healtcheck script has been changed and image need to be rebuilt ?Paraphraser commentedon Apr 25, 2025
I noticed that the
--buildin the presence ofCOMPOSE_BAKE=trueresulted inCACHEDbut the test sequence also shows the subsequentupwhere it is clear that the newly-built image does reflect the change to the health-check script.In other words, given that it is possible to prove that the health-check script has changed, it seems to me that any log line reporting
CACHEDis a bare-faced lie. Do you disagree?I have done as you asked but I am not sure what it has achieved.
Establish baseline conditions:
Change health-check file and prove it has changed:
Run requested command:
The
COPYandRUN datesteps aren't cached so it would seem that this process is aware of the change to the health-check script.But, wherever it is that the
buildx bakecommand wrote the image, that image isn't auto-instantiated:And, unlike the original test, the new image isn't available to
docker compose upeither:The new image doesn't even get instantiated on a re-create:
So, if I can, please, I'd really like to come back to the original point of the issue which is that, given these two assumptions:
The basic command:
The provable situation where the health-check script has changed,
then, if
COMPOSE_BAKEis:true, the build output falsely reports that the health-check script is cached, whereas a second "up" proves that the newly-built image does incorporate the change;I have no idea what
buildx bakedoes. I've never used it until today. I have no idea where it places the built image nor why that isn't available to an "up" command. I apologise if I am being obtuse but it seems to me that all this test has really shown is an awareness bybuildx bakethat the health-check script has changed.But how this helps get to the basic problem of the docker compose
--buildflag misbehaving in the presence ofCOMPOSE_BAKEis unclear. I must be missing something.ndeloof commentedon Apr 25, 2025
With
COMPOSE_BAKEvariable we prepare Docker Compose future to drop internal implementation of image builder, and fully delegate the image build process to Bake, which is dedicated to this topic.My request to run
docker buildx bake ...is to double check compose configures bake correctly as you runCOMPOSE_BAKE=true docker compose build. A surprising point is that you get a successful build, which seems to demonstrates changes have been introduced in the image (COPY iotstack_healthcheck.shis not tagged asCACHED) stilldocker compose updoesn't detect image has been updated.Can you please check image digest before and after bake build ? (
docker image inspect <image> -f '{{.ID}}'), which is used by compose to detect container must be recreated (can compare with running container labelcom.docker.compose.image)Paraphraser commentedon Apr 25, 2025
Maybe this will assist.
In words:
docker buildx bakewhich has to download the base image (ghcr.io/linuxserver/mariadb).health-check script changed.
docker buildx bakewhich should (and does) sense the change to the health-check script.Paraphraser commentedon Apr 27, 2025
@ndeloof
I've been reflecting on this but I am unable to convince myself that we're actually talking about the same thing. Please forgive me if I have misunderstood and wind up repeating the same material as earlier.
Here's what I see as the problematic command sequence:
That's the "up" command with the
--buildflag. That doesn't work. More specifically:Or, more simply, compose is ignoring the fact that it is an "up" command rather than a "build" command.
Conversely, this sequence always works reliably:
Baseline:
Interpretation:
COMPOSE_BAKEis set.Now, change the health-check script and use the "build" command (as distinct from the "up" command with the
--buildflag):Interpretation:
builddetects the change (not cached).upstarts the newly-built image.It's expected (at least by me) that a "build" command (step 3) will always need a subsequent "up" command (step 4).
Change the health-check script again and this time use the "problematic"
up --buildcommand:Interpretation:
up -d --builddetects the change (not cached).ff2e75564b2c) is still running but the "IMAGE" column has changed fromiotstack-gitea_db(tagged) to0a936ac4d372(untagged).upstarts the newly-built image. The "Container ID" changes to5460a0fce489and the "IMAGE" column returns to the tagged name.It's not expected (at least not by me) that an
up -d --build(step 3) will need another explicitup(step 5).And, for the record:
ndeloof commentedon Apr 28, 2025
It's not.
I've found a possible cause for this issue: if your service uses an explicit
imageattribute for service image to be built, looks like Compose incorrectly collects the built digest and as such won't detect image has been updated and recreate container accordingly. Proposed a fix as #12784