Container needlessly recreated and then won't start on docker-compose up

Description

Following a change introduced in 2.3.4, most likely #9261, we're hitting a scenario where a container is recreated when it needn't be, and then will fail to start following that

Steps to reproduce the issue:

Create a docker-compose.yml similar to the below:

version: '3.7'
services:
  db:
    image: postgres:11
    hostname: postgres
    container_name: engage-db
    ports:
      - "5437:5432"
    environment:
      - POSTGRES_DB=somedb
      - POSTGRES_USER=someuser
      - POSTGRES_PASSWORD=somepassword

  wait-for-db:
    image: postgres:11
    command: [ sh, -c, "until pg_isready --username=someuser --host=db --port=5432; do sleep 5; done" ]
    links:
      - db

Ensure postgres:11 is not already in the docker image cache
Run docker-compose up -d db to start the db in the background, this will also pull the postgres:11 image.
Run docker-compose up wait-for-db to start the wait in the foreground

Describe the results you received:
When the last step is run to start the wait-for-db container, it will result in the db container being recreated, despite the image etc. not having since been rebuilt or changed in any way:

$ docker-compose up wait-for-db
[+] Running 2/2
 ⠿ Container engage-db             Recreated                                                                                                     0.3s
 ⠿ Container engage-wait-for-db-1  Created                                                                                                       0.1s
Attaching to engage-wait-for-db-1

Following this, the recreated db container will also not be running, it will be sat in a Created state:

125fee896799   postgres:11   "docker-entrypoint.s…"   About a minute ago   Created                                engage-db

$docker inspect 125fee896799

``` [ { "Id": "125fee896799d4367c6ccae891065e120ded3ad6ccc8c525a45253abd5c88c27", "Created": "2022-04-06T10:05:51.95171076Z", "Path": "docker-entrypoint.sh", "Args": [ "postgres" ], "State": { "Status": "created", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 0, "Error": "", "StartedAt": "0001-01-01T00:00:00Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:04a660188f48f78bc42b2a03597123f9b0504f2db039daf8f2d98f93b04c2711", "ResolvConfPath": "", "HostnamePath": "", "HostsPath": "", "LogPath": "", "Name": "/engage-db", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "engage_default", "PortBindings": { "5432/tcp": [ { "HostIp": "", "HostPort": "5437" } ] }, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "CgroupnsMode": "private", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "Mounts": [ { "Type": "volume", "Source": "26b0d26dc817cc36f1ad1290dfc27f1bc9984a9c5796a948eb3d6c9f27b2abe6", "Target": "/var/lib/postgresql/data" } ], "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/10d96b1e87acb2e1617cc0c714fda6dfc727806d697033bd961637a3e372758d-init/diff:/var/lib/docker/overlay2/0d6e1a46381ea25e8cf96a9106f039946dd3e4427627e6cb1083005c718675e9/diff:/var/lib/docker/overlay2/4477f40fb0f241080e56a5450f965526ef4d7db71d62d81027cb1910b2e00f9d/diff:/var/lib/docker/overlay2/8bb3fbfc8a6bad7774fcac5117745a279a3d93a40cd8b1c4d577c34e2e9e5963/diff:/var/lib/docker/overlay2/112cd7cff09b5078cad860b6ae71dc9fb37e351656a229ac79f5ca68d0a2344b/diff:/var/lib/docker/overlay2/f6eb995351d9cbf225916217b20936bff5d4206931cddb1b9046cfeb22624ac7/diff:/var/lib/docker/overlay2/f5816c70b0e48705ce9827561cb91b99d67bb1671a341eb4101e329f780a13ba/diff:/var/lib/docker/overlay2/12453a6429f68bfe21e3b4f21b9febfb8117a896210dd2e29492001e5dc0bf1b/diff:/var/lib/docker/overlay2/42badba2cfaf487b90abdfc0417f3f814c0909f1b35242f0163a3535ccc23f8b/diff:/var/lib/docker/overlay2/ae6ff9f97ba34f0b9964492930f1e800e78ea65883e1b53d26e1136762895fde/diff:/var/lib/docker/overlay2/0f426746f21a5c0ddc48638a6b341190b522fce06b5b8f446760f33180556818/diff:/var/lib/docker/overlay2/fa6edca9880603046c4c5e5b65bbe59577cb7f6ec14264c81cf50c70600108bf/diff:/var/lib/docker/overlay2/9d45d8aedf1e0fcbe07c0c87822f8568c91d35f0dbb65e6cade336b000273725/diff:/var/lib/docker/overlay2/74b44bb155159c41af7b7ec4fad504f6753c0666d191bf2f4f234485e37d82b1/diff", "MergedDir": "/var/lib/docker/overlay2/10d96b1e87acb2e1617cc0c714fda6dfc727806d697033bd961637a3e372758d/merged", "UpperDir": "/var/lib/docker/overlay2/10d96b1e87acb2e1617cc0c714fda6dfc727806d697033bd961637a3e372758d/diff", "WorkDir": "/var/lib/docker/overlay2/10d96b1e87acb2e1617cc0c714fda6dfc727806d697033bd961637a3e372758d/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "volume", "Name": "26b0d26dc817cc36f1ad1290dfc27f1bc9984a9c5796a948eb3d6c9f27b2abe6", "Source": "/var/lib/docker/volumes/26b0d26dc817cc36f1ad1290dfc27f1bc9984a9c5796a948eb3d6c9f27b2abe6/_data", "Destination": "/var/lib/postgresql/data", "Driver": "local", "Mode": "z", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "postgres", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "ExposedPorts": { "5432/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "POSTGRES_PASSWORD=somepassword", "POSTGRES_DB=somedb", "POSTGRES_USER=someuser", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/11/bin", "GOSU_VERSION=1.14", "LANG=en_US.utf8", "PG_MAJOR=11", "PG_VERSION=11.15-1.pgdg90+1", "PGDATA=/var/lib/postgresql/data" ], "Cmd": [ "postgres" ], "Image": "postgres:11", "Volumes": { "/var/lib/postgresql/data": {} }, "WorkingDir": "", "Entrypoint": [ "docker-entrypoint.sh" ], "OnBuild": null, "Labels": { "com.docker.compose.config-hash": "2cfb5e6a3de7283e5cf2ef6ff09ada28fbdc8c9447ae459da4f7fa2c00c1ec6c", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:04a660188f48f78bc42b2a03597123f9b0504f2db039daf8f2d98f93b04c2711", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "engage", "com.docker.compose.project.config_files": "/Users/Michael.Eves/Workspace/Docker/engage/docker-compose.yml", "com.docker.compose.project.working_dir": "/Users/Michael.Eves/Workspace/Docker/engage", "com.docker.compose.service": "db", "com.docker.compose.version": "2.3.4" }, "StopSignal": "SIGINT" }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "engage_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "engage-db", "db", "125fee896799", "postgres" ], "NetworkID": "", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "DriverOpts": null } } } } ] ```

If you stop everything docker-compose down -v and try again now the image postgres:11 is already pulled, the container is not recreated and everything works as expected.

Describe the results you expected:
The container should not be recreated as nothing has changed about the image. Adding --no-recreate on the second compose command also works around the issue, but I don't believe the behaviour is correct currently on two points:

The container shouldn't be recreated
If a container is recreated, it should start back up

Output of docker compose version: Happens from vesion 2.3.4 inc. current release 2.4.1.

Output of docker info: 20.10.11/20.10.8

root cause is: as service image is pulled during up, com.docker.compose.image label is not set

@ndeloof

@ndeloof Your fix from #9366 was included in v2.5.0, but I can still reproduce this issue (com.docker.compose.image is unset after initial up) with docker compose v2.6.0. This issue should be re-opened.

Indeed this bug is still reproducible in v2.10.1 (using docker:22.06.0-beta.0-dind).

@ndeloof

@ndeloof @ulyssessouza Could we please re-open this? It was not fixed by #9366.

I also faced this issue.
I'm using Docker Compose version v2.7.0.
When I start my app with around 15 containers, just the mysql container is randomly recreated.

docker inspect dreg.***.io/dhub/library/mysql:5.7.32
[
    {
        "Id": "sha256:cc8775c0fe94cd6cbfc494688d09b078cda9bced9d9324d5c28bddf032344fc3",
        "RepoTags": [
            "dreg.***.io/dhub/library/mysql:5.7.32"
        ],
        "RepoDigests": [
            "dreg.***.io/dhub/library/mysql@sha256:e08834258fcc0efd01df358222333919df53d4a0d9b2a54da05b204b822e3b7b"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2021-01-12T10:12:44.131914824Z",
        "Container": "56ee61b1be5c0c698c895c3990f655333ec10c3fc44471b1b97ec88f1b49e6f0",
        "ContainerConfig": {
            "Hostname": "56ee61b1be5c",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306/tcp": {},
                "33060/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.12",
                "MYSQL_MAJOR=5.7",
                "MYSQL_VERSION=5.7.32-1debian10"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"mysqld\"]"
            ],
            "Image": "sha256:18ad8b374d80c80c3473ea052bf5dbf440ae0808066c0be850270dde54020746",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "19.03.12",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306/tcp": {},
                "33060/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.12",
                "MYSQL_MAJOR=5.7",
                "MYSQL_VERSION=5.7.32-1debian10"
            ],
            "Cmd": [
                "mysqld"
            ],
            "Image": "sha256:18ad8b374d80c80c3473ea052bf5dbf440ae0808066c0be850270dde54020746",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 448500822,
        "VirtualSize": 448500822,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/1d325c335062c051ae7268095e73680256061c3a690be2cf643a834f3a3f8f78/diff:/var/lib/docker/overlay2/e2eb630a87fc15ec4016b2fc66e336cd44993e662074095c9beb7cd88ae6f7f3/diff:/var/lib/docker/overlay2/d1438d837b19cab5d300131531327916c693de57b285aa7d161ed30f4bc39602/diff:/var/lib/docker/overlay2/a2f6aeda01e6a18ed1c9de625b81754d155e453f6404f37dbd1373f23b45c34d/diff:/var/lib/docker/overlay2/9649c7eb9315d3a0c5c9f2ea41d4677683715924459a69359919185b4920b37d/diff:/var/lib/docker/overlay2/cb7837f1a6a77992faee02b65080728b73a73a639b96aa254ab1383b51d866f1/diff:/var/lib/docker/overlay2/667070aeec58e97fff2cb8b20f3ab39238d52ccff7f84ef619fede73f3064b73/diff:/var/lib/docker/overlay2/44f7fc51144d497cd2933f4e1eba1c4a935cd60ff69215b8c0ff31093ba8b620/diff:/var/lib/docker/overlay2/b6871b7d5e24b1d415327330c6a5ac00267d3d9c1595430513126b34cd71723d/diff:/var/lib/docker/overlay2/79c371a05c2de598582db73608d65446660219c67c76ec466ce4a39a17f9709d/diff",
                "MergedDir": "/var/lib/docker/overlay2/3b920c483e4b0f94d0293df4ed15662a3e596415c62fbfa761ceac68ac911de5/merged",
                "UpperDir": "/var/lib/docker/overlay2/3b920c483e4b0f94d0293df4ed15662a3e596415c62fbfa761ceac68ac911de5/diff",
                "WorkDir": "/var/lib/docker/overlay2/3b920c483e4b0f94d0293df4ed15662a3e596415c62fbfa761ceac68ac911de5/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:cb42413394c4059335228c137fe884ff3ab8946a014014309676c25e3ac86864",
                "sha256:ef4a33cee7a02ffa3c1dd5d2618dcb95e37b899fb7e90d0bb2df32a37396e174",
                "sha256:74c86dffd46f095d4961730b731a33b19c1abd19f0a7ec1b81ddf09e6d3dfe85",
                "sha256:6d23902c2a54f36f80946d7225044b2681bac18570af3a58970e325b40ac0d60",
                "sha256:c484a3b6d84133f3618780f11216facc49f743cefc86e04409f28784a3a0a733",
                "sha256:0394a41efa739604258181808606eb47798a83383ff52241547583939ffb297c",
                "sha256:98d98806c8acefda16226c37045e3733a751c3cb165fde1182cbc324cde06f78",
                "sha256:8bdb6ee41f57372d9290e64d3de05a54f0f95cdd168dab725e39e96d9387cff3",
                "sha256:569a2d6e5302878870ada8b70e77a218ebad0659ad7d0d068187c1d320b15cea",
                "sha256:902dedce4cb4de96b4037a07e851f3f8b40c6c0ebc7172923cd58aa229a95862",
                "sha256:df1ee06a30c4b4463fc9f6ab182f6622f1b83b0e31f73468ce598fedc8b12564"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

i have the same issue on dc v2.6.1

Docker Compose version v2.12.2 the issue is still represented

Seeing this issue on v2.14.0 too - any word on a fix? We have a use cae where we need to gradually add some services to the compose file and start them only after confirming the previous service is happy and healthy.

@captainfalcon23

Tried to reproduce on latest codebase:

$ docker-compose --ansi=never up -d db
db Pulling 
45f14b91a40f Pulling fs layer 
...
0b9bf888f4ce Pull complete 
db Pulled 
Container engage-db  Creating
Container engage-db  Created
Container engage-db  Starting
Container engage-db  Started

$ docker-compose --ansi=never up wait-for-db
Container engage-db  Running
Container truc-wait-for-db-1  Creating
Container truc-wait-for-db-1  Created
Attaching to truc-wait-for-db-1
truc-wait-for-db-1  | db:5432 - accepting connections
truc-wait-for-db-1 exited with code 0

@captainfalcon23 you could use depends_on with a service_healthy condition for this use-case

Let's close this one as reproduction scenario doesn't help anymore and follow up on this with #9600

Navigation Menu

Repository navigation

Container needlessly recreated and then won't start on docker-compose up #9357

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Container needlessly recreated and then won't start on docker-compose up #9357

Description

Activity

ndeloof commented on Apr 7, 2022

benlangfeld commented on Aug 25, 2022

benlangfeld commented on Aug 25, 2022

benlangfeld commented on Aug 30, 2022

bibendi commented on Aug 30, 2022

xyauheni-zededa commented on Sep 14, 2022

g-lashuk commented on Nov 14, 2022

captainfalcon23 commented on Dec 20, 2022

ndeloof commented on Dec 20, 2022

ndeloof commented on Dec 20, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions