Repository navigation
Port conflict with multiple "host:<port range>:port" services #7188
Description
Activity
I am experiencing a similar issue that may come from the same cause.
The context is the same: upgrading to Docker Desktop 2.2.0 on MacOS breaks the port publishing when trying to run an NFS server container.
To reproduce:docker run -d --privileged --restart=always -v /exported_folder:/exported_folder -e NFS_EXPORT_DIR_1=/exported_folder -e NFS_EXPORT_DOMAIN_1=\* -e NFS_EXPORT_OPTIONS_1=rw,no_root_squash -p 111:111 -p 111:111/udp -p 2049:2049 -p 2049:2049/udp -p 32765:32765 -p 32765:32765/udp -p 32766:32766 -p 32766:32766/udp -p 32767:32767 -p 32767:32767/udp fuzzle/docker-nfs-server:latestThe error shown:
docker: Error response from daemon: driver failed programming external connectivity on endpoint <CONTAINER_NAME> (ad6b1dccbfe77e3708e687b5eed9311cf1a6f828e36c739f18f5daa5803461b7): Error starting userland proxy: listen tcp 0.0.0.0:111: bind: address already in useI have already taken down every component related to NFS on the Mac itself, and nothing is occupying that port.
The output ofsudo lsof -iTCP -sTCP:LISTEN -n -P:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME launchd 1 root 8u IPv4 0xd9679bbeeddece79 0t0 TCP *:22 (LISTEN) launchd 1 root 9u IPv6 0xd9679bbeedde7269 0t0 TCP *:22 (LISTEN) launchd 1 root 11u IPv4 0xd9679bbeeddece79 0t0 TCP *:22 (LISTEN) launchd 1 root 14u IPv6 0xd9679bbeedde7269 0t0 TCP *:22 (LISTEN) dnscrypt- 955 nobody 8u IPv4 0xd9679bbef0fb0e79 0t0 TCP 127.0.0.1:53 (LISTEN) com.docke 1190 <USERNAME> 8u IPv4 0xd9679bbef5ad2801 0t0 TCP 127.0.0.1:49273 (LISTEN)Just downgrading to Docker Desktop 2.1.x solves everything.
Does anybody have any idea for a workaround?Reacted by istincescu, rocristyc, madacrt, Radu Grigoras, Reaver Daniel, madaGrigoras27, varvaruc and Martin LyneWondering how this would've worked before, as it wouldn't be possible to have two processes listening on the same port 🤔
Reacted by Francois van der Merwe@thaJeztah should I file a separate issue, as it is not directly related to Docker compose?
Wondering how this would've worked before, as it wouldn't be possible to have two processes listening on the same port 🤔
Also when you add scale and run the command multiple times, it will create more containers than the "scale" value. For example when I ran it with --scale=3 for the first time it creates one container successfully and two containers fail with port conflicts. Second time 2 containers success 1 fails, 3rd time 2 success 1 fails. when I execute
docker psit shows 5 containers.
Note: I also have port range in my compose file. Before the 2.2.0 update It worked perfectly and created 3 containers and assigned ports within the given range.Any news on this? After upgrading docker on mac to > 2.0 my compose script using port ranges on scalable service also fails. It works perfectly on docker < 2.0.
@MartinLyne Did you find any solutions for this?
Same problem on MacOS with Docker Desktop v3.1.0 (51484) with the following docker-compose.yml:
version: '3.5' services: zookeeper: image: strimzi/kafka:0.19.0-kafka-2.5.0 command: - sh - -c - bin/zookeeper-server-start.sh config/zookeeper.properties ports: - "2181-2182:2181" environment: LOG_DIR: /tmp/logs kafka: image: strimzi/kafka:0.19.0-kafka-2.5.0 command: - sh - -c - bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT} --override num.partitions=$${KAFKA_NUM_PARTITIONS} depends_on: - zookeeper ports: - "9092-9094:9092" environment: LOG_DIR: /tmp/logs KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_NUM_PARTITIONS: 3First call fails, second call works:
# first call fails docker-compose up -d --scale kafka=3 --scale zookeeper=2 kafka Creating network "kafka_default" with the default driver WARNING: The "zookeeper" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash. Creating kafka_zookeeper_1 ... error Creating kafka_zookeeper_2 ... done ERROR: for kafka_zookeeper_1 Cannot start service zookeeper: Ports are not available: listen tcp 0.0.0.0:2181: bind: address already in use ERROR: for zookeeper Cannot start service zookeeper: Ports are not available: listen tcp 0.0.0.0:2181: bind: address already in use ERROR: Encountered errors while bringing up the project. # second call works docker-compose up -d --scale kafka=3 --scale zookeeper=2 kafka WARNING: The "zookeeper" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash. Starting kafka_zookeeper_1 ... done WARNING: The "kafka" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash. Creating kafka_kafka_1 ... done Creating kafka_kafka_2 ... done Creating kafka_kafka_3 ... doneSo I tried:
docker-compose up -d kafka && docker-compose up -d --scale kafka=3 --scale zookeeper=2 kafkaand this works as well. It seems there is a problem with initial start & scaling and the same time.Reacted by Rathgeber Stéphane, jheinzel, andriusvolkovas82, T-Berger and Vinícius AssisThe error occurs whenever a services which exposes a host port range is scaled, e. g.
version : "3" services: hello: image: tutum/hello-world ports: - "8080-8082:80"Running 3 instances of this services with
docker-compose up --scale hello=3results in a port conflict. Running this command three times resolves the problem.
I use docker version 19.03.12 and docker-compose 1.29.0 on Windows 10 (WSL 2).
Reacted by andriusvolkovas82, T-Berger and Marcus HaddonStill occurs on docker version v20.10.7, docker-compose 1.29.2 on Windows 10 (WSL 2).
Reacted by andriusvolkovas82 and Mitchell RodríguezSame issue with latest versions in Win 10 (WSL 2)
Similar issue on Docker v20.10.7 and docker-compose v2.0.0 on Windows 10 (WSL 2).
Compose File
version: "3.8" services: app: image: mishroleapp environment: MONGO_URL: "mongodb://mongodb:27017/test" depends_on: - mongodb ports: - "3000-3001:3000" mongodb: image: mongodocker-compose up[+] Running 2/4 - Network docker_default Created 0.2s - Container docker_mongodb_1 Started 5.4s - Container docker_app_2 Starting 4.9s - Container docker_app_1 Starting 4.9s Error response from daemon: Ports are not available: listen tcp 0.0.0.0:3000: bind: Solo se permite un uso de cada dirección de socket (protocolo/dirección de red/puerto)
docker-compose psNAME COMMAND SERVICE STATUS PORTS docker_app_1 "docker-entrypoint.s…" app running 0.0.0.0:3001->3000/tcp, :::3001->3000/tcp, 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp docker_app_2 "docker-entrypoint.s…" app created docker_mongodb_1 "docker-entrypoint.s…" mongodb running 27017/tcp
Updates? 😪
Same issue pulling down the most recent Confluent platform binaries:
https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html
curl --silent --output docker-compose.yml
https://raw.githubusercontent.com/confluentinc/cp-all-in-one/7.0.0-post/cp-all-in-one/docker-compose.ymldocker-compose up -d
ERROR: for zookeeper Cannot start service zookeeper: Ports are not available: listen tcp 0.0.0.0:2181: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
Issue happens with both Docker Desktop 3.0.0 / 4.2.0
Update
After upgrading and rebooting machine, was able to launch the instance. So may be a timing issue with initial startup.Here's a snippet of the yaml with the definitions:
`
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.0.0
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000broker:
image: confluentinc/cp-server:7.0.0
hostname: broker
container_name: broker
depends_on:
- zookeeper
ports:
- "9092:9092"
- "9101:9101"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
`13 remaining items
Hi any news on this issue? There are no good workarounds for us
Hi. Same problem, anything new about this?
8000-8010:80syntax already tells engine to select a random available port within the 8000-8010 range
since #10067 (v2.14.1) we start containers sequentially to workaround race condition in engine selecting such a port.
Can you please confirm this issue persists with a recent release ?I am using version 2.23.0 and the problem persists
using Docker Desktop 4.26.1 - same problem.
Any updates on that?Hello.
This might be related: we experience a
port is already allocatederror with a single port when using two configuration files.I described a simple repro here:
[BUG] Regression: array items[0,1] must be unique starting from 2.24.1 #11371
main.yml:
name: my-project services: myfirstservice: image: node:latest myservice: image: node:latest environment: - MYVAR=MyVarValue ports: - 8080:8080 links: - myfirstservice command: - node - -e - "require('http').createServer((req, res) => res.end(`Hello World! $${new Date()}`)).listen(8080);"secondary.yml:
name: my-project services: myservice: extends: file: ${PWD}/main.yml service: myserviceIf you run the command of
docker compose -f main.yml, the service will start as expected.If you run the command by using both the configuration files,
docker compose -f main.yml -f secondary.yml, you will get the error of[+] Running 1/0 ✔ Container my-project-myfirstservice-1 Created 0.0s Attaching to myfirstservice-1, myservice-1 Error response from daemon: driver failed programming external connectivity on endpoint my-project-myservice-1 (6e5afc6bd5546dc89feb0f4a019ba2f783e6b39d535fa4ed36a1eefb67664621): Bind for 0.0.0.0:8080 failed: port is already allocatedVersions used:
Docker version 25.0.2, build 29cf629 Docker Compose version v2.24.5@ErjanGavalji thanks for reporting, unrelated to original issue AFAICT, this is a parsing error fixed by compose-spec/compose-go#565. Any reason you use
extendsin your secondary file if this one is used as an override ?@ndeloof, I hope I got your question properly.
This is an attempt to provide the simplest repro, hence the service in the secondary.yml empty. In the full scenario we have volumes and env variables declared. We do not override the port or declare other ports, though.
Sure, was just wondering this was revealing some unknown use-case. But to confirm, if your
secondary.yamlis used to tweak (aka "override") the service declared in themain.yaml, you don't needextends: compose will merge the original definition with additional attributes.Reacted by Erjan Gavalji@ndeloof Thanks for the tip, had missed that!
🥇This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Description of the issue
If a compose file defines multiple services that share an overlapping port range, a port conflict occurs when
docker-compose upis executed. This behavior started to happen for me when upgrading to Docker Desktop 2.2.0 (Stable) for Mac. The version I was previously running (I believe 2.1.5) was able to select distinct ports for the two services without conflicting. Here is a POC Docker Compose file:Context information (for bug reports)
Output of
docker-compose versionOutput of
docker versionOutput of
docker-compose config(Make sure to add the relevant
-fand other flags)Steps to reproduce the issue
docker-compose upObserved result
Services fail to start and a port conflict error occurs
Expected result
Services start up. An available port within the overlapping range is selected for each service.
Stacktrace / full error message
Additional information
I uploaded diagnostic information with the ID
37C41A27-71E7-4431-AFBE-5DA0EEC74A3C/20200126225028