Skip to content

[BUG] fatal error: concurrent map writes when pulling #12747

Closed
#12752
Closed
[BUG] fatal error: concurrent map writes when pulling#12747
#12752
Labels
kind/bug

Description

@samifruit514
samifruit514
opened on Apr 15, 2025

Description

From time to time, docker-compose up trigger errors when pulling (before bringing up the containers)
here's the error/exception:

fatal error: concurrent map writes

goroutine 116 [running]:
github.com/docker/compose/v2/pkg/compose.(*composeService).pullRequiredImages.func1.1()
        github.com/docker/compose/v2/pkg/compose/pull.go:328 +0x198
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:79 +0x54
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 114
        golang.org/x/[email protected]/errgroup/errgroup.go:76 +0x98

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x400056a058?)
        runtime/sema.go:71 +0x2c
sync.(*WaitGroup).Wait(0x40005b6d88)
        sync/waitgroup.go:118 +0x74
golang.org/x/sync/errgroup.(*Group).Wait(0x40005b6d80)
        golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x2c
github.com/docker/compose/v2/pkg/progress.RunWithStatus({0x26435c0, 0x4000834510}, 0x4000694520, 0x400070d920, {0x22653ce, 0x7})
        github.com/docker/compose/v2/pkg/progress/writer.go:97 +0x1dc
github.com/docker/compose/v2/pkg/progress.Run({0x26435c0, 0x4000834510}, 0x40005b6d40, 0x400070d920)
        github.com/docker/compose/v2/pkg/progress/writer.go:61 +0x84
github.com/docker/compose/v2/pkg/compose.(*composeService).Up(0x40004a1e80, {0x26435c0, 0x4000834510}, _, {{0x40002a1c80, {0x400049e500, 0x0, 0x2}, 0x0, 0x0, ...}, ...})
        github.com/docker/compose/v2/pkg/compose/up.go:40 +0x198

Steps To Reproduce

I dont know how to reproduce but it seems that it happens when several docker-compose on the same node pulls the same image?

Compose Version

v2.35.0

Docker Environment

Anything else?

No response

Activity

  1. samifruit514
    added
    kind/bug
    status/0-triage
    on Apr 15, 2025

  2. glours commented on Apr 16, 2025

    @glours
    glours
    on Apr 16, 2025
    Contributor

    Hey @samifruit514 👋

    Can you provide a minimal Compose file to reproducible the issue, please

  3. glours
    removed
    status/0-triage
    on Apr 16, 2025

  4. samifruit514 commented on Apr 16, 2025

    @samifruit514
    samifruit514
    on Apr 16, 2025
    Author

    Hello, Im not able to reproduce now and also cannot share sensitive info, but I can tell that its when its pulling out several images at the same time. So here is an example of docker-compose with several images:

    services:
  mysql:
    image: docker.io/mysql:8.0.32
  redis:
    image: docker.io/redis:6.0.20
  postgres:
    image: docker.io/postgres:16-alpine
  nginx:
    image: docker.io/nginx:latest
  mysql2:
    image: docker.io/mysql:8.0.32
  redis2:
    image: docker.io/redis:6.0.20
  postgres2:
    image: docker.io/postgres:16-alpine
  nginx2:
    image: docker.io/nginx:latest
  nginx3:
    image: docker.io/nginx:latest

volumes:
  mysql_data:
  postgres_data:

    and the command I run: docker compose down --rmi all && docker-compose up --pull always

    but unfortunately was not able to reproduce.. yet :(

  5. manx98 commented on Apr 20, 2025

    @manx98
    manx98
    on Apr 20, 2025

    A similar error occurred when I tried to run RAGFlow on WSL2.
    Docker Compose version v2.35.1
    RAGFlow: https://github.com/infiniflow/ragflow

    fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 47 [running]:
github.com/docker/compose/v2/pkg/compose.(*composeService).pullRequiredImages.func1.1()
        github.com/docker/compose/v2/pkg/compose/pull.go:328 +0x236
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:79 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 42
        golang.org/x/[email protected]/errgroup/errgroup.go:76 +0x96

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc000126378?)
        runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0x25c5480?)
        sync/waitgroup.go:118 +0x48
golang.org/x/sync/errgroup.(*Group).Wait(0xc0007940c0)
        golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25
github.com/docker/compose/v2/pkg/progress.RunWithStatus({0x2ba9388, 0xc00067ea50}, 0xc000782cf0, 0xc000383770, {0x27c8359, 0x7})
        github.com/docker/compose/v2/pkg/progress/writer.go:97 +0x225
github.com/docker/compose/v2/pkg/progress.Run({0x2ba9388, 0xc00067ea50}, 0xc000794080, 0xc000383770)
        github.com/docker/compose/v2/pkg/progress/writer.go:61 +0x85
github.com/docker/compose/v2/pkg/compose.(*composeService).Up(0xc0003e8780, {0x2ba9388, 0xc00067ea50}, _, {{0xc0001a7400, {0xc00040db90, 0x0, 0x1}, 0x0, 0x0, ...}, ...})
        github.com/docker/compose/v2/pkg/compose/up.go:40 +0x213
github.com/docker/compose/v2/cmd/compose.runUp({_, _}, {_, _}, {_, _}, {0x0, 0x0, {0x27c670c, 0x6}, ...}, ...)
        github.com/docker/compose/v2/cmd/compose/up.go:319 +0xb54
github.com/docker/compose/v2/cmd/compose.upCommand.func2({0x2ba9388, 0xc00067ea50}, 0xc0001b86c0, {0xc00040db90, 0x0, 0x1})
        github.com/docker/compose/v2/cmd/compose/up.go:143 +0x29f
github.com/docker/compose/v2/cmd/compose.upCommand.(*ProjectOptions).WithServices.func5({0x2ba93c0, 0xc0004020f0}, {0xc00040db90, 0x0, 0x1})
        github.com/docker/compose/v2/cmd/compose/compose.go:187 +0x22d
github.com/docker/compose/v2/cmd/compose.upCommand.(*ProjectOptions).WithServices.Adapt.func7({0x2ba93c0?, 0xc0004020f0?}, 0x2?, {0xc00040db90?, 0x2b8dfe8?, 0x11df768?})
        github.com/docker/compose/v2/cmd/compose/compose.go:137 +0x30
github.com/docker/compose/v2/cmd/compose.upCommand.(*ProjectOptions).WithServices.Adapt.AdaptCmd.func8(0xc000684908, {0xc00040db90, 0x0, 0x1})
        github.com/docker/compose/v2/cmd/compose/compose.go:121 +0x143
github.com/docker/cli/cli-plugins/plugin.RunPlugin.func1.1.2(0xc000684908, {0xc00040db90, 0x0, 0x1})
        github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:65 +0x6c
github.com/docker/compose/v2/cmd/cmdtrace.Setup.wrapRunE.func2(0xc000684908?, {0xc00040db90?, 0x0?, 0x1?})
        github.com/docker/compose/v2/cmd/cmdtrace/cmd_span.go:85 +0x63
github.com/spf13/cobra.(*Command).execute(0xc000684908, {0xc0004026d0, 0x1, 0x1})
        github.com/spf13/[email protected]/command.go:1015 +0xa94
github.com/spf13/cobra.(*Command).ExecuteC(0xc0006fef08)
        github.com/spf13/[email protected]/command.go:1148 +0x40c
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/[email protected]/command.go:1071
github.com/docker/cli/cli-plugins/plugin.RunPlugin(0xc000259040, 0xc000684608, {{0x27c5150, 0x5}, {0x27cf09e, 0xb}, {0x2b790c8, 0x7}, {0x0, 0x0}, ...})
        github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:80 +0x145
github.com/docker/cli/cli-plugins/plugin.Run(0x29213b0, {{0x27c5150, 0x5}, {0x27cf09e, 0xb}, {0x2b790c8, 0x7}, {0x0, 0x0}, {0x0, ...}})
        github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:95 +0x105
main.pluginMain()
        github.com/docker/compose/v2/cmd/main.go:38 +0xa5
main.main()
        github.com/docker/compose/v2/cmd/main.go:98 +0x19c

goroutine 23 [IO wait]:
internal/poll.runtime_pollWait(0x7fda182296b0, 0x72)
        runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0001a6380?, 0xc000017000?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001a6380, {0xc000017000, 0x1000, 0x1000})
        internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc0001a6380, {0xc000017000?, 0x0?, 0x2b805a0?})
        net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0005a4f30, {0xc000017000?, 0x0?, 0x0?})
        net/net.go:189 +0x45
net/http.(*persistConn).Read(0xc0003a6120, {0xc000017000?, 0x777f25?, 0x2379d40?})
        net/http/transport.go:2052 +0x4a
bufio.(*Reader).fill(0xc000227c20)
        bufio/bufio.go:110 +0x103
bufio.(*Reader).Peek(0xc000227c20, 0x1)
        bufio/bufio.go:148 +0x53
net/http.(*persistConn).readLoop(0xc0003a6120)
        net/http/transport.go:2205 +0x185
created by net/http.(*Transport).dialConn in goroutine 22
        net/http/transport.go:1874 +0x154f

goroutine 24 [select]:
net/http.(*persistConn).writeLoop(0xc0003a6120)
        net/http/transport.go:2519 +0xe7
created by net/http.(*Transport).dialConn in goroutine 22
        net/http/transport.go:1875 +0x15a5

goroutine 11 [select]:
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0xc00032c0a0)
        go.opentelemetry.io/otel/[email protected]/trace/batch_span_processor.go:302 +0x114
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1()
        go.opentelemetry.io/otel/[email protected]/trace/batch_span_processor.go:117 +0x4e
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor in goroutine 1
        go.opentelemetry.io/otel/[email protected]/trace/batch_span_processor.go:115 +0x2e5

goroutine 49 [syscall]:
os/signal.signal_recv()
        runtime/sigqueue.go:152 +0x29
os/signal.loop()
        os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
        os/signal/signal.go:151 +0x1f

goroutine 13 [chan receive]:
github.com/docker/compose/v2/cmd/compose.upCommand.AdaptCmd.func4.1()
        github.com/docker/compose/v2/cmd/compose/compose.go:115 +0x27
created by github.com/docker/compose/v2/cmd/compose.upCommand.AdaptCmd.func4 in goroutine 1
        github.com/docker/compose/v2/cmd/compose/compose.go:114 +0x10a

goroutine 14 [chan receive]:
github.com/docker/compose/v2/cmd/compose.upCommand.(*ProjectOptions).WithServices.Adapt.AdaptCmd.func8.1()
        github.com/docker/compose/v2/cmd/compose/compose.go:115 +0x27
created by github.com/docker/compose/v2/cmd/compose.upCommand.(*ProjectOptions).WithServices.Adapt.AdaptCmd.func8 in goroutine 1
        github.com/docker/compose/v2/cmd/compose/compose.go:114 +0x10a

goroutine 15 [select]:
github.com/docker/compose/v2/pkg/progress.(*ttyWriter).Start(0xc0001a6000, {0x2ba8f50, 0x412eac0})
        github.com/docker/compose/v2/pkg/progress/tty.go:54 +0xfa
github.com/docker/compose/v2/pkg/progress.RunWithStatus.func1()
        github.com/docker/compose/v2/pkg/progress/writer.go:83 +0x2a
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:79 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        golang.org/x/[email protected]/errgroup/errgroup.go:76 +0x96

goroutine 16 [semacquire]:
sync.runtime_Semacquire(0xc000010798?)
        runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0x25c5480?)
        sync/waitgroup.go:118 +0x48
golang.org/x/sync/errgroup.(*Group).Wait(0xc0003e9580)
        golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25
github.com/docker/compose/v2/pkg/progress.RunWithStatus({0x2ba9388, 0xc000a0d440}, 0xc000276a00, 0xc000383770, {0x27c8359, 0x7})
        github.com/docker/compose/v2/pkg/progress/writer.go:97 +0x225
github.com/docker/compose/v2/pkg/progress.Run({0x2ba9388, 0xc000a0d440}, 0xc000a0d4a0, 0xc000383770)
        github.com/docker/compose/v2/pkg/progress/writer.go:61 +0x85
github.com/docker/compose/v2/pkg/compose.(*composeService).pullRequiredImages(0xc0003e8780, {0x2ba9388, 0xc000a0d440}, 0xc0001b86c0, 0xc000780690, 0x0)
        github.com/docker/compose/v2/pkg/compose/pull.go:320 +0x2ef
github.com/docker/compose/v2/pkg/compose.(*composeService).ensureImagesExists.func1({0x2ba9388?, 0xc000a0d440?})
        github.com/docker/compose/v2/pkg/compose/build.go:278 +0x37
github.com/docker/compose/v2/pkg/compose.(*composeService).ensureImagesExists.SpanWrapFunc.func3({0x2ba9388, 0xc000780660})
        github.com/docker/compose/v2/internal/tracing/wrap.go:43 +0x13d
github.com/docker/compose/v2/pkg/compose.(*composeService).ensureImagesExists(0xc0003e8780, {0x2ba9388, 0xc000780660}, 0xc0001b86c0, 0xc0001a7400, 0x0)
        github.com/docker/compose/v2/pkg/compose/build.go:280 +0x29e
github.com/docker/compose/v2/pkg/compose.(*composeService).create(0xc0003e8780, {0x2ba9388, 0xc000780660}, 0xc0001b86c0, {0xc0001a7400, {0xc0001a6080, 0x5, 0x8}, 0x0, 0x0, ...})
        github.com/docker/compose/v2/pkg/compose/create.go:83 +0xdf
github.com/docker/compose/v2/pkg/compose.(*composeService).Up.func1({0x2ba9388, 0xc000780660})
        github.com/docker/compose/v2/pkg/compose/up.go:41 +0x85
github.com/docker/compose/v2/pkg/compose.(*composeService).Up.SpanWrapFunc.func5({0x2ba9388, 0xc000780630})
        github.com/docker/compose/v2/internal/tracing/wrap.go:43 +0x13d
github.com/docker/compose/v2/pkg/progress.Run.func1({0x2ba9388?, 0xc000780630?})
        github.com/docker/compose/v2/pkg/progress/writer.go:62 +0x22
github.com/docker/compose/v2/pkg/progress.RunWithStatus.func2()
        github.com/docker/compose/v2/pkg/progress/writer.go:90 +0x70
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:79 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        golang.org/x/[email protected]/errgroup/errgroup.go:76 +0x96

goroutine 41 [select]:
github.com/docker/compose/v2/pkg/progress.(*ttyWriter).Start(0xc000142600, {0x2ba8f50, 0x412eac0})
        github.com/docker/compose/v2/pkg/progress/tty.go:54 +0xfa
github.com/docker/compose/v2/pkg/progress.RunWithStatus.func1()
        github.com/docker/compose/v2/pkg/progress/writer.go:83 +0x2a
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:79 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 16
        golang.org/x/[email protected]/errgroup/errgroup.go:76 +0x96

goroutine 42 [semacquire]:
sync.runtime_Semacquire(0x0?)
        runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc000a0d470?)
        sync/waitgroup.go:118 +0x48
golang.org/x/sync/errgroup.(*Group).Wait(0xc0003e9600)
        golang.org/x/[email protected]/errgroup/errgroup.go:56 +0x25
github.com/docker/compose/v2/pkg/compose.(*composeService).pullRequiredImages.func1({0x2ba9388, 0xc000a0d500})
        github.com/docker/compose/v2/pkg/compose/pull.go:340 +0x39b
github.com/docker/compose/v2/pkg/progress.Run.func1({0x2ba9388?, 0xc000a0d500?})
        github.com/docker/compose/v2/pkg/progress/writer.go:62 +0x22
github.com/docker/compose/v2/pkg/progress.RunWithStatus.func2()
        github.com/docker/compose/v2/pkg/progress/writer.go:90 +0x70
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:79 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 16
        golang.org/x/[email protected]/errgroup/errgroup.go:76 +0x96

goroutine 98 [IO wait]:
internal/poll.runtime_pollWait(0x7fda18229250, 0x72)
        runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0002d2000?, 0xc0004fa000?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002d2000, {0xc0004fa000, 0x1000, 0x1000})
        internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc0002d2000, {0xc0004fa000?, 0x0?, 0x2b805a0?})
        net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000128008, {0xc0004fa000?, 0x0?, 0x0?})
        net/net.go:189 +0x45
net/http.(*persistConn).Read(0xc000034d80, {0xc0004fa000?, 0x777f25?, 0x2379d40?})
        net/http/transport.go:2052 +0x4a
bufio.(*Reader).fill(0xc0001181e0)
        bufio/bufio.go:110 +0x103
bufio.(*Reader).Peek(0xc0001181e0, 0x1)
        bufio/bufio.go:148 +0x53
net/http.(*persistConn).readLoop(0xc000034d80)
        net/http/transport.go:2205 +0x185
created by net/http.(*Transport).dialConn in goroutine 97
        net/http/transport.go:1874 +0x154f

goroutine 99 [select]:
net/http.(*persistConn).writeLoop(0xc000034d80)
        net/http/transport.go:2519 +0xe7
created by net/http.(*Transport).dialConn in goroutine 97
        net/http/transport.go:1875 +0x15a5

goroutine 45 [running]:
        goroutine running on other thread; stack unavailable
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 42
        golang.org/x/[email protected]/errgroup/errgroup.go:76 +0x96

goroutine 45 [running]:
github.com/docker/compose/v2/pkg/compose.(*composeService).pullRequiredImages.func1.1()
        github.com/docker/compose/v2/pkg/compose/pull.go:328 +0x236
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/[email protected]/errgroup/errgroup.go:79 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 42
        golang.org/x/[email protected]/errgroup/errgroup.go:76 +0x96
  6. ndeloof
    closed this as completedin #12752on Apr 22, 2025
  7. glours
    marked [BUG] concurrent map writes when running multiple containers #12780 as a duplicate of this issue on Apr 25, 2025
  8. glours
    marked [BUG] concurrent map writes on docker compose up #12787 as a duplicate of this issue on Apr 28, 2025
  9. glours
    marked [BUG] docker compose up fails with concurrent map writes #12834 as a duplicate of this issue on May 15, 2025
  10. glours
    marked [BUG] Map memory race when pulling in images #12844 as a duplicate of this issue on May 17, 2025
  11. wilkinsona
    added a commit that references this issue on May 21, 2025

    Work around thread-safety bug in Docker Compose

    0425652
  12. sdewilde
    added a commit that references this issue on Apr 15, 2026

    EBP-2030: bump docker/compose to v2.36.0 to fix concurrent map writes…

    2954903
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions