Skip to content

node --watch doesn't work well within a container #47061

Closed as not planned
Closed as not planned
node --watch doesn't work well within a container#47061
Labels
invalidkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/24.0

Description

@ndeloof
ndeloof
opened on Jan 11, 2024

Description

node --watch only detect first change in a file when overridden by cp command

Reproduce

  1. docker run --rm -i ndeloof/test:debug
  2. in a second terminal, exec inside container
  3. run cp src/index.mjs test && cp test src/index.mjs
  4. check in first terminal file change has been detected by node, logging "Restarting 'src/index.mjs'"
  5. run cp test src/index.mjs again.
  6. nodejs doesn't report "Restarting 'src/index.mjs'" as expected

Expected behavior

nodejs detect file changes on each and every subsequent change in file

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.25.2 (129061)
 Engine:
  Version:          master
  API version:      1.44 (minimum version 1.12)
  Go version:       go1.21.3
  Git commit:       0253fedf03f4964c20906795e119404633cb9c1a
  Built:            Thu Oct 19 08:35:54 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/nicolas/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.0-birthday.10-35-g2bf2b22fb.m
    Path:     /Users/nicolas/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/nicolas/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/nicolas/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.9
    Path:     /Users/nicolas/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/nicolas/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/nicolas/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.9
    Path:     /Users/nicolas/.docker/cli-plugins/docker-scout

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 25
 Server Version: master
 Storage Driver: stargz
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.661GiB
 Name: linuxkit-86a327832005
 ID: 96013128-e875-4a8d-8e0e-d242b089c2f0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Additional Info

I used strace to check how doing a cp inside container is implemented at system level:

execve("/bin/cp", ["cp", "test", "src/index.mjs"], 0xffffcfdaefa0 /* 9 vars */) = 0
set_tid_address(0xffffabefde58)         = 18
brk(NULL)                               = 0xaaaaf3c9a000
brk(0xaaaaf3c9c000)                     = 0xaaaaf3c9c000
mmap(0xaaaaf3c9a000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xaaaaf3c9a000
mprotect(0xaaaad462c000, 16384, PROT_READ) = 0
getuid()                                = 0
newfstatat(AT_FDCWD, "test", {st_mode=S_IFREG|0644, st_size=189, ...}, 0) = 0
newfstatat(AT_FDCWD, "src/index.mjs", {st_mode=S_IFREG|0644, st_size=189, ...}, 0) = 0
newfstatat(AT_FDCWD, "test", {st_mode=S_IFREG|0644, st_size=189, ...}, 0) = 0
newfstatat(AT_FDCWD, "src/index.mjs", {st_mode=S_IFREG|0644, st_size=189, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "test", O_RDONLY|O_LARGEFILE) = 3
openat(AT_FDCWD, "src/index.mjs", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0100644) = -1 EEXIST (File exists)
unlinkat(AT_FDCWD, "src/index.mjs", 0)  = 0
openat(AT_FDCWD, "src/index.mjs", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0100644) = 4
sendfile(4, 3, NULL, 16777216)          = 189
sendfile(4, 3, NULL, 16777216)          = 0
close(4)                                = 0
close(3)                                = 0
exit_group(0)                           = ?

Note the unlinkat then openat calls with O_CREAT, which basically means file is recreated. Seems overlayfs doesn't work well with inotify, which expect a fixed inode for monitored file. Not sure about a possible fix/workaround

Activity

  1. ndeloof
    added
    kind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
    status/0-triage
    on Jan 11, 2024

  2. thaJeztah commented on Jan 11, 2024

    @thaJeztah
    thaJeztah
    on Jan 11, 2024
    Member

    A workaround could be to use an (anonymous) volume for these files; in that case, those files would be stored in a volume, so not depending on overlay-filesystem (in addition to dodging the overhead of CoW filesystems), but I guess that's not a solution for all situations.

  3. thaJeztah
    added
    area/storage/overlay
    version/24.0
    on Jan 11, 2024

  4. corhere commented on Jan 16, 2024

    @corhere
    corhere
    on Jan 16, 2024
    Contributor

    Do we have any evidence that node --watch works well outside a container? Unlink followed by open will generally change the inode for the path unless you get lucky and the just-freed inode number gets reused for the new file. It's a generalization of the atomic-replace idiom (write to temp file, rename over top of old file) which guarantees that the inode for the path changes. This smells like a node.js bug; it should be watching the directory inode in addition to the file inode to reliably detect file modifications after unlink or atomic-replace. Note that node --watch is marked Experimental.

    Unless you can demonstrate that node --watch works reliably with replaced files under a tmpfs mount, this is not a Moby issue.

  5. ndeloof commented on Jan 17, 2024

    @ndeloof
    ndeloof
    on Jan 17, 2024
    ContributorAuthor

    I don't have such evidence, but the scenario I documented here can be reproduced in a moby container and such issue doesn't take place on a plain Linux hosts. Maybe that's a node fs.Event issue, or an actual overlayfs limitation Moby can't workaround.

  6. corhere commented on Jan 17, 2024

    @corhere
    corhere
    on Jan 17, 2024
    Contributor

    It's an fs.Event issue. I was able to reproduce the issue under a tmpfs with mv.

  7. corhere
    closed this as not plannedon Jan 17, 2024
  8. corhere
    added
    invalid
    and removed
    status/0-triage
    area/storage/overlay
    on Jan 17, 2024

  9. mbrevda commented on Jan 17, 2024

    @mbrevda
    mbrevda
    on Jan 17, 2024 · Hidden as off-topic
  10. moby
    locked as off topic and limited conversation to collaborators on Jan 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    invalidkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/24.0

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions