Repository navigation
[BUG] watch sync does not pass project files to container / does not reflect file changes between container restarts #11102
Description
Activity
watchindeed is just a shortcut workflow so you don't have to rebuild your container image, but this won't replace it outside single container lifecycleI don't have a problem with watch per-se, more of the
action: syncpart of the service configuration.It seems highly impractical to rebuild the container(s) every single time I bring
upthe containers just so that I get to have changes in since last build.
And I can't volume in the directory, because thenaction: sync/watchwill complain, that the directory is already in and won't work.WARN[0002] path '/path/to/project/docker-compose-watch-test' also declared by a bind mount volume, this path won't be monitored!I'm looking for a middle ground really, have
syncpull / copy the directory into the container onup, like a volume / mount would and have watch still sync in files as I change them.Reacted by Marek Pritasil, gabu, Yamato Sasaki, Dirk Heniges, TiLogic, Coroliov Oleg, conorsheehan-ow, andreichernov, Muslim Idris, Danko Petrovic and 7 moreI found a sentence from the Docker blog to be quite confusing: 'Docker Compose Watch now automatically builds and starts all required services at launch. One command is all you need:
docker compose watch.' (You can read it here: https://www.docker.com/blog/announcing-docker-compose-watch-ga-release/)However, when I use the watch command, it doesn't rebuild my container if it already exists. So, in reality, I need two commands:
compose buildandcompose watch. This is fine by me, but it's not what I expected after reading the article and the documentation.Also wording on the command itself is misleading
Usage: docker compose watch [SERVICE...] Watch build context for service and rebuild/refresh containers when files are updated Options: --dry-run Execute command in dry run mode --no-up Do not build & start services before watching <----- this suggests that containers are rebuild --quiet hide build outputThe challenge here is that we have no (simple) way to compare local files with those inside container and image used to create one. From a technical standpoint we could check all files in target using the ContainerArchiveInfo and compare with local watch source to detect a mismatch, but this would trigger hundreds API calls.
Unfortunately, I don't know enough about docker internal workings and I currently do not have the time resources to allocate to learn in order to help here in any way.
I don't technically need to "sync" only changed files, just take everything from host
pathdefinition and put it to thetargetlocation in container (I realise this is probably a gross oversimplification) and go from there. Is a comparison for every single file required?Reacted by Petr PřikrylA temporary workaround to this is to open an additional new terminal session and run
docker compose cp . app:/var/www/htmlafterwatchis running.I wanted to get a PR out for this, tho I can't quite determine why the calling
*composeService.Copy()inWatch()doesn't have the same behavior as the cli command above. The copy occurs, but the container doesn't get updated. I had hoped to do something like: if the watch config action is set to sync or sync+restart and a new watcher is created, run a copy from the watch source path to the watch target.Reacted by Thomas Emmerling, Dirk Heniges, Alex, Brennan Holzer, Ray and Hendrik MansSame problem over here. As is, the
docker compose watchfeature is a great direction but usable in most of our use cases. I followed reports and blog posts about this new feature closely and we are usingdocker composeheavily in our development team, however, it not have crossed my mind that this would not take care of syncing the initial state of the configured files and folders at container startup. My guess is that a lot of people will try to usedocker compose watchas a direct replacement for bind mounts (e.g. in OSes with inefficiencies resulting from bind mounts) and will run into this very same problem. And due to the different experience that bind mounts give, a lot of users might have a hard time tracking this down.I will try to go for a coordinated
docker compose cplike @kimdcottrell suggested - my initial tests for this looked good. Thanks!An MR with maybe a flag to decide if one wants to copy current state of the synced folder on container startup would be very nice! 👍
Reacted by Max Taggart, Petr Přikryl and yoh-000At least for me,
docker compose cpis a no go as it caused file permissions issues, like log files (in a symfony application) could not be written. Due to time constraints I haven't had the opportunity to debug why yet, just adding my two cents that it may not be as straight forward of a solution as it looks.Hey @edvordo @SystematicCZ @thomastweets @kimdcottrell 👋
Can you test this PR #11213 and let me know if this fixes, at least, the issue of the initial sync for you?I think the problem stems from the container does not
syncon the (re)start which I think it should. Why wait for a change to be made to sync only on "newly updated" (requires manualtouchif not actually updating them) files after the start? This brings a mismatch between the expectation and what actually happens.Here is why I think the current behavior is bad
I will explain with a simple scenario:
- image is built with files
a,b,c - after sometime of development, all of those file have new contents
- let's refer the updated files with
a`,b`,c` - these are all picked up while
watchwas running
- let's refer the updated files with
- however now you do
downandwatchagain and the files are back toa,b,c- and you made a change to
a`again so now it'sa``,b,cin the container - which deviated both from the original image and the state of local files
- it neither reflects the original nor the state of the local files which causes the developers to keep track of the messy state
- and you made a change to
What I think a solution should be
if
watchjust pretend (or assume) that all the files foraction: syncwas updated (mostly likely that's the case with use cases forwatch!) on start (and copy them to the right place), the state will meet the expectation and developers shouldn't manuallytouch **/*to nudge thewatchfeature.I think the current state is begging for a solution.
Reacted by Brennan Holzer, Zoli Szabó, Josh, Bruno Gardlo, Ray, Anton Bessonov, Muslim Idris, Danko Petrovic, Shang Chi Wu, Oleksandr Prypkhan and 5 more- image is built with files
The main reason is that doing so, a bunch of files will need to be copied to container on first run/restart.
But it doesn't seem we have a better way to address this(Adding on the above)
Can copying unnecessary files (that are not updated) be prevented? e.g. comparing hash (or updated date) between files and only copy the ones are actually updated.I'm not sure how feasible this is for Docker's case but I'm just sharing what that could be the potential approach to the performance/efficiency concern
19 remaining items
@jhrotko I tried using x-initialSync for traefik, but it doesn't seem to be working. I actually don't see the watch configuration in the debug logs for the traefik service. I'm assuming because traefik doesn't have a build context.
services: traefik: restart: unless-stopped image: traefik:v2.10.7 command: --providers.docker=true stop_grace_period: 1s ports: - '80:80' - '443:443' - '8080:8080' labels: - 'traefik.http.services.traefik.loadbalancer.server.port=8080' volumes: - /var/run/docker.sock:/var/run/docker.sock:ro develop: watch: - action: sync+restart x-initialSync: true path: ./traefik target: /etc/traefik - action: sync+restart x-initialSync: true path: ./certs target: /etc/certs
docker compose version Docker Compose version v2.29.2-desktop.2
Edit: Not urgent since I believe traefik watches its on config and would restart using volumes
@tjhiggins yes, you need to create a
DockerfileFROM traefik:v2.10.7and add a a build section in your service
services: traefik: build: . ...@tjhiggins
watchis designed to give you a faster development loop, but application should be able to run with required files with a plaindocker compose up. In your case, a possible workaround is to add a build declaration to copy required files, at least for first run:services: traefik: build: dockerfile-inline: | FROM: traefik:v2.10.7 COPY ./traefik /etc/traefik develop: watch: - action: sync+restart path: ./traefik target: /etc/traefik
Reacted by TiLogicClosing this issue as we let users experiment with
x-initialSyncand collect feedback so we decide how to best address this need- added a commit that references this issue
on Mar 3, 2025 Chiming in so you can collect the feedback (is there a better place than here?).
x-initialSyncworks really well if the image contains the files we want to watch (thanks!), but it doesn't work if the image never included the files to begin with, which makes this not a viable alternative to mounting volumes.Either of these would solve this case:
- Allow
x-initialSyncor another config to copy all path files into the target directory on startup (a more expensive operation, but creates a predictable and reliable state on startup that we can rely on). - Alternatively, if it were possible to watch a host machine directory that is being mounted, we could copy files on startup ourselves with
rsync, and let watch take over from there.
The reason I'm excited about watching > mounting is because it seems like less maintenance compared to remembering to masking off all of the node_module folders across a monorepo by mounting named volumes on top of each of them, if we can use the ignore configuration to prevent syncing those folders in one go.
Reacted by Nish Sinha- Allow
@yoh-000 I just found this issue today because I was fed up with watch not picking up files that were changed while it wasn't running. Adding
x-initialSyncto my project has that working now. I can add new files to the host directory while the container is stopped and they show up when I bring it up.I'm using
Docker Compose version v2.36.0-desktop.1Reacted by Michał Pniakwhat @tcarterBAMF said works for me but the reverse doesn't work, if you delete a file while the containers are down and then start compose with watch, the file isn't removed.
Noting here for posterity: experimental
x-initialSyncis now deprecated asinitial_synchas been promoted to stable via #13232
Description
I've wanted to try the newest
docker compose watch(released with latest desktop), but I have two issue with it.The announcement blog post seems to suggest, that adding a config such as this one in an example repository and running
docker compose watchshould result in a container namedappand the folder/var/www/htmlshould have the contents of the current directory.Thing is, the directory is empty and only new changes to files are copied over.
This lead me to believe, that I would need to add
COPY . /var/www/htmltoDockerfileso files are copied in on image build, but this creates a second problem, as any changes I made to project files since image build are not reflected to the container between container restarts (say across multiple days of development).Anyway, here is a repository with an example setup, including steps to reproduce
https://github.com/edvordo/docker-compose-watch-test
I'm more than willing to concede I'm reading the docs wrong, but I've tried support channels like the (unofficial?) discord server and we could not figure out what could be wrong.
The issue is the same using the official examples provided here:
https://github.com/dockersamples/avatars
While yes, the containers will have content after first build (thanks to the
COPYinstruction), they will not have the changes made to the project files between container restarts:<title>inweb/index.htmlto whateverlocalhost:5735and in containerwatch, rundocker compose downand rundocker compose watchagain<title>in browser / container will NOT have your changesSteps To Reproduce
No response
Compose Version
Docker Environment
Anything else?
No response