Repository navigation
FROM --platform=$BUILDPLATFORM a as b not overriding TARGETPLATFORM for multi-stage builds #4929
Description
I'm currently investing in making our builds cross-platorm compatible. I want to e.g. download the uv package manager and use it in both our build and target images. I then hit the following issue, when trying to keep our build instructions DRY:
FROM busybox as base
ARG TARGETPLATFORM
RUN echo $TARGETPLATFORM > /plt
FROM --platform=$TARGETPLATFORM base as base-target
FROM --platform=$BUILDPLATFORM base as base-buildI would expect this to build two different variants of the base stage when BUILD!=TARGET, one for each. Like this works for multi platform images pulled from the registry. But this is not the case:
#On linux/amd64:
$ docker build --platform=linux/arm64 -f test.Dockerfile --target=base-build .
...
=> CACHED [base 2/2] RUN echo linux/arm64 > /plt
...
$ docker build --platform=linux/arm64 -f test.Dockerfile --target=base-target .
...
=> CACHED [base 2/2] RUN echo linux/arm64 > /plt
...FROM --platform=$TARGETPLATFORM alpine as base-target
FROM --platform=$BUILDPLATFORM alpine as base-buildHere's the different behavior with images from the registry:
$ docker build --platform=linux/arm64 -f test.Dockerfile --target=base-build -t target .
$ docker run --rm -it target cat /etc/apk/arch
aarch64
$ docker build --platform=linux/arm64 -f test.Dockerfile --target=base-build -t build .
$ docker run --rm -it build cat /etc/apk/arch
x86_64It seems like there's no other way to get this working besides duplicating the base stage for the different platforms and giving them a unique stage name. I'm happy to hear any other workarounds.
Possibly related to
Activity
tonistiigi commentedon May 15, 2024
Yes, this is same as #4237 .
--platformis a hint for pulling the image if it is a manifest list. In your case it is a stage that is already resolved to a specific platform.Implicitly your code is:
Additionally
--platform=$BUILDPLATFORMdoes not change the meaning ofTARGETPLATFORMinside the stage.hanikesn commentedon May 16, 2024
So basically
FROM --platform=$BUILDPLATFORMdoesn't do anything for intermediate stages? I guess I agree with AkihiroSuda from the other thread that that is quite surprising and a warning would be nice. Also the docs are very lightweight on this topic.Do you think there's any other way to achieve what I want (besides duplication)?
tonistiigi commentedon May 20, 2024
It only has an effect atm if the source parameter for
FROMis multi-platform image or multi-platform build result(via named context). The stage itself can be any stage in the Dockerfile.If the commands are same for the stages defined with different platform sources(that result in same platforms based on config) then buildkit will internally merge the commands and they only run once.
huww98 commentedon May 24, 2024
@tonistiigi I think we are hitting the same issue. In my case, I want to insert an arch independent step into the build process, so that it does not need to be done multiple times for each arch.
With this Dockerfile (I'm building on an arm64 host)
I expect the build to success, but I got
I would expect the gen stage only run once on arm64, but it also runs on amd64. Any suggestions please?
hanikesn commentedon May 24, 2024
You need to do something similar to this:
huww98 commentedon May 24, 2024
I kind of understand the current behavior. But this is still confusing. Do you consider overriding the TARGETPLATFORM of stage as a valid feature request, and to be included in the future releases? If so, I can investigate more and may create PR for this.
The workaround currently requires duplicating many code in Dockerfile, or just spliting the Dockerfile into multiple Dockerfile and build them in sequence. They are not ideal.
tonistiigi commentedon May 24, 2024
Supporting
FROM --platform=.. <basestage>essentially means that the basestage needs to be evaluated multiple times.Eg.
Building this Dockerfile with default args would run
cmdtwice for both architectures (in the internal LLB graph this RUN would generate 2 different container exec steps).Same as above:
So the answer to your question depends on how much a change like that would affect the codebase and if it would increase the complexity of generating LLB graph a lot. If not then I would be ok with it. Otherwise we can add a warning to let user know that
--platformonly affects resolving multi-platform images.jberkenbilt commentedon Mar 4, 2025
In the interim, maybe the docs on FROM should be modified to mention this restriction, possibly with a link to this issue. If that's done, the issue should also include a reminder to update the docs. Before reporting docker/buildx#3034, I spent a few hours on this, and although I tried to find an existing issue, I didn't find this issue because it wasn't obvious to me that it belonged in this project. It would be nice to understand exactly what's happening with ARG and FROM and --platform. There's a section on the docs about it, but it's not that detailed. Here's a suggestion of would I would have found helpful in the docs. I'm describing it here for now rather than as a PR because I'm not sure my understanding is correct. If not, a corrected version of my understanding would be useful. I don't know anything about the internals; for example, I don't know what the LLB graph is, though I can guess because I understand that multi-platform builds create a fork in the graph of layers.
In FROM, it would be nice to see something like
as one of the choices, particularly since the link for base image doesn't contain any content.
In the bulleted list, I'd add something like this:
--platformflag is ignored when<image>is a prior build stage rather than an external image.Somewhere, there needs to be an explanation of how layers work with multi-platform builds. I don't know whether it belongs here or in some other doc, but if some other doc, there should be a link here, either in FROM, or Understand how ARG and FROM interact, or both. Perhaps there should be a link to https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/. Here are a few points that would be useful to call out explicitly. The diagrams in your blog post above make this clear.
FROM --platform=<platform>with a multi-platform builds, a version of the base image for<platform>is used for all steps on all targets, and the layers created by the steps can be parents of layers for an image for the other platforms.ARGcommand that includes one of the automaticTARGET*variables, the layer tree splits. The result is that subsequent layers for all target platforms that differ from the build platform will be from a base image of a different platform. As such, these images are not runnable (without emulation) from their target platform, and cross-compiled targets need to be copied (usingCOPY --from=...) to a new stage that originated from a multi-platform stage. (This is done in all the examples, and the diagrams show it, but I don't know that it's ever spelled out. I'm also not sure whether there are other things besidesARG TARGET*that causes the layer tree to "split". I didn't find anything that exactly specifies what causes the split.)--platform=linux/arm64,linux/amd64so you have (and here I'm looking at the image whose caption is "Sample Dockerfile commands as run on Apple M1 machine. Blue contains x86 binaries, yellow ARM." in the blog):If you create a new stage continuing from
build(after the layer tree has already split), the new stage will already be split, even if--from=<platform>is specified. (This is where several of us are saying the behavior should change or there should be a warning/error.)In this situation,
do somethingwill be run forlinux/amd64using the arm64 image that also has x86 binaries on that were previously created. (This is where we were expecting the yellow side to be the single layer.) This would result in arm binaries being run in an emulated x86 layer on an arm system. I'm not sure what this actually does....but it seems empirically that it's using two layers of emulation. Anyway, for my build, it worked, but it was very slow. It's hard to imagine that this is ever desirable behavior.So, in summary, here's what's missing from the docs:
--platform=...is ignored inFROM --platform=... <image>when<image>is a previous stage. This should possibly be accompanied by a link to this issue until either it becomes an error or the behavior is changed.ARG TARGET*splits the layer tree for a stage started withFROM --platform=...into separate layers for each target platform. (This is clear in the diagrams in the blog, but it appears to be omitted from the docs on FROM and FROM/ARG interaction. If there are other things that cause the split, state them.)ARG TARGET*(or anything else) command caused it to split, the new stage will be multi-platform. You can't create a new stage with a sequence of layers from only one of the platforms. (If this is deemed to be a bug, there should be a link to this issue, and that item should be clarified with an example when the bug is fixed.)bpaquet commentedon Jun 18, 2025
I'm also hit by this issue. The only workaround I have is to split the build in 2 and push the intermediate stage into a registry, launch a second build reusing it. This is far away of being ideal.
On the other side, may be we can fix the issue for a local multi arch build, but if we have multiple remote builder, it's a way more complicated.
@tonistiigi Is it an option to add a --remote-image in the FROM to ask buildkit to push a temporary image somewhere?
tonistiigi commentedon Jun 18, 2025
No, if we would fix it we would just make
--platformreevaluate the parent stage as well as described in #4929 (comment) .In your example case, you just need to move the
--platformto theFROM debian:bookworm-slimstage, assuming you don't use that stage for anything else.Fotoblysk commentedon Nov 2, 2025
I'm also facing this issue