Repository navigation
support file block copy on write on Windows #86681
Description
On ReFS, Windows supports a block copy operation, like copy on write but at a sub-file level.
https://learn.microsoft.com/en-us/windows/win32/fileio/block-cloning
https://learn.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_duplicate_extents_to_file
This has existed for several years, but is suddenly more interesting now because of the announcement of dev drive.
We should enlighten File.Copy to use this on Windows when possible, so that MSBuild, Nuget etc work extra fast on dev drive.
Activity
danmoseley commentedon May 24, 2023
@hamarb123 if you comment on this issue, I'll be able to assign it to you.
hamarb123 commentedon May 24, 2023
Comment :)
danmoseley commentedon May 24, 2023
cc @rainersigwald this should speed up MSBuild if it isn't already doing hardlinks.
cc @aortiz-msft this should help Nuget, but only where it's doing File.Copy or equivalent -- it looks like you do FileStream copy in some places which wouldn't benefit .. we might want to take a look at that at some point to see whether it's necessary.
rainersigwald commentedon May 24, 2023
MSBuild has an option to do hardlinks (the
Copytask'sUseHardlinksIfPossibleparameter, exposed in common targets as$(CreateHardLinksForCopyFilesToOutputDirectoryIfPossible)) but you have to be very careful with them since modifying (what looks like) a file in your output can silently corrupt your NuGet cache or other files, so they're not widely used.danmoseley commentedon May 24, 2023
sharing today's post for context -- https://devblogs.microsoft.com/engineering-at-microsoft/dev-drive-and-copy-on-write-for-developer-performance/
jkoritzinsky commentedon May 25, 2023
There's a Copy On Write MSBuild SDK that utilizes the ReFS CloneFile API for a custom implementation of the Copy MSBuild task. Might be useful.
danmoseley commentedon May 25, 2023
Yep, I think @hamarb123 is going to crib from that (https://github.com/microsoft/CopyOnWrite)
hamarb123 commentedon Jun 21, 2023
I have been delayed in making a PR for this due to BSODs caused by my code/Windows, which were hard to track down. I've now found the root cause, and documented it at: microsoft/CopyOnWrite#24. Hopefully I can make a PR with a workaround soon.
13 remaining items
huoyaoyuan commentedon Oct 26, 2023
It sounds like a new syscall available?
erikmav commentedon Oct 26, 2023
Not a new syscall, the Windows team liked the CoW approach and numbers from the .NET approach (https://aka.ms/EngMSDevDrive) and I've been working with them to turn it on by default in the
CopyFile(Ex)API calls. Unless something goes wrong between now and release of this Canary feature to cause the code to change, on Dev Drive (and ReFS) copying a file will just clone instead.I'm partway through perf testing the canary bits on internal repos with and without the .NET package turned on. Keep an eye on the Eng@MS blog for more info as soon as next week.
erikmav commentedon Oct 26, 2023
Assuming the feature makes it to the February release without semantic changes, it would resolve this issue. Better to wait to resolve this until it's known that clone is on by default in the final release.
erikmav commentedon Nov 7, 2023
FYI latest in blog series (encapsulates what's already here): https://devblogs.microsoft.com/engineering-at-microsoft/copy-on-write-in-win32-api-early-access/
huoyaoyuan commentedon Oct 8, 2024
I updated my machine to 24H2 recently, and the plain copy command seems to be CoW cloning on regular ReFS drive (not Dev Drive). Doing it in the syscall itself would probably have better performance.
Do we still need this functionality? It will only benefit a limited set of configurations, and not required for "the future".
hamarb123 commentedon Oct 8, 2024
No, I don't think so - I assumed the conclusion was that we will just benefit from OS support when available, and not worry about the other cases, since they're rare anyway.
If it's still wanted, I could have a go at implementing it some time, but I had just assumed it wasn't needed anymore.
danmoseley commentedon Mar 18, 2026
As mentioned above (sorry I missed it), this can be closed as it seems modern Windows versions give it for free (on dev drive/ReFS)
built into CopyFile APIs (Nov 2023)
perf data
Original Dev Drive + CoW story
danmoseley commentedon Mar 18, 2026
fyi @adamsitnik in case you weren't aware ^^
KalleOlaviNiemitalo commentedon Mar 18, 2026
That is not provided by the CopyFile API of Windows but could still be implemented in .NET by making FileStream override CopyTo(Stream, Int32) and detect if the other Stream is also a FileStream.