Repository navigation
Improve CreateDirectory on Windows #61954
Description
The CreateDirectory implementation was improved on Unix by #58799 and #61777.
The first PR eliminates exists syscalls by deducting that information from the directory creation syscall. The second PR reduces allocations when creating parent directories by allocating the parent paths on the stack instead of creating strings.
Similar improvements can be made to the Windows implementation.
Activity
danmoseley commentedon Nov 26, 2021
@iSazonov you thumbsed up this, any interest in a PR? Seems relevant to Powershell. 😸
iSazonov commentedon Nov 28, 2021
@danmoseley PowerShell MSFT team agreed that PS community will start experimental work on new File System Provider in next milestone. We already have some API enhancement requests (important for PowerShell) in .Net repository and I hope we get them early so that we have a time for adoption. I ping-ed .Net in one issue but without answer - but I hope.
As for the issue, it is nice to have but it is not so critical as issues I mentioned.
deeprobin commentedon Dec 15, 2021
Would like to have a look :)
If you want you can assign the issue to me.
I hope I can improve the performance in FileSystem.DirectoryCreation.Windows.cs
danmoseley commentedon Dec 15, 2021
@deeprobin it's yours.
deeprobin commentedon Mar 16, 2022
Unfortunately, I could not achieve any relevant performance improvement here. I think the poor performance is mainly due to the WinAPI calls.
The rest is just minimal "validation overhead". We could perhaps make minimal improvements by using a ReadOnlySpan instead of a string in Path & PathInternal.
We could also analyze whether
CreateDirectoryis the best way to create a folder.We should consider
CreateDirectoryAandCreateDirectoryW.I would look at it again this afternoon, however if anyone has any ideas please let me know.
14 remaining items
iSazonov commentedon Oct 28, 2022
By looking at this code more closely I think we can close this issue.
Obviously, creating a large number of new directories and even more so new nested directories (without having intermediate directories) are not common scenarios.
Common scenario is:
Current Windows code addresses the scenario very well. This code first checks if a directory exists and only then tries to create it. The Unix code first tries to create a directory and then checks if it exists, i.e. it makes an extra call for this general scenario. I believe we don't want follow the pattern and to do extra pinvoke on Windows in the common scenario.
The Unix code pattern excludes one more pinvoke if creating intermediate directories nut again it is one time operation and not common scenario.
Using Unix code pattern and replacing string stack with int stack makes little sense too. Today all (5 or 6) used helper methods use string type for the path parameter. And in common scenario this would exclude one allocation if terminate slash presents.
It doesn't seem to be worth the effort until we want to convert all path API operations from strings to spans.
tmds commentedon Oct 28, 2022
When I changed the Unix implementation to omit the exist checks, the rationale was that perf-sensitive code shouldn't try to create the same directory over and over again in a loop. That's what you describe here as the common scenario.
iSazonov commentedon Oct 28, 2022
I don't understand what your comment is about. If you're talking about a high-performance scenario, there's only one way --- to create a directory and block it from being deleted - only then this code can stop worrying about recreating the directory. Optimizations made earlier in CreateDirectory (or which may have been made) have no effect on this scenario, since it excludes this code altogether.
We are left with only the scenario I described. In this scenario it is rather advantageous to check the existence of the directory before trying to create it.
tmds commentedon Oct 28, 2022
I mean that if the new files created in a loop will always be written to the same directory, it's best to move directory creation out of the loop.
iSazonov commentedon Oct 28, 2022
Only if we lock the directory. Otherwise another thread/process can remove it. So having CreateDirectory in the first place is more reliable and is common practice.
tmds commentedon Oct 28, 2022
Most code doesn't account for directories that were just created go missing.
When that happens, an exception is thrown, and the operation aborts.
Moving the
CreateNewDirectoryIfNotExistin the loop doesn't make the code thread-safe. It can even cause the directory to get re-created and mask the directory was deleted while the operation was on-going.iSazonov commentedon Oct 29, 2022
You say the obvious things and again I don't understand how this relates to improving CreateDirectory on Windows requested in the issue.
I stated two things above:
if (!Directory.Exists) Directory.Create(). (I remember that we were fixing it in PowerShell code!)If, on the other hand, we put the pinvoke of directory creation first, this is only beneficial in the scenario where we need to create a lot of directories that are known to not exist. This is obviously a very rare scenario. So my preference is to have the existence check pinvoke in the first place.
So I don't see the point of int stack on Windows just for the sake of CreateDirectory.
Based on these two points, I conclude that we can close this issue.
A more promising issue should be that we need to analyze the entire Path API stack to see if it is possible to reduce the number of allocations.
Since this code is ported from the .Net Framework and saved as is for compatibility with the .Net Framework, we have hope that improvements are possible. Not an easy job, though.
tmds commentedon Oct 29, 2022
You're trying to optimize for the case ensuring directories exist, while I'm trying to optimize for the case where new directories get created. Both are valid use-cases.
NedAnd1 commentedon Aug 30, 2023
@danmoseley w/ the profiling provided by deeprobin I see room for improvement on the runtime side & in the flow of directory creation... I have some promising mockups, so I'd love to pick this up!
deeprobin commentedon Aug 31, 2023
Very good. If this is also okay on the part of the others, I am curious about your results!
A thought that still occurred to me would be perhaps to distinguish between the WinAPI calls, depending on whether the path is representable in ANSI (A call) or Unicode (W call).
But this would have to be checked before, if this brings a relevant performance boost.
NedAnd1 commentedon Aug 31, 2023
fyi .Net strings are already in the encoding used by CreateDirectoryW, plus the fact that CreateDirectoryA is a wrapper over CreateDirectoryW (converting the ANSI to Unicode under the hood) so it would only ever be slower (& the same is true at least for other file-system related win32 calls).
tmds commentedon May 16, 2025
I'm going to close this as I don't think anyone will be looking into it because there is no real perf issue to be solved for CreateDirectory on Windows.