Missing Warning Leads to Silent Data Loss When Committing Large Files (>4GB)

Existing issues matching what you're seeing

I was not able to find an open or closed issue matching what I'm seeing

Git for Windows version

git version 2.52.0.windows.1

Windows version

Windows 10

Windows CPU architecture

x86_64 (64-bit)

Additional Windows version information

Options set during installation

all default

Other interesting things

No response

Terminal/shell

cmd.exe

Commands that trigger the issue

create test.txt (>4GB)
git add .
git commit -m "test"
git ls-tree -l HEAD -- test.txt

Expected behaviour

corrent file length

Actual behaviour

file length dismatch

Repository

Git commit does not support individual files larger than 4 GB. If a file exceeds 4 GB, due to integer overflow, its length is truncated to len_new = len & 0xffffffff, and Git will only record the first len_new bytes of the file. Critically, this process produces no error or warning, making it very easy to lose content from large files.

Previously, through #3487 , Git LFS was introduced to allow large files to be stored. However, Git itself does not proactively warn users that files larger than 4 GB cannot be committed natively, leaving users unaware of which files should be submitted via LFS.

Is there any plan to fix this bug, or at least consider providing a warning to users to prevent loss of file content?

Git commit does not support individual files larger than 4 GB.

Almost no part of Git supports files bigger than 4GB on plattforms where long is 32 bit.

Previously, through #3487 , Git LFS was introduced to allow large files to be stored.

#3487 didn't introduce Git LFS, it just made it possible to store files over 4GB in Git LFS.

However, Git itself does not proactively warn users that files larger than 4 GB cannot be committed natively, leaving users unaware of which files should be submitted via LFS.

Oh, the threshold for when Git LFS should be used is significantly lower than 4GB. If we look at what GitHub allows for an example, they issue a warning that you should use Git LFS if your files exceed 50MB and reject all files over 100MB.

Is there any plan to fix this bug, or at least consider providing a warning to users to prevent loss of file content?

#2179 is the closest thing we have to a plan. And admittedly, it has seen better days.

@rimrul

Thank you for your prompt response, and Merry Christmas! @rimrul

I understand that Git is not well suited for handling large files, but if it cannot process them, it should at least issue a warning or an error instead of silently truncating the file.

GitHub and other platforms do impose limits on files larger than 100 MB, but Git itself should not rely on external services for secondary validation of its own bugs—for example, some users may only need Git for local repository management. Furthermore, if a file is 4 GB + 98 MB in size, due to an integer overflow, Git may interpret its size as only 98 MB. This size would then appear to be acceptable for upload to GitHub, inevitably leading to unexpected behavior.

Overall, this is a bug rather than a feature.

Thank you for your prompt response, and Merry Christmas

Merry Christmas.

Overall, this is a bug rather than a feature.

Yes, I fully agree.

GitHub and other platforms do impose limits on files larger than 100 MB, but Git itself should not rely on external services for secondary validation of its own bugs—for example, some users may only need Git for local repository management.

What I meant to express was, that there are other soft limits much lower than 4GB, where files over certain sizes often arent well suited for use with Git. And that those situations where Git often doesn't interact all that great with big (10s of MBs) files are both why forges like GitHub enforce such limits and why it often makes sense to use Git LFS with files significantly smaller than 4GB (as 4GB was originally mentioned as some sort of cutoff for "which files should be submitted via LFS").

I understand that Git is not well suited for handling large files, but if it cannot process them, it should at least issue a warning or an error instead of silently truncating the file.

Yes, but detecting the situation to issue the appropriate error or warning probably requires most of the work towards "just" fixing the issue, so fixing it seems like the better solution.

For what it's worth, I am slowly chipping away at the 4GB+ issues. See e.g. gitgitgadget#2102, gitgitgadget#2138 and gitgitgadget#2137. Once that work concludes, I am optimistic that this here bug will be addressed, too. Not by a warning, but by handling the data correctly.

I believe that this is done, as of v2.55.0.

Gah, it was broken again.

Gah, it was broken again.

Or not... I had only taken care of the packfiles, apparently, not the loose objects :-(