Issue 365 in msysgit: Out of memory? mmap failed

[msy...@googlecode.com]

Status: New
Owner: ----

New issue 365 by bannii: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

What steps will reproduce the problem?
1. Clonsed a large git archive (22000 files / 6GB)
2. fatal: Out of memory? mmap failed: No such file or directory ---- at end
of checkout-process

And there is no checked out files there. Just the ".git" directory with
1.6GB.
If i want to "git add ." or "git status" the directory, it reproduces the
same error message.

On smaler archives, there is no problem with this issue.

Using Windows 7 on a Lenovo Ideapad S12 with 2GB of RAM and an Atom N270.

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings

[msy...@googlecode.com]

Comment #1 on issue 365 by patthoyts: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

I wonder if you look in .git/objects/pack if any of the pack files are
larger than 2GB?
I don't know how these files are accessed in msysGit but you cannot memory
map >2GB
in 32bit windows because you run out of address space. If the msysGit
mapping code
does not map a suitable window but just maps the whole file then there is
some work
to be done to handle such large pack files.

[msy...@googlecode.com]

Comment #2 on issue 365 by johanthart: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

If I remember correctly, git reads objects (file-ish, tree-ish etc)
entirely in
memory for performance reasons. I don't know what it does with packs, but I
cannot
imagine an entire pack-file is read into memory... Anyone?

If a file-ish gets >2GB, git on 32 bit cannot process it.. I don't know how
much
memory can be mapped in one block on a Ideapad with 2GB ram..?

[msy...@googlecode.com]

Comment #3 on issue 365 by bannii: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Is there a possibility to split the large object-file up to some smaller
files
without creating a new repository?

The first check-in indeed was very big (around 4GB)

[msy...@googlecode.com]

Comment #4 on issue 365 by johanthart: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

I think it was not about complete commits but about individual objects. So
is there a
file in that first 4GB commit which was very big? (+1GB or so)

[msy...@googlecode.com]

Comment #5 on issue 365 by alexander.jaehrling: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Hi, we have the same problem here. Large repository (4.13GiB/876642
objects). After
cloning there is only a single 4.2G pack file. The clone already failed
with:

fatal: Out of memory? mmap failed: No such file or directory

The single files are mostly small a few are up to 200MB in size.

We tried to re-compress it with a smaller pack size, but this was only
possible using
git with Cygwin. After this it also worked with msysgit.

[msy...@googlecode.com]

Comment #6 on issue 365 by alexander.jaehrling: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

btw. it seems that the pack.packsizelimit option is ignored by msysgit
which IMHO might
cause this issue.

[msy...@googlecode.com]

Comment #7 on issue 365 by ian.mclean: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Fix attached:
Fix for issue 365: "Out of memory? mmap failed"

The git_mmap implementation was broken for file sizes that wouldn't fit
into a
size_t (32 bits). Changed 'len' to be stored in an off_t (64 bit)
variable, and
moved the xsize_t cast to only occur when an attempt is made to map past
the end-of-file.

Attachments:
commit-a52ec2b 1.1 KB

[Janos Laube]

hi,

thanks for caring. did the testsuite pass with that patch applied?
while i'm fine with storing the filesize in an 64bit variable, i found
that we can't trust the output of fstat for files larger than 2gb. can
you verify this please? if it's working i'm fine with it. on a side
note, please post your patches on the mailinglist inlined if possible,
that makes reviewing and commenting easier. the issue tracker is good
for having all bugs and issues at one place, but most don't like to
work with it :-)

janos

[msy...@googlecode.com]

Comment #8 on issue 365 by johannes.schindelin: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Unfortunately, this patch is not uncontroversial. And even more
unfortunately, you chose
a particularly unfit procedure to offer the patch for discussion.
The "length" variable
is still a size_t, and if that is different than off_t, the parameter of
mmap() cannot
be a size_t.

Oh, and I am _very_ unhappy that you forced this discussion into the issue
tracker. Very.

[msy...@googlecode.com]

Comment #9 on issue 365 by kusmabite: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

In other words: please send the patch to the mailing list so we can discuss
it, Ian.

[msy...@googlecode.com]

Comment #10 on issue 365 by john.bauman: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

This patch seems to fix my issues with git checkout of a large repository
(and I
can't see how it would be worse than what's currently there), but I get
this error
with "git reset --hard":

fatal: Out of memory? mmap failed: No error
fatal: Could not reset index file to revision 'HEAD'.

I'm not sure if that is related to this bug, but doing the same command
from a linux
system (over CIFS) works.

[msy...@googlecode.com]

Comment #11 on issue 365 by vitaly.berov: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Any progress with this bug? We have the same issue on the latest version
(Git-1.7.0.2-
preview20100309.exe).

[msy...@googlecode.com]

Comment #12 on issue 365 by johannes.schindelin: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

vitaly.berov: as mentioned all over the place, msysGit is driven forward by
volunteers. If you want the issue to be worked on, you should do so
yourself, or at
least entice others to work on it.

[msy...@googlecode.com]

Updates:
Status: SEP

Comment #13 on issue 365 by johannes.schindelin: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Changing the status accordingly.

[msy...@googlecode.com]

Comment #14 on issue 365 by vitaly.berov: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

That's fine but it's a blocker bug and it's opened from november. Seems
strange for
me. Does owners evolve the project?

By the way, I see that the bug is closed.
Am I understand correctly that the fix will be in the next version?

[msy...@googlecode.com]

Comment #15 on issue 365 by kusmabite: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

What do you mean by "owners"? Development on Git for Windows is done on a
volunteer
basis - you basically have to scratch your own itch to get somewhere.

It seems that the author of the patch that was posted lost interest, and
dropped it to
the floor. If you care enough about the issue, why don't you pick up the
work and
finish the patch? The most important step is to mail it to the mailing-list
for
discussion. Not everyone involved in the project follows the issue-tracker.

[msy...@googlecode.com]

Comment #16 on issue 365 by johannes.schindelin: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

vitaly.berov: to add to kusmabite's comment: no, there is no fix. Nobody
seems to be
willing to work on the bug. So no, there will not be a fix in the next
version.

Unless somebody interested picks up the work. That somebody could be you.
Or that
somebody might not exist.

[msy...@googlecode.com]

Comment #17 on issue 365 by kusmabite: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

It seems I'm a nicer person than I though; I just forwarded the patch to
the mailing-
list. This is just the first step, though. It might have to take some
rounds of
polishing to make it into a release.

http://groups.google.com/group/msysgit/browse_thread/thread/fd104ff00a51b017

[msy...@googlecode.com]

Comment #18 on issue 365 by sanjeevakp: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Thanks for the patch
Looking forward for the release.

Hitting error after downloading the files.

$ git clone git...@xxx.xx.xx.xx:/Docs.git
Initialized empty Git repository in c:/repositories/Repo/.git/
remote: Counting objects: 3788, done.
remote: Compressing objects: 100% (3779/3779), done.
Receiving objects: 100% (3788/3788), 4.22 GiB | 1.01 MiB/s, done.
remote: Total 3788 (delta 42), reused 3655 (delta 1)
Resolving deltas: 100% (42/42), done.

fatal: Out of memory? mmap failed: No such file or directory

[msy...@googlecode.com]

Comment #19 on issue 365 by kusmabite: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Are you getting the error with the patch applied, or am I misunderstanding
you?

[Sanjeev]

I am getting error without the patch.

On Apr 15, 3:35 am, msys...@googlecode.com wrote:
> Comment #19 on issue 365 by kusmabite: Out of memory? mmap failedhttp://code.google.com/p/msysgit/issues/detail?id=365

[Johannes Schindelin]

Hi,

On Thu, 15 Apr 2010, Sanjeev wrote:

> I am getting error without the patch.

And what exactly did you want to contribute with your comment, then? Why
did you not try the patch?

Sheesh,
Dscho

[Sanjeev]

I just started on Git and MsysGit and willing to contribute.
Still setting up my laptop with development Environment.
Can someone please locate where to start and which branch to map to
get the latest change

On Apr 15, 4:07 am, Johannes Schindelin <Johannes.Schinde...@gmx.de>
wrote:

[Erik Faye-Lund]

In the future, please don't top-post as it makes it trickier to track
the discussion.

I've corrected it locally in the reply, though.

On Thu, Apr 15, 2010 at 10:07 PM, Sanjeev <sanje...@gmail.com> wrote:
> On Apr 15, 4:07 am, Johannes Schindelin <Johannes.Schinde...@gmx.de>
> wrote:
>> Hi,
>>
>> On Thu, 15 Apr 2010, Sanjeev wrote:
>> > I am getting error without the patch.
>>
>> And what exactly did you want to contribute with your comment, then? Why
>> did you not try the patch?
>>
>> Sheesh,
>> Dscho
>
> I just started on Git and MsysGit and willing to contribute.
> Still setting up my laptop with development Environment.
> Can someone please locate where to start and which branch to map to
> get the latest change
>

I just pushed out the commit to the 'mob'-branch in 4msysgit.git. You
can cherry-pick it from there onto devel to get the latest version
with this patch included.

--
Erik "kusma" Faye-Lund

[Johannes Schindelin]

Hi,

On Thu, 15 Apr 2010, Erik Faye-Lund wrote:

> In the future, please don't top-post as it makes it trickier to track
> the discussion.

Actually, I encourage all people who top-post to continue to do so. It is
a very convenient indicator.

Ciao,
Dscho



--
Subscription settings: http://groups.google.com/group/msysgit/subscribe?hl=en

[msy...@googlecode.com]

Updates:
Status: Accepted
Owner: Johannes.Sixt

Comment #20 on issue 365 by Johannes.Sixt: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Reopening because I'll carry the patch in my tree. I don't have large enough
repositories in production, though, that I could observe the bug.

[msy...@googlecode.com]

Updates:
Status: Fixed

Comment #21 on issue 365 by Johannes.Sixt: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

Fixed in 1.7.1.1

[msy...@googlecode.com]

Comment #22 on issue 365 by diannepabst77: Out of memory? mmap failed
http://code.google.com/p/msysgit/issues/detail?id=365

I'm still getting the error:

D:\Users\Princess Di>git push usb-hd master
Counting objects: 4865, done.
Delta compression using up to 4 threads.
fatal: Out of memory? mmap failed: No error
error: pack-objects died with strange error
error: failed to push some refs to 'h:\Git-Dianne-Toshiba-2010-06'

D:\Users\Princess Di>git --version
git version 1.7.2.3.msysgit.0