[msysGit] Fetching from msysgit.git (pack has bad object)

[Jonathan Nieder]

Hi,

Downloaded msysgit for $dayjob today, and it works quite nicely ---
thanks! Unfortunately, I cannot fetch the rest of the branches from
msysgit.git:

$ cd /
$ git fetch origin
remote: Counting objects: 8991, done.
remote: Compressing objects: 100% (4091/4091), done.
error: inflate: data stream error (incorrect data check)B/s
fatal: pack has bad object at offset 64539890: inflate returned -3
fatal: write error: Broken pipe
fatal: index-pack failed
$ git remote show origin | head -2
* remote origin
Fetch URL: git://repo.or.cz/msysgit.git
$ git version
git version 1.7.1.166.gf2086.dirty

Same problem under Linux:

$ cd /tmp
$ git clone git://repo.or.cz/msysgit.git
Initialized empty Git repository in /tmp/msysgit/.git/
remote: Counting objects: 19471, done.
remote: Compressing objects: 100% (9081/9081), done.
error: inflate: data stream error (incorrect data check) MiB/s
fatal: pack has bad object at offset 129707676: inflate returned -3
fatal: index-pack failed
$ git clone http://repo.or.cz/r/msysgit.git
Initialized empty Git repository in /tmp/msysgit/.git/
error: /tmp/msysgit/.git/objects/pack/pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack SHA1 checksum mismatch
error: inflate: data stream error (incorrect data check)
error: cannot unpack eb13a0a8705adbd3980332e6a7c0c236217bc503 from /tmp/msysgit/.git/objects/pack/pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack at offset 129707676
error: Unable to find 26b48ce090ba897a598fc42bf0a8412d5a14fee9 under http://repo.or.cz/r/msysgit.git
Cannot obtain needed object 26b48ce090ba897a598fc42bf0a8412d5a14fee9
error: Fetch failed.
$ git version
git version 1.7.1

Is this a known problem? Anything I can do to help track it down?

Thanks,
Jonathan

[Johannes Schindelin]

Hi,

On Wed, 5 May 2010, Jonathan Nieder wrote:

> Same problem under Linux:
>
> $ cd /tmp
> $ git clone git://repo.or.cz/msysgit.git
> Initialized empty Git repository in /tmp/msysgit/.git/
> remote: Counting objects: 19471, done.
> remote: Compressing objects: 100% (9081/9081), done.
> error: inflate: data stream error (incorrect data check) MiB/s
> fatal: pack has bad object at offset 129707676: inflate returned -3
> fatal: index-pack failed
> $ git clone http://repo.or.cz/r/msysgit.git
> Initialized empty Git repository in /tmp/msysgit/.git/
> error: /tmp/msysgit/.git/objects/pack/pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack SHA1 checksum mismatch
> error: inflate: data stream error (incorrect data check)
> error: cannot unpack eb13a0a8705adbd3980332e6a7c0c236217bc503 from /tmp/msysgit/.git/objects/pack/pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack at offset 129707676
> error: Unable to find 26b48ce090ba897a598fc42bf0a8412d5a14fee9 under http://repo.or.cz/r/msysgit.git
> Cannot obtain needed object 26b48ce090ba897a598fc42bf0a8412d5a14fee9
> error: Fetch failed.
> $ git version
> git version 1.7.1

If it is also an issue with Linux, maybe you want to clone from

git://github.com/msysgit/msysgit.git

as it could be a repo.or.cz issue?

Ciao,
Dscho

[Jonathan Nieder]

Hi,

Johannes Schindelin wrote:
> On Wed, 5 May 2010, Jonathan Nieder wrote:

>> error: cannot unpack eb13a0a8705adbd3980332e6a7c0c236217bc503 from /tmp/msysgit/.git/objects/pack/pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack at offset 129707676
>> error: Unable to find 26b48ce090ba897a598fc42bf0a8412d5a14fee9 under http://repo.or.cz/r/msysgit.git

[...]

> If it is also an issue with Linux, maybe you want to clone from
>
> git://github.com/msysgit/msysgit.git

Looking at just master and devel (from repo.or.cz or github) indeed
works.

Over git protocol, I can fetch most other branches, too, except for
work/autoconf. Exploding the pack 5b51af9... with ‘git unpack-objects
-r’ provides, among other things, the tip of the work/autoconf branch;
and after I create a ref to point to it, ‘git fetch origin’ works
again.

Here’s what ‘git fsck --full’ says:

broken link from tree 2d9454b21974f0b493a8b176578dbe465d57225b
to blob 25b1d15fec3867dbe0e5cf8e719b7e14bed7a945
broken link from tree 1f1b51d53c2e06c2360ba7e85ec1f16d740d0964
to blob eb13a0a8705adbd3980332e6a7c0c236217bc503
missing blob eb13a0a8705adbd3980332e6a7c0c236217bc503
missing blob 25b1d15fec3867dbe0e5cf8e719b7e14bed7a945

The first missing blob is mingw/bin/perl58.dll from work/autoconf~8
(2a213da --- Install Perl 5.8.8, 2009-05-13). The other missing
blob is mingw/bin/perl58.dll from work/autoconf~4 (dae81ad ---
Install Perl 5.8.8, 2009-05-14).

If you have spare copies of them around, I would be interested. It
would be nice to know what kind of corruption this was.

Thanks for the pointer,
Jonathan

[Johannes Schindelin]

Hi,

[Pasky Cc:ed]

On Thu, 6 May 2010, Jonathan Nieder wrote:

> missing blob eb13a0a8705adbd3980332e6a7c0c236217bc503
> missing blob 25b1d15fec3867dbe0e5cf8e719b7e14bed7a945

Attached.

> It would be nice to know what kind of corruption this was.

Pasky, it seems that something happened to your box. If I download
http://repo.or.cz/r/msysgit.git/objects/pack/pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack
directly and use git verify-pack on it, it says:

error: pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack SHA1 checksum mismatch

error: inflate: data stream error (incorrect data check)

error: cannot unpack eb13a0a8705adbd3980332e6a7c0c236217bc503 from pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack at offset 129707676

Either your box has a disk that's about to go bad, or the repack code is
borked.

Ciao,
Dscho

[Petr Baudis]

Hi!

Yes, I have noticed recently that the number of bad repositories is
quite suspicious by now, and has nothing to do with forks this time.
I'm a bit at loss now on what might be the culprit, though - I have run
memory tests some time ago and the disks are in RAID1.

I will now run a RAID1 check and then let the memory test run for a
while; if that doesn't help, it has to be either the net or the SATA
controller (damaging data to both disks equally). I think it can't be
the net since git checks the checksums of incoming data, right? So it
would have to be the SATA datapath, I'm thinking badblocks might catch
that. Any ideas are welcome.

--
Petr "Pasky" Baudis
When I feel like exercising, I just lie down until the feeling
goes away. -- xed_over

[Johannes Schindelin]

Hi,

On Thu, 6 May 2010, Johannes Schindelin wrote:

> Pasky, it seems that something happened to your box. If I download
> http://repo.or.cz/r/msysgit.git/objects/pack/pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack
> directly and use git verify-pack on it, it says:
>
> error: pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack SHA1 checksum mismatch
> error: inflate: data stream error (incorrect data check)
> error: cannot unpack eb13a0a8705adbd3980332e6a7c0c236217bc503 from pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack at offset 129707676
>
> Either your box has a disk that's about to go bad, or the repack code is
> borked.

Note that I think that Git is pretty broken (I seem to remember that this
was not the case earlier): cat <attachment> | git hash-object -w --stdin
does _not_ write a new loose object. (IOW after said command, there is
_still_ no valid 25b1d1 and neither eb13a0!!!) I had to initialize a dummy
repository and copy over .git/objects/??/* !!!

Another problem is that a "git gc" returned "Nothing to repack". Which is
clearly untrue. I had to use quite a lot of low-level tricks to get the
whole thing to repack correctly.

So, clearly, Git sucks when it comes to fixing corruptions. We knew that,
but I thought that Nico worked on that part.

Apparently, the easiest way to get a correct clone is to

1) initialize a new, empty repository,

2) add the 'origin' remote thusly:

git remote add -f origin http://repo.or.cz/r/msysgit.git

3) in spite of the error, continue by fetching the pack manually:

(cd .git/objects/pack/ && curl -O \
http://repo.or.cz/r/msysgit.git/objects/pack/pack-5b51af9cb1b1122d27ddd53d40a8c3e4472ae9e4.pack)

4) adding the non-corrupted objects using the attached bundle:

git bundle unbundle borked.bundle

5) Now fetch from 'origin' again (to update the refs correctly):

git fetch origin

6) Manually (!!!) remove the borked temporary object files:

find .git -name \*.temp -exec rm {} \;

7) Now, 'git gc'

If you already have a (borked) checkout, you might be able to get away
with 4, 6 and 7.

If we did not have the dumb transport, we would be thoroughly fscked.

Unamazed,
Dscho

[Johannes Schindelin]

Hi,

On Thu, 6 May 2010, Petr Baudis wrote:

> On Thu, May 06, 2010 at 02:11:56PM +0200, Johannes Schindelin wrote:
>
> > Either your box has a disk that's about to go bad, or the repack code
> > is borked.
>
> Yes, I have noticed recently that the number of bad repositories is
> quite suspicious by now, and has nothing to do with forks this time. I'm
> a bit at loss now on what might be the culprit, though - I have run
> memory tests some time ago and the disks are in RAID1.

I'd be interested to know which Git version you are running. I am not at
all convinced that it is a hardware problem.

Ciao,
Johannes "who will fsck religiously again, exactly as 5 years ago"

[Petr Baudis]

Hi!

On Thu, May 06, 2010 at 02:51:59PM +0200, Johannes Schindelin wrote:
> On Thu, 6 May 2010, Petr Baudis wrote:
>
> > On Thu, May 06, 2010 at 02:11:56PM +0200, Johannes Schindelin wrote:
> >
> > > Either your box has a disk that's about to go bad, or the repack code
> > > is borked.
> >
> > Yes, I have noticed recently that the number of bad repositories is
> > quite suspicious by now, and has nothing to do with forks this time. I'm
> > a bit at loss now on what might be the culprit, though - I have run
> > memory tests some time ago and the disks are in RAID1.
>
> I'd be interested to know which Git version you are running. I am not at
> all convinced that it is a hardware problem.

I'm running (both outside and within the ssh chroot) git-1.5.6.5,
which is the default version in Debian lenny.

Hmm! Except that the git used for mirroring and gc is 1.6.5.GIT,
further details not readily known! It's hard to say if some bug is
lurking within that, we should bring that up to some stable version.

[Petr Baudis]

On Thu, May 06, 2010 at 02:51:59PM +0200, Johannes Schindelin wrote:

> Hi,
>
> On Thu, 6 May 2010, Petr Baudis wrote:
>
> > On Thu, May 06, 2010 at 02:11:56PM +0200, Johannes Schindelin wrote:
> >
> > > Either your box has a disk that's about to go bad, or the repack code
> > > is borked.
> >
> > Yes, I have noticed recently that the number of bad repositories is
> > quite suspicious by now, and has nothing to do with forks this time. I'm
> > a bit at loss now on what might be the culprit, though - I have run
> > memory tests some time ago and the disks are in RAID1.
>
> I'd be interested to know which Git version you are running. I am not at
> all convinced that it is a hardware problem.

Block Sequential : testing 120FAILURE: 0x7878787878787878 !=
0x787878787878787a at offset 0x084c490a.
Block Sequential : testing 108FAILURE: 0x6c6c6c6c6c6c6c6c !=
0x6c6c6c6c6c6c6c6e at offset 0x084c490a.

:-(

Now the very fun part of figuring out which DIMM module's fault it is;
since it happenned at the exact same memory address both times,
hopefully it will be just a faulty module.

I mistakenly restarted the memtester, so I really hope it mlock()ed
the faulty memory again.

[Jonathan Nieder]

Petr Baudis wrote:

> Block Sequential : testing 120FAILURE: 0x7878787878787878 !=
> 0x787878787878787a at offset 0x084c490a.
> Block Sequential : testing 108FAILURE: 0x6c6c6c6c6c6c6c6c !=
> 0x6c6c6c6c6c6c6c6e at offset 0x084c490a.
>
> :-(
>
> Now the very fun part of figuring out which DIMM module's fault it is;
> since it happenned at the exact same memory address both times,
> hopefully it will be just a faulty module.

To the very bit. Thanks for looking into it.

(FWIW msysgit.git is fixed now. Thanks!)

Jonathan

[Petr Baudis]

I've spent the better part of the evening in the rack; the memory
controller behaves most curiously, actually refusing to acknowledge one
or two DIMMs out of the six in various configurations, or showing them
after poweron, but then starting to pretend they are empty in the DMI
info after reboot from memtest to Linux.

But now, I have found a configuration of DIMMs that seems to work
reliably and where all DIMMs are visible. 7G out of the 9G are being
tested by memtester again now, let's see if errors are detected again
now. (admin: I have disabled the automatic gc'ing due to the memory
shortage for now.)

(Too bad my BIOS reports broken DMI info, making it seemingly impossible
to translate physical address to a DIMM. I have wrote about some of my
journey to get the physical address at: http://log.or.cz/?p=89)