"fatal: Unable to write new index file" on NTFS

[Trevor]

I'm working on a Source Safe to Git migration tool that generates
calls to Git very quickly but serially (i.e. it waits for the last
process to exit before invoking another). Every dozen or so commands,
I get a failure like "fatal: Unable to write new index file". (That's
the message for "git add"; "git commit" says something similar but
slightly different, such as "fatal: unable to write new_index".) If I
retry the command, it works. Any idea why this might be happening?

I'm running Git-1.6.2.1-preview20090322.exe on an NTFS partition on
Vista SP1. The same issue occurs with the Cygwin build. I've heard of
this happening on FAT32, but my one and only partition is NTFS.

Thanks for help,
Trevor

[Peter Harris]

On Mon, Mar 30, 2009 at 9:00 PM, Trevor wrote:
>
> I'm working on a Source Safe to Git migration tool that generates
> calls to Git very quickly but serially (i.e. it waits for the last
> process to exit before invoking another). Every dozen or so commands,
> I get a failure like "fatal: Unable to write new index file". (That's
> the message for "git add"; "git commit" says something similar but
> slightly different, such as "fatal: unable to write new_index".) If I
> retry the command, it works. Any idea why this might be happening?

Is your virus scanner (and/or "Windows Defender", on Vista) still
scanning the index file?

Peter Harris

[Trevor]

Yeah, looks like Windows Defender strikes again. I always forget to
disable that damn thing. I can't believe how broken it is. Destroys
Blu-ray playback at home, breaks Git at work...

Thanks,
Trevor

[Johannes Sixt]

On Dienstag, 31. März 2009, Trevor wrote:
> I'm working on a Source Safe to Git migration tool that generates
> calls to Git very quickly but serially (i.e. it waits for the last
> process to exit before invoking another). Every dozen or so commands,
> I get a failure like "fatal: Unable to write new index file". (That's
> the message for "git add"; "git commit" says something similar but
> slightly different, such as "fatal: unable to write new_index".) If I
> retry the command, it works. Any idea why this might be happening?

I see this also far too often. Incidentally, I've made a work-around just
today. You can pick it up from my repository at the tip of

git://repo.or.cz/git/mingw/j6t.git master

> I'm running Git-1.6.2.1-preview20090322.exe on an NTFS partition on
> Vista SP1. The same issue occurs with the Cygwin build. I've heard of
> this happening on FAT32, but my one and only partition is NTFS.

I've NTFS, too, but Windows XP SP2. I blame Explorer: Perhaps it looks into
the file while it updates the directory structure?

-- Hannes

[Johannes Schindelin]

Hi,

On Tue, 31 Mar 2009, Johannes Sixt wrote:

>
> On Dienstag, 31. März 2009, Trevor wrote:
> > I'm working on a Source Safe to Git migration tool that generates
> > calls to Git very quickly but serially (i.e. it waits for the last
> > process to exit before invoking another). Every dozen or so commands,
> > I get a failure like "fatal: Unable to write new index file". (That's
> > the message for "git add"; "git commit" says something similar but
> > slightly different, such as "fatal: unable to write new_index".) If I
> > retry the command, it works. Any idea why this might be happening?
>
> I see this also far too often. Incidentally, I've made a work-around
> just today. You can pick it up from my repository at the tip of
>
> git://repo.or.cz/git/mingw/j6t.git master

I made an intermediate release here:

http://msysgit.googlecode.com/files/Git-rename-harder.exe

> > I'm running Git-1.6.2.1-preview20090322.exe on an NTFS partition on
> > Vista SP1. The same issue occurs with the Cygwin build. I've heard of
> > this happening on FAT32, but my one and only partition is NTFS.
>
> I've NTFS, too, but Windows XP SP2. I blame Explorer: Perhaps it looks
> into the file while it updates the directory structure?

Incidentally, I got a bug report that link() (called from
move_temp_to_file() on the pack and its index file after an initial clone)
returns EEXIST.

Might this be a related issue?

(I am somewhat puzzled by link() returning EEXIST in that case, as there
is _definitely_ no other pack than the temporary one there.)

Ciao,
Dscho

[Peter Harris]

On Tue, Mar 31, 2009 at 3:08 PM, Johannes Sixt wrote:
>
> On Dienstag, 31. März 2009, Trevor wrote:
>> I'm working on a Source Safe to Git migration tool that generates
>> calls to Git very quickly but serially (i.e. it waits for the last
>> process to exit before invoking another). Every dozen or so commands,
>> I get a failure like "fatal: Unable to write new index file". (That's
>> the message for "git add"; "git commit" says something similar but
>> slightly different, such as "fatal: unable to write new_index".) If I
>> retry the command, it works. Any idea why this might be happening?
>
> I see this also far too often. Incidentally, I've made a work-around just
> today. You can pick it up from my repository at the tip of
>
>  git://repo.or.cz/git/mingw/j6t.git master

Very nice.

Minor nit: Sleep(0); doesn't actually do anything if there isn't
something waiting for CPU. So if the other task that has the file
locked (Explorer, or the virus scanner, or whatever) is blocked on IO,
your entire loop will expire before the other task gets a chance to
run. I suspect Sleep(1); would be a lot more effective here.

Peter Harris

[Johannes Schindelin]

Hi,

Sleep(1) would mean just 1 microsecond anyway, right?

Ciao,
Dscho

[Peter Harris]

1 millisecond, actually. Except Windows' scheduling granularity is on
the order of 15ms most of the time (there are APIs to change that, but
it's usually not worth the effort), so you probably won't wake up
again quite as quickly as 1ms.

(For reference, Subversion uses exponential backoff, and doesn't give
up until they've done a 128ms sleep, for 1/4 of a second total Sleep()
before really giving up. That's probably overkill on modern machines.)

Peter Harris

[Johannes Schindelin]

Hi,

On Tue, 31 Mar 2009, Peter Harris wrote:

> On Tue, Mar 31, 2009 at 8:46 PM, Johannes Schindelin wrote:
> > On Tue, 31 Mar 2009, Peter Harris wrote:
> >> Minor nit: Sleep(0); doesn't actually do anything if there isn't
> >> something waiting for CPU. So if the other task that has the file
> >> locked (Explorer, or the virus scanner, or whatever) is blocked on
> >> IO, your entire loop will expire before the other task gets a chance
> >> to run. I suspect Sleep(1); would be a lot more effective here.
> >
> > Sleep(1) would mean just 1 microsecond anyway, right?
>
> 1 millisecond, actually.

Heh, that's what I meant. I hear a lot about micro, nano and femtoseconds
these days, hence the confusion...

> Except Windows' scheduling granularity is on the order of 15ms most of
> the time (there are APIs to change that, but it's usually not worth the
> effort), so you probably won't wake up again quite as quickly as 1ms.
>
> (For reference, Subversion uses exponential backoff, and doesn't give up
> until they've done a 128ms sleep, for 1/4 of a second total Sleep()
> before really giving up. That's probably overkill on modern machines.)

Good to know!

Thanks,
Dscho

[Peter Harris]

On Tue, Mar 31, 2009 at 9:14 PM, Johannes Schindelin wrote:
> On Tue, 31 Mar 2009, Peter Harris wrote:
>> On Tue, Mar 31, 2009 at 8:46 PM, Johannes Schindelin wrote:
>> > On Tue, 31 Mar 2009, Peter Harris wrote:
>> >> Minor nit: Sleep(0); doesn't actually do anything if there isn't
>> >> something waiting for CPU. So if the other task that has the file
>> >> locked (Explorer, or the virus scanner, or whatever) is blocked on
>> >> IO, your entire loop will expire before the other task gets a chance
>> >> to run. I suspect Sleep(1); would be a lot more effective here.
>> >
>> > Sleep(1) would mean just 1 microsecond anyway, right?
>>
>> 1 millisecond, actually.
>
> Heh, that's what I meant.  I hear a lot about micro, nano and femtoseconds
> these days, hence the confusion...

That's the wonderful thing about standards. There are so many to choose from!

(Windows' Sleep() is milliseconds. APR uses microseconds. Linux has
nanosleep. Nobody uses the same units.)

Peter Harris

[Alexander Gavrilov]

On Tue, Mar 31, 2009 at 5:00 AM, Trevor <tre...@scurrilous.com> wrote:
> I'm working on a Source Safe to Git migration tool that generates

By the way, last summer I made a script for incremental exchange of
simple changes between VSS and Git, based on the logging feature of
VSS. However, it cannot convert old history, and there is no
documentation.

You can find it here: http://github.com/angavrilov/git-vss/

Alexander

[Johannes Sixt]

On Mittwoch, 1. April 2009, Peter Harris wrote:
> Minor nit: Sleep(0); doesn't actually do anything if there isn't
> something waiting for CPU. So if the other task that has the file
> locked (Explorer, or the virus scanner, or whatever) is blocked on IO,
> your entire loop will expire before the other task gets a chance to
> run.

I know. But in practice I have seen 3 retries at one occasion, all other cases
where the loop was taken it succeeded at the second try (1 retry).

> I suspect Sleep(1); would be a lot more effective here.

But it waits even if it is not necessary. And it *does* take a "long" time to
complete the function call if the failure with ERROR_ACCESS_DENIED is for a
good reason.

I've been bitten by Sleep(1) when I made the busy-loop in our poll()
implementation: If it is not Sleep(0) there, then every local fetch is slowed
down by a magnitude or two.

-- Hannes

[Peter Harris]

On Wed, Apr 1, 2009 at 2:23 PM, Johannes Sixt wrote:
> On Mittwoch, 1. April 2009, Peter Harris wrote:
>> Minor nit: Sleep(0); doesn't actually do anything if there isn't
>> something waiting for CPU. So if the other task that has the file
>> locked (Explorer, or the virus scanner, or whatever) is blocked on IO,
>> your entire loop will expire before the other task gets a chance to
>> run.
>
> I know. But in practice I have seen 3 retries at one occasion, all other cases
> where the loop was taken it succeeded at the second try (1 retry).

Colour me surprised. I didn't expect that result.

I can't argue with hard numbers.

Thanks,
Peter Harris

[Robin Rosenberg]

onsdag 01 april 2009 20:23:21 skrev Johannes Sixt <j...@kdbg.org>:
>
> On Mittwoch, 1. April 2009, Peter Harris wrote:
> > Minor nit: Sleep(0); doesn't actually do anything if there isn't
> > something waiting for CPU. So if the other task that has the file
> > locked (Explorer, or the virus scanner, or whatever) is blocked on IO,
> > your entire loop will expire before the other task gets a chance to
> > run.
>
> I know. But in practice I have seen 3 retries at one occasion, all other cases
> where the loop was taken it succeeded at the second try (1 retry).

Is there any difference on single/multi-core machines?

-- robin

[Johannes Sixt]

I cannot tell. I have only dual core machines. But I have tried this patch on
only one of them so far (the one where I observed the failure considerably
more often).

-- Hannes

[Trevor]

Thanks. My program actually exports full history from VSS to Git,
including synthesizing multi-file changesets/commits. It's basically
done, and I plan to open-source it soon. It's based on a library I
wrote to read the VSS data files directly, which might be useful for
people wanting to create other VSS tools. Not that anyone should be
using VSS anymore. ;-)