Issue 457 in msysgit: Push hangs in latest msysgit 1.7.0.2

[msy...@googlecode.com]

Status: New
Owner: ----

New issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

What steps will reproduce the problem?
1. Start a Git daemon, with receive-pack enabled
2. Push anything (new branch or update of current branch)
3. Git client hangs indefinitely...

What is the expected output? What do you see instead?

git daemon --export-all --enable=receive-pack --base-path=`pwd` --verbose

[13353] Connection from 10.1.1.3:56651
[13353] Extended attributes (15 bytes) exist <host=10.1.1.3>
[13353] Request receive-pack for '/blah'

Pushing with verbose on the client doesn't seem to have anything useful to
say.

What version of the product are you using? On what operating system?

Using the latest version of msysgit (1.7.0.2) on Windows and 1.7.0.3 on
Ubuntu as the server.

Please provide any additional information below.

I am able to push to my github account just fine (ie
g...@github.com:me/blah.git). Fetching and deleting branches works ok. After
reverting back to the previous msysgit version I can push again. I've
tested this on three separate Windows machines and two different Ubuntu
machines with the daemon, so I'm pretty sure it's not an isolated case.

I've attached two Wireshark traces. One from a working 1.6.5.1 and the
other obviously the unresponsive 1.7.0.2.

Can I hazard a guess it's related to 'receive-pack' service? Does anyone
have any idea from these scraps of information? Apologies in advance if
this has been answered elsewhere or I've missed something obvious.

Attachments:
bad.trace 1.6 KB
good.trace 2.0 KB

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings

[msy...@googlecode.com]

Comment #1 on issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

I have tracked this down to a886ba2, which was a feature developed by Shawn
Pearce. I
have contacted (harassed) him directly to see if he has any ideas. I'll
also see if I
can pin-point the issue further, but my C skills are pretty much
non-existent.

[msy...@googlecode.com]

Comment #2 on issue 457 by spea...@spearce.org: Push hangs in latest
msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

Yea, I have no clue what's going on here.

The side-band-64k shouldn't have had a big impact on
Win32. But apparently it does. bad.trace seems to
imply that the client is waiting for something after
it has sent the command list, but before it sends the
pack file. The server is most certainly waiting for
the pack. So classic case of deadlock due to both of
the peers trying to read at once?

This is likely the "if (use_sideband && cmds_sent)"
block of builtin-send-pack.c around line 474. There
must be a deadlock in the client while it is creating
the async thread to act as a side-band demux.

As for why GitHub works and local git-daemon doesn't,
they probably haven't upgraded their binaries to use
the side-band-64k protocol for send-pack/receive-pack
yet. So the option isn't being enabled, and the client
is bypassing this start_async invocation.

Sorry I can't be more help. But I just don't get the
async stuff used in Win32.

[msy...@googlecode.com]

Comment #3 on issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

Apologies if this seems obvious. Things I have determined:

In the 'main' thread it is stuck in pack_objects() on finish_command() (~
line 102).
I can also see that the cmd_pack_objects() function is never actually
reached as part
of that fork.

Finally, I found that calling sleep(1) at the beginning of sideband_demux()
avoids
the deadlock. Is there some missing synchronization required between the
two threads?

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings

--
Subscription settings: http://groups.google.com/group/msysgit/subscribe?hl=en

[msy...@googlecode.com]

Comment #4 on issue 457 by Johannes.Sixt: Push hangs in latest msysgit
1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

I can reproduce the problem. The problematic commit is 0c499ea60f, which
introduced
the sideband support in send_pack.

But when I insert the sleep(1) in sideband_demux, the dead-lock does not go
away,
it is only delayed: without it the hang happens before pack-object gets to
do any
work, with it, the hang happens when objects are written to the connection.
Can you
confirm this?

[Johannes Sixt]

On Samstag, 17. April 2010, msy...@googlecode.com wrote:
> Comment #3 on issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
> http://code.google.com/p/msysgit/issues/detail?id=457
>
> Apologies if this seems obvious. Things I have determined:
>
> In the 'main' thread it is stuck in pack_objects() on finish_command() (~
> line 102).
> I can also see that the cmd_pack_objects() function is never actually
> reached as part
> of that fork.
>
> Finally, I found that calling sleep(1) at the beginning of sideband_demux()
> avoids
> the deadlock. Is there some missing synchronization required between the
> two threads?

I can reproduce the problem.

I'm working on commit 0c499ea60f, which introduced the failure.

Question to Windows exports: Is it allowed that one thread writes to a socket
and a different thread reads from it? What if one thread writes to a socket,
and a different thread reads from a dup of it?

-- Hannes

[msy...@googlecode.com]

Comment #5 on issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

Hi Hannes,

Thanks for the reply.

The sleep(1) for me fixes the problem completely. That is to say I can push
successfully without any hanging. I'm working off of commit
v1.7.0.2.msysgit.0
(956d70ff7).

Is there anything I can do or provide that will help further? I wish I
could fix this
problem myself, but the Windows threading is way over my head.

Charles

[msy...@googlecode.com]

Comment #6 on issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

Maybe I should add that I'm testing this on a repository with only one
commit and one
file. Would that perhaps explain the difference in the hanging on your end?

[Johannes Sixt]

What happens is that git-send-pack runs git-pack-objects, but git-pack-objects
does not even enter main().

I've no clue how to fix it. I think we have to work around it by disabling
side-band-64k for the combination git protocol + Windows.

[msy...@googlecode.com]

Updates:
Status: WontFix

Comment #7 on issue 457 by johannes.schindelin: Push hangs in latest
msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

Seems nobody wants to work on this issue. Tentatively marking as WontFix

[Johannes Sixt]

On Sonntag, 25. April 2010, msy...@googlecode.com wrote:
> Updates:
> Status: WontFix
>
> Comment #7 on issue 457 by johannes.schindelin: Push hangs in latest
> msysgit 1.7.0.2
> http://code.google.com/p/msysgit/issues/detail?id=457
>
> Seems nobody wants to work on this issue. Tentatively marking as WontFix

Now come on!

If you don't care about the issues in the issue tracker anyway, why the
f*cking hell don't you just ignore the tracker completely instead of closing
reports at random?

If you kept the bugs open, we at least would get a lot less duplicated
reports!

-- Hannes

Oh, btw, this one's on my radar.

[msy...@googlecode.com]

Comment #8 on issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

I would just like to add that this is likely going to become a big problem
for anyone
trying to push to GitHub/repo.or.cz/etc, if and when those sites start using
side-band-64k. I understand the lack of interest though, so I'll continue
to work
around the issue.

[Johannes Schindelin]

Hi,

On Sun, 25 Apr 2010, Johannes Sixt wrote:

> On Sonntag, 25. April 2010, msy...@googlecode.com wrote:
> > Updates:
> > Status: WontFix
> >
> > Comment #7 on issue 457 by johannes.schindelin: Push hangs in latest
> > msysgit 1.7.0.2
> > http://code.google.com/p/msysgit/issues/detail?id=457
> >
> > Seems nobody wants to work on this issue. Tentatively marking as WontFix
>
> Now come on!
>
> If you don't care about the issues in the issue tracker anyway, why the
> f*cking hell don't you just ignore the tracker completely instead of
> closing reports at random?

Because they annoy the hell out of me.

> If you kept the bugs open, we at least would get a lot less duplicated
> reports!

No, that is not right. You would get a ton of useless bug reports, because
people like Robin would feel encouraged to write their personal diaries in
the issue tracker.

> Oh, btw, this one's on my radar.

Then you did a lousy job of advertising it. Very lousy.

Ciao,
Dscho

[msy...@googlecode.com]

Comment #9 on issue 457 by johannes.schindelin: Push hangs in latest
msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

Working around the issue is the wrong thing to do. You should try to _fix_
it.

[msy...@googlecode.com]

Comment #10 on issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

Hi Dscho,

Apologies if I came across as someone expecting you guys to magically spend
their
free time fixing this. I wasn't expecting anything from you kind folks who
have made
msysGit as awesome as it is. I was, however, hoping someone with more
experience
might be able to take one look at the problem and say 'aha - the problem is
this',
rather than me banging my head against the wall for weeks as I start from
scratch.
Obviously at this stage that doesn't seem likely.

Of _course_ I should try to fix this, if only I knew where to begin. Now I
like a
challenge as much as the next developer, and it would be good for me to get
my first
C/Git/Windows experience, but at the moment I feel a _little_ out of my
depth
tackling all three at once.

If you, or anyone, have any suggestions on places to start or things to
consider I
would very much appreciate it. In the meantime I will try my best to fix
this.

[Johannes Sixt]

"Job"? "JOB"?? If I should be obliged to announce what I'm about to do, then I
expect compensation. You should know better whether we doing any sort
of "job" here.

Just to make myself clear: I don't care so much whether you close any issues,
because I mostly ignore the issuer tracker anyway; I do care about the tone
with which it happens.

Why am I still whining? Because the rudeness of your tone pisses off not only
incompetent people, but also competent ones. It makes this place in the 'net
a cold corner where people don't want to live.

And it triggers unnecessary replies (appologies, mostly), which take away my
time because the subject line alone doesn't tell me whether I can safely skip
a message.

This particular issue is good example what could go wrong when you close
issues in a haste and after only superficial reading: charleso, the OP, did
already a great job by digging up a workaround. I've looked into the issue
long enough to know that this workaround is damn difficult to come up with.
Ergo, charleso is *competent*. With your snooty act of closing the bug, you
likely piss off people like him, whose contribution we urgently need.

So please, in the future, when you close issues that annoy the hell out of
you, calm down first. Don't be a snob; use the word "please".

-- Hannes

[msy...@googlecode.com]

Updates:
Status: Accepted
Owner: Johannes.Sixt

Comment #11 on issue 457 by Johannes.Sixt: Push hangs in latest msysgit
1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

First of all, this seems to be a problem only for pushes via git protocol,
because I
do regular pushes via ssh and sometimes also locally, both of which work as
expected,
even with long lists of refs and a non-trivial amount of commits.

According to Procmon, the difference between the good and the bad case (the
good is
when sleep(1) is inserted in sideband_demux) is that the bad case transmits
data to
the server earlier.

The hang happens in CreateProcess before the child process gets to load any
DLLs. The
child process is git-pack-objects, but note that it doesn't do anything to
the network.

I tried to use select() instead of sleep(), but it doesn't wait; it returns
-1, but
WSAGetError() is 0.

I'm clueless now. The only solution I see is to disable side-band-64k for
pushes via
git protocol.

[Johannes Schindelin]

Hi,

On Mon, 26 Apr 2010, Johannes Sixt wrote:

> On Montag, 26. April 2010, Johannes Schindelin wrote:
> > On Sun, 25 Apr 2010, Johannes Sixt wrote:
> > > On Sonntag, 25. April 2010, msy...@googlecode.com wrote:
> > > > Updates:
> > > > Status: WontFix
> > > >
> > > > Comment #7 on issue 457 by johannes.schindelin: Push hangs in latest
> > > > msysgit 1.7.0.2
> > > > http://code.google.com/p/msysgit/issues/detail?id=457
> > > >
> > > > Seems nobody wants to work on this issue. Tentatively marking as
> > > > WontFix
> > >

> > [...]

> >
> > > Oh, btw, this one's on my radar.
> >
> > Then you did a lousy job of advertising it. Very lousy.
>
> "Job"? "JOB"?? If I should be obliged to announce what I'm about to do,
> then I expect compensation. You should know better whether we doing any
> sort of "job" here.

You know what I meant.

> Just to make myself clear: I don't care so much whether you close any
> issues, because I mostly ignore the issuer tracker anyway; I do care
> about the tone with which it happens.
>
> Why am I still whining? Because the rudeness of your tone pisses off not
> only incompetent people, but also competent ones. It makes this place in
> the 'net a cold corner where people don't want to live.

Face it. Even when I tried to be nicer, the effect was not that people
started contributing to msysGit.

You didn't, for starters. You stayed away as far as possible, preferring
to go through git.git, graciously allowing me to pull from you.

And don't take this directness as rudeness. If I said anything wrong, you
have a right to complain about it, though.

Ciao,
Dscho

[msy...@googlecode.com]

Comment #12 on issue 457 by charleso: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

I was (foolishly) assuming that this would affect ssh as well, given that
it also
uses send-pack, which is largely why I was so worried about it. I've
finally had a
chance to test ssh with my 1.7 server and, as you say, it pushes just fine.
Does that
seem logical to you?

From my personal, and completely irrelevant, point-of-view - this is much
less of a
concern that I previously feared. Most people are almost certainly using
ssh. Our own
use of the git protocol was only a short-term arrangement.

In short, I'm all for disabling side-band-64k. It's certainly better than a
completely broken push I would have thought. I'm assuming it should/can be
done for
the git protocol only? Would you like me to have a stab at a patch?

Sorry to cause a fuss for something that is obviously now not
a 'show-shopper'. I
should have tested ssh in the beginning...

[msy...@googlecode.com]

Comment #13 on issue 457 by Johannes.Sixt: Push hangs in latest msysgit
1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

A patch would be great!

The problem certainly has to do with networking. We duplicate a socket
handle using
dup(), i.e., effectively DuplicateHandle(); this handle is inherited to the
child
process. This happens mostly by accident because the child does not do
anything
with it. To me it looks as if Windows has problems to deal with such a
handle
during the startup of the child.

With ssh or local transport, we only operate with pipes, and Windows does
not have
a problem.

[msy...@googlecode.com]

Comment #14 on issue 457 by dvornik: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

A comment from the peanut gallery: If one can actually get at the handle
underlying
the socket fd, it should be possible to prevent the handle from being
inherited by
calling SetHandleInformation(h, HANDLE_FLAG_INHERIT, 0). This is the
Windows
equivalent of setting FD_CLOEXEC on the socket.

(I'm sorry that I don't have any time to look into this myself at the
moment, and if
no one else does either, disabling side-band-64k should be the best way
forward.)

[msy...@googlecode.com]

Comment #15 on issue 457 by Johannes.Sixt: Push hangs in latest msysgit
1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

I tried your suggestion (SetHandleInformation()). It doesn't change
anything.

[msy...@googlecode.com]

Comment #16 on issue 457 by peter.is.a.geek: Push hangs in latest msysgit
1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

I still don't understand exactly what's going on, but moving the
start_async call after pack_objects avoids the problem for me (so far).

Seeing as "in" isn't passed into pack_objects, I don't see how the attached
patch could hurt anything. Is the sideband_demux thread needed for anything
before receive_status?

I don't really like to make changes like this without grokking the cause of
the problem, but at least it isn't "sleep".

Attachments:
patch 1.1 KB

[msy...@googlecode.com]

Comment #17 on issue 457 by Johannes.Sixt: Push hangs in latest msysgit
1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

sideband_demux is needed to consume error message that the server side
produces while pack_objects sends data to the server. By moving the demuxer
later, there is nobody who consumes data (error messages) sent back from
the server to the client; this could fill up network buffers and ultimately
dead-lock the communication.

I'm afraid your patch is not a solution.

BTW, patches are better submitted to the mailing list and discussed there.

[msy...@googlecode.com]

Comment #18 on issue 457 by nirfri: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

also occurs on Git-1.7.2.3-preview20100911.exe

[msy...@googlecode.com]

Comment #19 on issue 457 by alex.blewitt: Push hangs in latest msysgit
1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

This is still an ongoing issue with 1.7.2.3 on the Windows client and the
Linux server (same revision). Push hangs indefinitely. Push from Eclipse
EGit works from local (Windows) machine, and from other Linux hosts.

[msy...@googlecode.com]

Comment #20 on issue 457 by Zougloub: Push hangs in latest msysgit 1.7.0.2
http://code.google.com/p/msysgit/issues/detail?id=457

also occurs on Git-1.7.3.1-preview20101002.exe
Until someone figures out the solution, I also suggest the quick and dirty
patch:
sed -i -e 's@use_sideband = 1;@use_sideband = 0; // work-around for msysgit
bug 457@' builtin/send-pack.c

[Johannes Sixt]

Am 11.04.2013 18:12, schrieb mattden...@gmail.com:
> I'm still seeing this and it's driving me nuts.
>
> 1: This has been broken for 3 years, can we at least disable this
> use_sideband functionality until it has been fixed instead of leaving it
> in it's current state?
> 2: Can someone contact me offline with more details on this issue. If I
> can be brought up to speed, I'll take a look into it.

Great! Everything I can contribute is what you can see at
http://code.google.com/p/msysgit/issues/detail?id=457#c11 and later
comments. If you want to know more, you need to do your experiments
yourself using Procmon or some other debugging tools. Oh, and if you
want to find out the real cause, you better have your Windows source
code ready ;-)

You can also have a peek at

http://repo.or.cz/w/git/mingw/j6t.git/commitdiff/8eb3259b0046e36fe90cc51a385add0b8a5b24e9?hp=b0b00a3ee43b4813eb85728a482500f6422499fd

I'm not advocating this patch because it is like a sledgehammer and
disables too much.

-- Hannes

[Matt d'Entremont]

I'm all for avoiding the sledgehammer approach (although if that's the last option, I think it should be used if it prevents hanging; this is reproducible so frequently that I have a script which NEVER finishes on any computer using msysgit).

I'm looking into the code now, and I was wondering how far you (or anyone else that you know) has gotten while looking into this?

Given that I can reproduce this 100% of the time, and within a few minutes, if we can add a ton of logging here, I'm sure that we can get a real good idea where this is happening. Once we know that, it should be pretty easy to at the very least implement some sort of workaround.

I was also thinking that github has recently put out their own client. I assume that they have this issue as well?

[Johannes Sixt]

Am 12.04.2013 15:54, schrieb mattden...@gmail.com:
> I took a long look through:
> http://code.google.com/p/msysgit/issues/detail?id=457#c11
>
> What concerns me is that the assumption was that this only breaks when
> you are using the Git protocol, not SSH.
>
> My concern is that I'm seeing this issue when using the SSH protocol. Is
> it possible that I've run into a new issue?

Possibly. I push via ssh all the time, but it is a connection to a very
remote destination, not a host next door with a 10Gbit wire inbetween.

BTW, the hang I am talking about is visible very easily in Procmon: A
child process is started (I don't remember the details, which one it
is), and that child does not even attempt to load any DLLs, AFAIR. What
more can I say? I don't have Windows source code to see where things
could go wrong...

-- Hannes

[Philip Oakley]

From: "Johannes Sixt" <j...@kdbg.org>
Sent: Friday, April 12, 2013 7:57 PM

> --
Matt,
My desktop research of the various threads picked up these threads below
(including the above #457).

And yes, I have had it hang on ssh fetches from github, and it repeated
for a while (and still occasionally locks).

To me, it looks like one of the Linux (premature?) 'optimisations' (or
perhaps 'expectations') in the side-band-64k code is causing the problem
when we (msysgit/g4w) change over to operating with the microsoft C run
time (CRT) library.

Philip

Google :: "msysgit hangs when ssh fetch"
========================================

http://billauer.co.il/blog/2012/10/git-pull-windows-freeze-receive-pack/
Change "side-band-64k" to "side-bond-64k" so it never gets used.

http://devnet.jetbrains.net/message/5473415?tstart=0
remnants of several previous installations of Git. Sort the path
entries..

http://code.google.com/p/msysgit/issues/detail?id=74
Issue 74: git remote update/git fetch fails via ssh

http://code.google.com/p/msysgit/issues/detail?id=161
Issue 161: git-gui hangs on ssh for users with home directory mapped to
server share

http://code.google.com/p/msysgit/issues/detail?id=243
https://groups.google.com/forum/?fromgroups=#!topic/msysgit/mFnzYM3IA_4
Re: Issue 243 in msysgit: ssh hangs a minute than goes to next line
without performing action

http://code.google.com/p/msysgit/issues/detail?id=361
Issue 361: Clone/fetch/pull from github using git protocol hangs, http
works

http://code.google.com/p/msysgit/issues/detail?id=457
*** a good discussion.
Issue 457: Push over git protocol hangs in msysGit
The problematic commit is 0c499ea60f, which introduced the sideband
support in send_pack.

https://github.com/git/git/commit/0c499ea60f
also check it's child commits for comments. As well as the Msysgit fixes
(which I haven't fully researched).

[Johannes Sixt]

Am 13.04.2013 00:46, schrieb Philip Oakley:
> From: "Johannes Sixt" <j...@kdbg.org>
> Sent: Friday, April 12, 2013 7:57 PM
>> Am 12.04.2013 15:54, schrieb mattden...@gmail.com:
>>> I took a long look through:
>>> http://code.google.com/p/msysgit/issues/detail?id=457#c11
>>>
>>> What concerns me is that the assumption was that this only breaks when
>>> you are using the Git protocol, not SSH.
>>>
>>> My concern is that I'm seeing this issue when using the SSH protocol. Is
>>> it possible that I've run into a new issue?
>>
>> Possibly. I push via ssh all the time, but it is a connection to a very
>> remote destination, not a host next door with a 10Gbit wire inbetween.
>>
>> BTW, the hang I am talking about is visible very easily in Procmon: A
>> child process is started (I don't remember the details, which one it
>> is), and that child does not even attempt to load any DLLs, AFAIR. What
>> more can I say? I don't have Windows source code to see where things
>> could go wrong...
>>
>> -- Hannes
>>
>> --
> Matt,
> My desktop research of the various threads picked up these threads below
> (including the above #457).
>
> And yes, I have had it hang on ssh fetches from github, and it repeated
> for a while (and still occasionally locks).

Issue #457 is purely about push via native git protocol.

-- Hannes

[Philip Oakley]

From: "Johannes Sixt" <j...@kdbg.org>
Sent: Saturday, April 13, 2013 8:19 AM

> Am 13.04.2013 00:46, schrieb Philip Oakley:
>> From: "Johannes Sixt" <j...@kdbg.org>
>> Sent: Friday, April 12, 2013 7:57 PM
>>> Am 12.04.2013 15:54, schrieb mattden...@gmail.com:
>>>> I took a long look through:
>>>> http://code.google.com/p/msysgit/issues/detail?id=457#c11
>>>>
>>>> What concerns me is that the assumption was that this only breaks
>>>> when
>>>> you are using the Git protocol, not SSH.
>>>>

[...]

>> Matt,
>> My desktop research of the various threads picked up these threads
>> below
>> (including the above #457).
>>
>> And yes, I have had it hang on ssh fetches from github, and it
>> repeated
>> for a while (and still occasionally locks).
>
> Issue #457 is purely about push via native git protocol.
>
> -- Hannes

Thank you for the clarification. I've added it to my notes should I get
a proper chance to investigate among all my other tasks.
Philip