Re: Fwd: Git hangs after resolving deltas when using NTLM proxy on Windows

1554 views
Skip to first unread message

Heiko Voigt

unread,
Jan 6, 2013, 11:46:59 AM1/6/13
to Adam Baxter, msysGit Mailinglist
Hi,

On Fri, Jan 04, 2013 at 11:28:44AM +1100, Adam Baxter wrote:
> Git is authenticating to my corporate proxy correctly, but is hanging
> after "resolving deltas".
>
> This is when cloning via HTTPS.
>
> Apologies for linking to a Gist, but I'm having difficulty getting
> this list to accept a log file attachment.
> See https://gist.github.com/4448684 for the curl log.

I am not sure whether curl supports NTLM proxies. How about using
something like

http://ntlmaps.sourceforge.net/

instead? I use that on Windows to get outside with HTTP(S) at $dayjob.

Cheers Heiko

P.S.: BTW, since its Windows specific I moved this topic to the msysgit
mailing list.

Adam Baxter

unread,
Jan 6, 2013, 4:17:04 PM1/6/13
to Heiko Voigt, msysGit Mailinglist
Hi Heiko,

> I am not sure whether curl supports NTLM proxies. How about using
> something like
>
> http://ntlmaps.sourceforge.net/
>
Git/Curl supports NTLM for a long time now via the SSPI build of Curl.

See https://github.com/gitster/git/commit/dd6139971a18e25a5089c0f96dc80e454683ef0b

NTLMAPS and cNTLM are great until you need to use smartcard
authentication - you don't necessarily know the real password behind
the card. Git needs to pass one more option to Curl for "passwordless"
authentication but I'd like to get this bug fixed first.

> P.S.: BTW, since its Windows specific I moved this topic to the msysgit
> mailing list.

That's fine, but we'll eventually have to send the fix upstream, too.

Thanks,
Adam

Johannes Schindelin

unread,
Jan 6, 2013, 6:08:55 PM1/6/13
to Heiko Voigt, Adam Baxter, msysGit Mailinglist
Hi Heiko,

On Sun, 6 Jan 2013, Heiko Voigt wrote:

> On Fri, Jan 04, 2013 at 11:28:44AM +1100, Adam Baxter wrote:
> > Git is authenticating to my corporate proxy correctly, but is hanging
> > after "resolving deltas".
> >
> > This is when cloning via HTTPS.
> >
> > Apologies for linking to a Gist, but I'm having difficulty getting
> > this list to accept a log file attachment. See
> > https://gist.github.com/4448684 for the curl log.
>
> I am not sure whether curl supports NTLM proxies.

According to http://curl.haxx.se/docs/manpage.html#--ntlm it does. And
IIRC we upgraded curl in msysGit explicitly to allow for NTLM
authentication. See
https://github.com/msysgit/msysgit/commit/5361bdf5992789fccba09f58a04d808e4a062fad

Ciao,
Dscho

Adam Baxter

unread,
Jan 6, 2013, 7:49:04 PM1/6/13
to msy...@googlegroups.com
Also, should I open an issue on GitHub for this?

Heiko Voigt

unread,
Jan 7, 2013, 11:45:45 AM1/7/13
to Adam Baxter, msy...@googlegroups.com
On Sun, Jan 06, 2013 at 04:49:04PM -0800, Adam Baxter wrote:
> Also, should I open an issue on GitHub for this?

If you want you can but that does not mean anyone is going to work on
this. The best would be if you work on this, fix it and send a patch /
pull request with the fix. We can help you to get enough information so
you can do so.

Cheers Heiko

Adam Baxter

unread,
Jan 16, 2013, 12:36:07 AM1/16/13
to msy...@googlegroups.com, Adam Baxter
> We can help you to get enough information so you can do so.

At this stage I am stuck on being able to debug Git with GDB on Windows. I don't get any useful information in the backtrace.

(gdb) bt
#0 0x77ca000d in ntdll!LdrFindResource_U () from C:\Windows\SysWOW64\ntdll.dll
#1 0x77d2f896 in ntdll!RtlQueryTimeZoneInformation () from C:\Windows\SysWOW64\ntdll.dll
#2 0x7a978224 in ?? ()
#3 0x00000000 in ?? ()

Johannes Schindelin

unread,
Jan 16, 2013, 10:05:43 AM1/16/13
to Adam Baxter, msy...@googlegroups.com
Hi Adam,
Have you looked at other threads ("info threads")? I imagine that the hang
is not in the thread whose trace we see above.

Ciao,
Johannes

Adam Baxter

unread,
Jan 17, 2013, 10:45:42 PM1/17/13
to msy...@googlegroups.com, Adam Baxter

Have you looked at other threads ("info threads")? I imagine that the hang
is not in the thread whose trace we see above.


This got me further (also, I managed to compile git with symbols finally).
Still not great though:

(gdb) info threads
  3 thread 10084.0x1644  0x7769000d in ntdll!LdrFindResource_U ()
   from C:\Windows\SysWOW64\ntdll.dll
* 2 thread 10084.0x22a0  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()
   from C:\Windows\SysWOW64\ntdll.dll
  1 thread 10084.0x1898  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()
   from C:\Windows\SysWOW64\ntdll.dll
(gdb) help info threads
IDs of currently known threads.
(gdb) t 1
[Switching to thread 1 (thread 10084.0x1898)]#0  0x7769f8e5 in ntdll!RtlUpdateCl
onedSRWLock () from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt
#0  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()
   from C:\Windows\SysWOW64\ntdll.dll
#1  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()
   from C:\Windows\SysWOW64\ntdll.dll
#2  0x764cd348 in ReadFile () from C:\Windows\syswow64\KernelBase.dll
#3  0x0000009c in ?? ()
#4  0x00000000 in ?? ()
(gdb) t 2
[Switching to thread 2 (thread 10084.0x22a0)]#0  0x7769f8e5 in ntdll!RtlUpdateCl
onedSRWLock () from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt
#0  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()
   from C:\Windows\SysWOW64\ntdll.dll
#1  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()
   from C:\Windows\SysWOW64\ntdll.dll
#2  0x764cd348 in ReadFile () from C:\Windows\syswow64\KernelBase.dll
#3  0x00000074 in ?? ()
#4  0x00000000 in ?? ()
(gdb) t 3
[Switching to thread 3 (thread 10084.0x1644)]#0  0x7769000d in ntdll!LdrFindReso
urce_U () from C:\Windows\SysWOW64\ntdll.dll
(gdb) bt
#0  0x7769000d in ntdll!LdrFindResource_U ()
   from C:\Windows\SysWOW64\ntdll.dll
#1  0x7771f896 in ntdll!RtlQueryTimeZoneInformation ()
   from C:\Windows\SysWOW64\ntdll.dll
#2  0x7130a1aa in ?? ()
#3  0x00000000 in ?? ()
(gdb)
 

Adam Baxter

unread,
Jan 18, 2013, 12:42:49 AM1/18/13
to msy...@googlegroups.com, Adam Baxter
Note that I am now debugging git-remote-ftps.exe - sorry, I forgot to mention this earlier.

The stack from the windows side of things: (from here, I can see curl's thread)

wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x56b
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64LdrpInitialize+0x429
ntdll.dll!RtlUniform+0x6e6
ntdll.dll!RtlCreateTagHeap+0xa7
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!ZwDelayExecution+0x15
KERNELBASE.dll!Sleep+0xf
libcurl-4.dll!curl_getdate+0x41
libcurl-4.dll!curl_getdate+0x4f6
libcurl-4.dll!curl_maprintf+0x49b0
git-remote-https.exe+0x1f8f
git-remote-https.exe+0x273c
git-remote-https.exe+0x2993
git-remote-https.exe+0x2dd3
git-remote-https.exe+0x2e5d
git-remote-https.exe+0x3055
git-remote-https.exe+0x3786
git-remote-https.exe+0x35ae
git-remote-https.exe+0x124b
git-remote-https.exe+0x1298
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36

It turns out that curl is always loaded in thread 1 according to GDB, but I still can't see it there, even after issuing 'dll-symbols'.

Oliver Schindler

unread,
Mar 18, 2013, 10:37:22 AM3/18/13
to msy...@googlegroups.com, Adam Baxter
Since this problem is also bugging me, I tried to locate it.
 
I discovered that cloning works with a NTLM proxy if I recompile git with USE_CURL_MULTI disabled. While further hunting down the problem, it looks like call to curl_easy_perform() from run_slot() never returns. To me it looks like some wired curl behavior after a POST when a NTLM proxy is used. I guess it's somehow related to the "rewindaftersend" flag.
 
Are multi connections even supported with NTLM proxies?
 
>>curl_easy_perform
* Couldn't find host github.com in the _netrc file; using defaults
* About to connect() to proxy <PROXY> port <PORT> (#0)
*   Trying <IP>...
* connected
* Connected to <PROXY> (<IP>) port <PORT> (#0)
* Establish HTTP proxy tunnel to github.com:443
* Proxy auth using NTLM with user ''
> CONNECT github.com:443 HTTP/1.1
Host: github.com:443
Proxy-Authorization: NTLM <TOKEN>
User-Agent: git/1.8.1.msysgit.1.dirty
Proxy-Connection: Keep-Alive
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NTLM <TOKEN>
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: Keep-Alive
< Set-Cookie: <COOKIE>; Path=/
< Connection: Keep-Alive
< Content-Length: 838
<
* Ignore 838 bytes of response-body
* Rewind stream after send
* TUNNEL_STATE switched to: 0
* Establish HTTP proxy tunnel to github.com:443
* Proxy auth using NTLM with user '<USER>'
> CONNECT github.com:443 HTTP/1.1
Host: github.com:443
Proxy-Authorization: NTLM <TOKEN>
User-Agent: git/1.8.1.msysgit.1.dirty
Proxy-Connection: Keep-Alive
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: C:\msysgit/mingw/bin/curl-ca-bundle.crt
  CApath: none
* SSL re-using session ID
* SSL connection using RC4-SHA
* old SSL session ID is stale, removing
* Server certificate:
*        subject: businessCategory=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=California; serialNumber=C3268102; C=US; ST=Calif
ornia; L=San Francisco; O=GitHub, Inc.; CN=github.com
*        start date: 2011-05-27 00:00:00 GMT
*        expire date: 2013-07-29 12:00:00 GMT
*        subjectAltName: github.com matched
*        issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance EV CA-1
*        SSL certificate verify ok.
> POST /AutoMapper/AutoMapper.git/git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.msysgit.1.dirty
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 806
* upload completely sent off: 806 out of 806 bytes
< HTTP/1.1 200 OK
< Server: GitHub.com
< Date: Mon, 18 Mar 2013 13:57:25 GMT
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Connection: keep-alive
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
* Keep sending data to get tossed away!
<
remote: Counting objects: 5913, done.
remote: Compressing objects: 100% (2204/2204), done.
remote: Total 5913 (delta 4037), reused 5465 (delta 3629)
Receiving objects: 100% (5913/5913), 39.92 MiB | 603 KiB/s, done.
Resolving deltas: 100% (4037/4037), done.
 

Oliver Schindler

unread,
Mar 18, 2013, 12:57:00 PM3/18/13
to msy...@googlegroups.com, Adam Baxter
As speculated I believe the POST over HTTPS hangs after authenticating to a NTLM proxy because the "rewindaftersend" flag isn't cleared for the document request.
But we really need someone from curl to have a look at it. The attached change seems to fix the problem. But I have absolutely no clue what side effects it could have.
 
diff --git a/lib/http_proxy.c b/lib/http_proxy.c
index 245e1c1..1e53169 100644
--- a/lib/http_proxy.c
+++ b/lib/http_proxy.c
@@ -594,6 +594,7 @@ CURLcode Curl_proxyCONNECT(struct connectdata *conn,
 
   infof (data, "Proxy replied OK to CONNECT request\n");
   data->req.ignorebody = FALSE; /* put it (back) to non-ignore state */
+  conn->bits.rewindaftersend = FALSE; /* make sure this isn't set for the document request  */
   return CURLE_OK;
 }
 #endif /* CURL_DISABLE_PROXY */

Johannes Schindelin

unread,
Mar 18, 2013, 1:25:43 PM3/18/13
to dan...@haxx.se, Oliver Schindler, msy...@googlegroups.com, Adam Baxter
Hi Daniel,

could you have a look at this patch, please?

Thanks,
Johannes
> --
> --
> *** Please reply-to-all at all times ***
> *** (do not pretend to know who is subscribed and who is not) ***
> *** Please avoid top-posting. ***
> The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.
>
> You received this message because you are subscribed to the Google
> Groups "msysGit" group.
> To post to this group, send email to msy...@googlegroups.com
> To unsubscribe from this group, send email to
> msysgit+u...@googlegroups.com
> For more options, and view previous threads, visit this group at
> http://groups.google.com/group/msysgit?hl=en_US?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "msysGit" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

Konstantin Khomoutov

unread,
Mar 18, 2013, 1:29:55 PM3/18/13
to Oliver Schindler, msy...@googlegroups.com, Adam Baxter
On Mon, 18 Mar 2013 09:57:00 -0700 (PDT)
Oliver Schindler <schin...@gmail.com> wrote:

> As speculated I believe the POST over HTTPS hangs after
> authenticating to a NTLM proxy because the "rewindaftersend" flag
> isn't cleared for the document request.
> But we really need someone from curl to have a look at it. The
> attached change seems to fix the problem. But I have absolutely no
> clue what side effects it could have.

Looks like a good candidate for discussing with the cURL upstream IMO.

> diff --git a/lib/http_proxy.c b/lib/http_proxy.c
[...]

Daniel Stenberg

unread,
Mar 18, 2013, 5:53:55 PM3/18/13
to Johannes Schindelin, Oliver Schindler, msy...@googlegroups.com, Adam Baxter
On Mon, 18 Mar 2013, Johannes Schindelin wrote:

> Hi Daniel,
>
> could you have a look at this patch, please?

Thanks! I read your thread and I can see how this might confuse libcurl at
times. I've not managed to write up a test case for this so I didn't bother to
move this initialization to another place but I did verify that all existing
tests run fine still.

I'll merge and push this to curl's git just now (as commit d1d0ee075f).

--

/ daniel.haxx.se

Oliver Schindler

unread,
Mar 19, 2013, 4:50:46 AM3/19/13
to msy...@googlegroups.com, Johannes Schindelin, Oliver Schindler, Adam Baxter, dan...@haxx.se
Thanks Daniel.
 
I did run into the problem when git was compiled with USE_CURL_MULTI
enabled. Which of course is the default with newer libcurl versions.
When libcurl is set up for one connection it seems not to bother with
rewinding the connection. Maybe this helps to create a test case.
 
Reply all
Reply to author
Forward
0 new messages