Re: Fwd: Git hangs after resolving deltas when using NTLM proxy on Windows

[Heiko Voigt]

Hi,

On Fri, Jan 04, 2013 at 11:28:44AM +1100, Adam Baxter wrote:
> Git is authenticating to my corporate proxy correctly, but is hanging
> after "resolving deltas".
>
> This is when cloning via HTTPS.
>
> Apologies for linking to a Gist, but I'm having difficulty getting
> this list to accept a log file attachment.
> See https://gist.github.com/4448684 for the curl log.

I am not sure whether curl supports NTLM proxies. How about using
something like

http://ntlmaps.sourceforge.net/

instead? I use that on Windows to get outside with HTTP(S) at $dayjob.

Cheers Heiko

P.S.: BTW, since its Windows specific I moved this topic to the msysgit
mailing list.

[Adam Baxter]

Hi Heiko,

> I am not sure whether curl supports NTLM proxies. How about using
> something like
>
> http://ntlmaps.sourceforge.net/
>

Git/Curl supports NTLM for a long time now via the SSPI build of Curl.

See https://github.com/gitster/git/commit/dd6139971a18e25a5089c0f96dc80e454683ef0b

NTLMAPS and cNTLM are great until you need to use smartcard
authentication - you don't necessarily know the real password behind
the card. Git needs to pass one more option to Curl for "passwordless"
authentication but I'd like to get this bug fixed first.

> P.S.: BTW, since its Windows specific I moved this topic to the msysgit
> mailing list.

That's fine, but we'll eventually have to send the fix upstream, too.

Thanks,
Adam

[Johannes Schindelin]

Hi Heiko,

On Sun, 6 Jan 2013, Heiko Voigt wrote:

> On Fri, Jan 04, 2013 at 11:28:44AM +1100, Adam Baxter wrote:
> > Git is authenticating to my corporate proxy correctly, but is hanging
> > after "resolving deltas".
> >
> > This is when cloning via HTTPS.
> >
> > Apologies for linking to a Gist, but I'm having difficulty getting
> > this list to accept a log file attachment. See
> > https://gist.github.com/4448684 for the curl log.
>
> I am not sure whether curl supports NTLM proxies.

According to http://curl.haxx.se/docs/manpage.html#--ntlm it does. And
IIRC we upgraded curl in msysGit explicitly to allow for NTLM
authentication. See
https://github.com/msysgit/msysgit/commit/5361bdf5992789fccba09f58a04d808e4a062fad

Ciao,
Dscho

[Adam Baxter]

Also, should I open an issue on GitHub for this?

[Heiko Voigt]

On Sun, Jan 06, 2013 at 04:49:04PM -0800, Adam Baxter wrote:
> Also, should I open an issue on GitHub for this?

If you want you can but that does not mean anyone is going to work on
this. The best would be if you work on this, fix it and send a patch /
pull request with the fix. We can help you to get enough information so
you can do so.

Cheers Heiko

[Adam Baxter]

> We can help you to get enough information so you can do so.

At this stage I am stuck on being able to debug Git with GDB on Windows. I don't get any useful information in the backtrace.

(gdb) bt
#0 0x77ca000d in ntdll!LdrFindResource_U () from C:\Windows\SysWOW64\ntdll.dll
#1 0x77d2f896 in ntdll!RtlQueryTimeZoneInformation () from C:\Windows\SysWOW64\ntdll.dll
#2 0x7a978224 in ?? ()
#3 0x00000000 in ?? ()

[Johannes Schindelin]

Hi Adam,

Have you looked at other threads ("info threads")? I imagine that the hang
is not in the thread whose trace we see above.

Ciao,
Johannes

[Adam Baxter]

Have you looked at other threads ("info threads")? I imagine that the hang
is not in the thread whose trace we see above.

This got me further (also, I managed to compile git with symbols finally).

Still not great though:

(gdb) info threads

  3 thread 10084.0x1644  0x7769000d in ntdll!LdrFindResource_U ()

   from C:\Windows\SysWOW64\ntdll.dll

* 2 thread 10084.0x22a0  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()

   from C:\Windows\SysWOW64\ntdll.dll

  1 thread 10084.0x1898  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()

   from C:\Windows\SysWOW64\ntdll.dll

(gdb) help info threads

IDs of currently known threads.

(gdb) t 1

[Switching to thread 1 (thread 10084.0x1898)]#0  0x7769f8e5 in ntdll!RtlUpdateCl

onedSRWLock () from C:\Windows\SysWOW64\ntdll.dll

(gdb) bt

#0  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()

   from C:\Windows\SysWOW64\ntdll.dll

#1  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()

   from C:\Windows\SysWOW64\ntdll.dll

#2  0x764cd348 in ReadFile () from C:\Windows\syswow64\KernelBase.dll

#3  0x0000009c in ?? ()

#4  0x00000000 in ?? ()

(gdb) t 2

[Switching to thread 2 (thread 10084.0x22a0)]#0  0x7769f8e5 in ntdll!RtlUpdateCl

onedSRWLock () from C:\Windows\SysWOW64\ntdll.dll

(gdb) bt

#0  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()

   from C:\Windows\SysWOW64\ntdll.dll

#1  0x7769f8e5 in ntdll!RtlUpdateClonedSRWLock ()

   from C:\Windows\SysWOW64\ntdll.dll

#2  0x764cd348 in ReadFile () from C:\Windows\syswow64\KernelBase.dll

#3  0x00000074 in ?? ()

#4  0x00000000 in ?? ()

(gdb) t 3

[Switching to thread 3 (thread 10084.0x1644)]#0  0x7769000d in ntdll!LdrFindReso

urce_U () from C:\Windows\SysWOW64\ntdll.dll

(gdb) bt

#0  0x7769000d in ntdll!LdrFindResource_U ()

   from C:\Windows\SysWOW64\ntdll.dll

#1  0x7771f896 in ntdll!RtlQueryTimeZoneInformation ()

   from C:\Windows\SysWOW64\ntdll.dll

#2  0x7130a1aa in ?? ()

#3  0x00000000 in ?? ()

(gdb)

 

[Adam Baxter]

Note that I am now debugging git-remote-ftps.exe - sorry, I forgot to mention this earlier.

The stack from the windows side of things: (from here, I can see curl's thread)

wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0

wow64cpu.dll!TurboDispatchJumpAddressEnd+0x56b

wow64.dll!Wow64SystemServiceEx+0x1ce

wow64.dll!Wow64LdrpInitialize+0x429

ntdll.dll!RtlUniform+0x6e6

ntdll.dll!RtlCreateTagHeap+0xa7

ntdll.dll!LdrInitializeThunk+0xe

ntdll.dll!ZwDelayExecution+0x15

KERNELBASE.dll!Sleep+0xf

libcurl-4.dll!curl_getdate+0x41

libcurl-4.dll!curl_getdate+0x4f6

libcurl-4.dll!curl_maprintf+0x49b0

git-remote-https.exe+0x1f8f

git-remote-https.exe+0x273c

git-remote-https.exe+0x2993

git-remote-https.exe+0x2dd3

git-remote-https.exe+0x2e5d

git-remote-https.exe+0x3055

git-remote-https.exe+0x3786

git-remote-https.exe+0x35ae

git-remote-https.exe+0x124b

git-remote-https.exe+0x1298

ntdll.dll!RtlInitializeExceptionChain+0x63

ntdll.dll!RtlInitializeExceptionChain+0x36

It turns out that curl is always loaded in thread 1 according to GDB, but I still can't see it there, even after issuing 'dll-symbols'.

[Oliver Schindler]

Since this problem is also bugging me, I tried to locate it.

 

I discovered that cloning works with a NTLM proxy if I recompile git with USE_CURL_MULTI disabled. While further hunting down the problem, it looks like call to curl_easy_perform() from run_slot() never returns. To me it looks like some wired curl behavior after a POST when a NTLM proxy is used. I guess it's somehow related to the "rewindaftersend" flag.

 

Are multi connections even supported with NTLM proxies?

 

>>curl_easy_perform
* Couldn't find host github.com in the _netrc file; using defaults
* About to connect() to proxy <PROXY> port <PORT> (#0)
*   Trying <IP>...
* connected
* Connected to <PROXY> (<IP>) port <PORT> (#0)
* Establish HTTP proxy tunnel to github.com:443
* Proxy auth using NTLM with user ''
> CONNECT github.com:443 HTTP/1.1
Host: github.com:443
Proxy-Authorization: NTLM <TOKEN>
User-Agent: git/1.8.1.msysgit.1.dirty
Proxy-Connection: Keep-Alive
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result

< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NTLM <TOKEN>
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: Keep-Alive
< Set-Cookie: <COOKIE>; Path=/
< Connection: Keep-Alive
< Content-Length: 838
<
* Ignore 838 bytes of response-body
* Rewind stream after send
* TUNNEL_STATE switched to: 0
* Establish HTTP proxy tunnel to github.com:443
* Proxy auth using NTLM with user '<USER>'
> CONNECT github.com:443 HTTP/1.1
Host: github.com:443
Proxy-Authorization: NTLM <TOKEN>
User-Agent: git/1.8.1.msysgit.1.dirty
Proxy-Connection: Keep-Alive
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result

< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: C:\msysgit/mingw/bin/curl-ca-bundle.crt
  CApath: none
* SSL re-using session ID
* SSL connection using RC4-SHA
* old SSL session ID is stale, removing
* Server certificate:
*        subject: businessCategory=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=California; serialNumber=C3268102; C=US; ST=Calif
ornia; L=San Francisco; O=GitHub, Inc.; CN=github.com
*        start date: 2011-05-27 00:00:00 GMT
*        expire date: 2013-07-29 12:00:00 GMT
*        subjectAltName: github.com matched
*        issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance EV CA-1
*        SSL certificate verify ok.
> POST /AutoMapper/AutoMapper.git/git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.msysgit.1.dirty
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 806

* upload completely sent off: 806 out of 806 bytes
< HTTP/1.1 200 OK
< Server: GitHub.com
< Date: Mon, 18 Mar 2013 13:57:25 GMT
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Connection: keep-alive
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
* Keep sending data to get tossed away!
<
remote: Counting objects: 5913, done.
remote: Compressing objects: 100% (2204/2204), done.
remote: Total 5913 (delta 4037), reused 5465 (delta 3629)
Receiving objects: 100% (5913/5913), 39.92 MiB | 603 KiB/s, done.
Resolving deltas: 100% (4037/4037), done.

 

[Oliver Schindler]

As speculated I believe the POST over HTTPS hangs after authenticating to a NTLM proxy because the "rewindaftersend" flag isn't cleared for the document request.
But we really need someone from curl to have a look at it. The attached change seems to fix the problem. But I have absolutely no clue what side effects it could have.

 

diff --git a/lib/http_proxy.c b/lib/http_proxy.c
index 245e1c1..1e53169 100644
--- a/lib/http_proxy.c
+++ b/lib/http_proxy.c
@@ -594,6 +594,7 @@ CURLcode Curl_proxyCONNECT(struct connectdata *conn,
 
   infof (data, "Proxy replied OK to CONNECT request\n");
   data->req.ignorebody = FALSE; /* put it (back) to non-ignore state */
+  conn->bits.rewindaftersend = FALSE; /* make sure this isn't set for the document request  */
   return CURLE_OK;
 }
 #endif /* CURL_DISABLE_PROXY */

[Johannes Schindelin]

Hi Daniel,

could you have a look at this patch, please?

Thanks,
Johannes

> --
> --
> *** Please reply-to-all at all times ***
> *** (do not pretend to know who is subscribed and who is not) ***
> *** Please avoid top-posting. ***
> The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.
>
> You received this message because you are subscribed to the Google
> Groups "msysGit" group.
> To post to this group, send email to msy...@googlegroups.com
> To unsubscribe from this group, send email to
> msysgit+u...@googlegroups.com
> For more options, and view previous threads, visit this group at
> http://groups.google.com/group/msysgit?hl=en_US?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "msysGit" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

[Konstantin Khomoutov]

On Mon, 18 Mar 2013 09:57:00 -0700 (PDT)
Oliver Schindler <schin...@gmail.com> wrote:

> As speculated I believe the POST over HTTPS hangs after
> authenticating to a NTLM proxy because the "rewindaftersend" flag
> isn't cleared for the document request.
> But we really need someone from curl to have a look at it. The
> attached change seems to fix the problem. But I have absolutely no
> clue what side effects it could have.

Looks like a good candidate for discussing with the cURL upstream IMO.

> diff --git a/lib/http_proxy.c b/lib/http_proxy.c
[...]

[Daniel Stenberg]

On Mon, 18 Mar 2013, Johannes Schindelin wrote:

> Hi Daniel,
>
> could you have a look at this patch, please?

Thanks! I read your thread and I can see how this might confuse libcurl at
times. I've not managed to write up a test case for this so I didn't bother to
move this initialization to another place but I did verify that all existing
tests run fine still.

I'll merge and push this to curl's git just now (as commit d1d0ee075f).

--

/ daniel.haxx.se

[Oliver Schindler]

Thanks Daniel.

 

I did run into the problem when git was compiled with USE_CURL_MULTI

enabled. Which of course is the default with newer libcurl versions.

When libcurl is set up for one connection it seems not to bother with

rewinding the connection. Maybe this helps to create a test case.