Potential curl keep-alive bug in Jenkins

[Micks80]

Okay, after pulling out half of my hair, it seems like this is a bug
on Jenkins side. Here's the scenario:

I have created a job in Jenkins to pull code from a remote Git repo.
The job is unable to pull code and
gets stuck at the following clone step -

git clone -o origin http://<remote_server_url>/project.git <Workspace>

But when I try to clone it using Tortoise Git, it works fine. The
Tortoise Git uses the following command to clone:
git clone --progress -v http://<remote_server_url>/project.git
<Workspace>

So, I created a new job with the same command used by Tortoise Git and
bang, it works !!!


I examined the header and it looks like when "--progress -v" is not
passed, Jenkins (or Curl) is closing the connection
too fast (Connection: close) i.e.



* About to connect() to <server_url> port 80 (#0)
* Trying xx.xx.xx.xx... * 0x8e3300 is at send pipe head!
* Connected to <server_url> (xx.xx.xx.xx) port 80 (#0)
> GET /project.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.4.3282.g564cb
Host: <server_url>
Accept: */*
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Wed, 16 Mar 2011 14:54:48 GMT
< Server: Apache/2.2.3 (Red Hat)
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/x-git-upload-pack-advertisement
<
* Expire cleared
* Closing connection #0
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
* About to connect() to <server_url> port 80 (#0)
* Trying xx.xx.xx.xx... * connected
* Connected to <server_url> (xx.xx.xx.xx) port 80 (#0)
> POST /project.git/git-upload-pack HTTP/1.1
User-Agent: git/1.7.4.3282.g844cb
Host: <server_url>
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 178

< HTTP/1.1 200 OK
< Date: Wed, 16 Mar 2011 14:54:48 GMT
< Server: Apache/2.2.3 (Red Hat)
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/x-git-upload-pack-result
<
* transfer closed with outstanding read data remaining
* Closing connection #0
error: RPC failed; result=18, HTTP code = 200
warning: http unexpectedly said: '0000'



So, can some-one please tell how to set the header so that it can be
changed to "Connection: keep-alive" in my job/Jenkins or how to pass
the "--progress -v" in the Git plugin call from Jenkins to get this
issue resolved on my end asap.

Thanks
Mick

[Micks80]

***bump***

Can someone please confirm/deny the fact or provide any kind of
workaround/alternative to resolve this situation???

Thanks

[Mark Waite]

My use of the git plugin works on 20+ different machines including various Linux and Windows variants.  That doesn't mean the problem you're describing doesn't exist, just that I don't see the problem you're describing and I don't remember ever hearing mention of that sort of problem on the mailing list before.

If you see the same behavior from the git command line, then you probably have detected an issue with git.

If you don't see the same behavior from the git command line, then I'm quite surprised and don't have any other suggestions to offer.

 Mark Waite

---

From: Micks80 <mic...@gmail.com>
To: Jenkins Users <jenkins...@googlegroups.com>
Sent: Thu, March 17, 2011 8:15:32 AM
Subject: Re: Potential curl keep-alive bug in Jenkins

[Micks80]

Hi Mark,
Finally, it was great to see a reply from someone :)

You have raised a good point but I am unable to test it on command
line because Git plugin/Jenkins sets up some extra internal parameters
e.g. "--stateless-rpc" before making the call as shown below:

C:\workspace>git clone http://<server_url>/project.git C:\Projects\
trace: built-in: git 'clone' 'http://<server_url>/project.git' 'C:
\Projects\'
trace: run_command: 'git-remote-http' 'origin' 'http://<server_url>/
project.git'
Cloning into C:\Projects\...
trace: run_command: 'fetch-pack' '--stateless-rpc' '--lock-pack' '--
thin' '--no-progress' 'http://<server_url>/project.git' 'refs/heads/
master' 'refs/heads/m3/master'

When I try to pass those parameters on command line, its throws error
saying that they don't exist.
The Git Plugin/Jenkins should allow the user to set these parameters
manually as well..
Not sure what else to try here..

Thanks

On Mar 18, 6:42 am, Mark Waite <markwa...@yahoo.com> wrote:
> My use of the git plugin works on 20+ different machines including various Linux
> and Windows variants.  That doesn't mean the problem you're describing doesn't
> exist, just that I don't see the problem you're describing and I don't remember
> ever hearing mention of that sort of problem on the mailing list before.
>
> If you see the same behavior from the git command line, then you probably have
> detected an issue with git.
>
> If you don't see the same behavior from the git command line, then I'm quite
> surprised and don't have any other suggestions to offer.
>
>  Mark Waite
>
> ________________________________

> From: Micks80 <mick...@gmail.com>

> > Mick- Hide quoted text -
>
> - Show quoted text -

[Mark Waite]

I don't think the plugin is setting extra parameters, or at least it does not seem to set extra parameters on my invocation of git. 

 

I think the parameters you are seeing there are likely coming from "inside git" as it calls its own git internal operations.  I believe "git clone" would be considered "porcelain" (a user interface component), while "git fetch-pack" is more "plumbing" than "porcelain".

 

One other alternative you could consider is to use the more efficient git protocol for your clone rather than using the http protocol.  I've used both git and http protocol successfully, but since I'm hosting my own repositories, I've preferred the git protocol for cloning.

 

Another alternative might be to attempt to clone from a public repository (like github) to see if the problem is with the specific repository you're trying to clone, or with the Jenkins plugin.

 

Thanks,

Mark Waite

---

From: Micks80 <mic...@gmail.com>
To: Jenkins Users <jenkins...@googlegroups.com>
Sent: Fri, March 18, 2011 9:46:27 AM

[Micks80]

On Mar 18, 1:15 pm, Mark Waite <markwa...@yahoo.com> wrote:
> I don't think the plugin is setting extra parameters, or at least it does not
> seem to set extra parameters on my invocation of git. 

You need to set the following environment variables to view the
invocations:
set GIT_CURL_VERBOSE to 1
set GIT_TRACE to 1

>
> I think the parameters you are seeing there are likely coming from "inside git"
> as it calls its own git internal operations.  I believe "git clone" would be
> considered "porcelain" (a user interface component), while "git fetch-pack" is
> more "plumbing" than "porcelain".
>
> One other alternative you could consider is to use the more efficient git
> protocol for your clone rather than using the http protocol.  I've used both git
> and http protocol successfully, but since I'm hosting my own repositories, I've
> preferred the git protocol for cloning.

Our build system account only has read access over http, so I cannot
use ssh..

>
> Another alternative might be to attempt to clone from a public repository (like
> github) to see if the problem is with the specific repository you're trying to
> clone, or with the Jenkins plugin.
>

I did not tried github but jenkins is able to fetch smaller
repositories from the same server.
This repo is big (15 GB) and that's where it's choking, how big are
your repositories?

As a workaround, I have created the initial clone using Tortoise Git
and then using Jenkins
to fetch further delta's and it's working fine. It's only failing
during the first clone step...


> Thanks,
> Mark Waite
>

I have also posted it in the Git mailing list, so lets see...

Thanks again !!! ________________________________

> > - Show quoted text -- Hide quoted text -

[Mark Waite]

> From: Micks80 <mic...@gmail.com>

> To: Jenkins Users <jenkins...@googlegroups.com>

> Sent: Fri, March 18, 2011 11:33:14 AM

> Subject: Re: Potential curl keep-alive bug in Jenkins

>

>

>

> On Mar 18, 1:15 pm, Mark Waite <markwa...@yahoo.com> wrote:

> > I don't think the plugin is setting extra parameters, or at least

> > it does not seem to set extra parameters on my invocation of git.

>

> You need to set the following environment variables to view the

> invocations:

> set GIT_CURL_VERBOSE to 1

> set GIT_TRACE to 1

I believe the parameters you're seeing are from inside git, as the

porcelain passes arguments to the plumbing commands.  As far as I

know, you can't pass internal arguments (like you see from your

traces) through the command line.

> >

> > I think the parameters you are seeing there are likely coming from

> > "inside git" as it calls its own git internal operations.  I

> > believe "git clone" would be considered "porcelain" (a user

> > interface component), while "git fetch-pack" is more "plumbing"

> > than "porcelain".

> >

> > One other alternative you could consider is to use the more

> > efficient git protocol for your clone rather than using the http

> > protocol.  I've used both git and http protocol successfully, but

> > since I'm hosting my own repositories, I've preferred the git

> > protocol for cloning.

>

> Our build system account only has read access over http, so I cannot

> use ssh..

If you allow read access over http, you probably also could consider

providing read access through the git:// protocol.  For example, I

have a Debian Linux server where I can access repositories with the

URL git://mwaite1.ptcnet.ptc.com/git/sample.

That does not require opening the machine for ssh access, nor does it

require that Jenkins use ssh to access the repository.

> >

> > Another alternative might be to attempt to clone from a public

> > repository (like github) to see if the problem is with the

> > specific repository you're trying to clone, or with the Jenkins

> > plugin.

> >

> I did not tried github but jenkins is able to fetch smaller

> repositories from the same server.

Since Jenkins is able to fetch smaller repositories, I infer that the

git plugin is working and something is wrong within the git program on

the Jenkins side, or with the git program on the server side.

> This repo is big (15 GB) and that's where it's choking, how big are

> your repositories?

A 15 GB repository is a huge git repository.  We made the tragic

mistake of placing large binaries in our git repository and it has now

reached an unwieldy 6+ GB.  We can no longer garbage collect our

repository on a 32 bit machine because of our mistake.

[Micks80]

A new ticket was opened to resolve this issue here -

http://issues.jenkins-ci.org/browse/JENKINS-9168

Thanks

On Mar 18, 9:53 pm, Mark Waite <markwa...@yahoo.com> wrote:
> > From: Micks80 <mick...@gmail.com>

> > Thanks again !!! ________________________________- Hide quoted text -