Windows git plugin / git client plugin problems

[Dunnigan, Terrence J]

Hi all,

 

I’ve run into an issue where the latest git plugin / git client plugins time out. It pulls down the files in .git\objects\pack and elsewhere, but nothing beyond the .git folder.

 

Here’s my setup:

 

Jenkins server:

Jenkins LTS 1.532.1 running on a Windows 7 PC

Jenkins GIT client plugin 1.6.2 (also on 1.6.1)

Jenkins GIT plugin 2.0.1

 

Windows 7 slave PC:

Jenkins is running as a service using an administrative account

Git 1.8.3 is installed, along with PuTTY.

GIT_SSH is set to C:\PuTTY\plink.exe

Path includes C:\PuTTY

 

Here’s the error message once I hit 10 minutes:

 

Started by user Terry Dunnigan

Building remotely on slave in workspace C:\hudson_remote\workspace\job name

Cloning the remote Git repository

Cloning repository g...@myrepo.git

Fetching upstream changes from g...@myrepo.git

using GIT_SSH to set credentials 

Fetching upstream changes from g...@myrepo.git

using GIT_SSH to set credentials 

ERROR: Timeout after 10 minutes

FATAL: Failed to fetch from g...@myrepo.git

hudson.plugins.git.GitException: Failed to fetch from g...@myrepo.git

        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:625)

        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:847)

        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:872)

        at hudson.model.AbstractProject.checkout(AbstractProject.java:1411)

        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652)

        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)

        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:557)

        at hudson.model.Run.execute(Run.java:1665)

        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)

        at hudson.model.ResourceController.execute(ResourceController.java:88)

        at hudson.model.Executor.run(Executor.java:246)

Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress g...@myrepo.git +refs/heads/*:refs/remotes/origin/*" returned status code -1:

 

stderr: trace: built-in: git 'fetch' '--tags' '--progress' 'g...@myrepo.git' '+refs/heads/*:refs/remotes/origin/*'

trace: run_command: 'C:\Users\me\AppData\Local\Temp\ssh6588671275929067852.bat' 'g...@myrepo.git' 'git-upload-pack '\''myrepo.git'\'''

 

trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'

trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all'

trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all'

 

 

Any thoughts?

 

Thanks everyone.

 

Terry Dunnigan

[Slide]

What is the path to Git? I use the one in ProgramFiles\Git\cmd instead of ProgramFiles\Git\bin.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Website: http://earl-of-code.com

[Curtis Kline]

I ran into the same problem today for the first time. I am using Git plugin 1.5.0 (still scared to jump to 2.0) and Git Client Plugin 1.6.1. I don't use plink, but rather the ssh included with msysgit 1.8.5. My Windows slaves are connected using JNLP.

I desperately need some way to control that timeout value, as running the clone command locally on the box indicates that the first clone will take at least 15 minutes. I've reviewed several issues on the Jenkins issue tracker, and haven't seen any solutions to controlling the timeout on JNLP-connected slaves.

Any suggestions would be greatly appreciated!

[Mark Waite]

My technique is the opposite of Slide's.  I always use the git executable in the bin directory.  There were issues reported when the git executable in the cmd directory was used instead of using the executable in the bin directory.

You might consider a "test drive" of the JGit implementation in your environment to see if it behaves any differently.  The JGit implementation does not have all the features of the command line git implementation, but it may be sufficient for your use case.

Was this job upgraded from a previous Git plugin version?  If so, have you defined the credentials and configured the job to use those credentials?

Is there some environment variable that is causing those unexpected "trace:" and "run_command:" strings in the output?  I don't see those in my use of command line git on Windows.

Thanks!

Mark Waite

[Mark Waite]

Curtis, I think your problem is a different problem than the original description from Terry.

I believe Terry is attempting to clone a repository which should clone in significantly less than 10 minutes, while you already know that your clone will take more than 10 minutes (due to the size of the repository and the limitation in msys git that it cannot transfer more than about 1 MB/second using ssh).

The next release of the git plugin (after 2.0.1) adds a timeout field to the job configuration so that users with large repositories (like you and me) can set a larger timeout value for those jobs which need it.  If you need it sooner, or are willing to assist with the testing, you could compile the latest git plugin and install it on a separate Jenkins instance for testing.

Refer to https://issues.jenkins-ci.org/browse/JENKINS-20445.

Refer to https://github.com/jenkinsci/git-plugin/pull/197 for the changes to git-plugin.

Refer to https://github.com/jenkinsci/git-client-plugin/pull/92 for the changes to git-client-plugin, already available in 1.6.2

You should probably avoid the JGit implementation when using large repositories.  The recent JGit versions no longer hang when cloning large repositories, but they do significantly increase the size of the Java virtual machine while they are cloning large repositories.

Thanks,

Mark Waite

--

Thanks!

Mark Waite

[Curtis Kline]

Thanks Mark. I think you're right.

My current problem involves projects that really can't wait for my testing of the next release of the git plugin (although I am definitely interested in doing that). Any short-term suggestions on how to get this to work with my current setup? I thought about switching to Plink instead of the ssh that comes with msysgit in hopes that I could clone my huge repo inside the 10 minute limit, but I've tried various people's tutorials with no luck. For example, the Msysgit install routine is supposed to give you an option for this if you've used Plink on that system before, but that option never appears.

Thanks again.

Curtis

[Mark Waite]

If you have control of the git repository, and if you don't mind allowing the source code to be readable by anyone inside your network, you could enable a git protocol server in addition to your ssh protocol server.  The download speed through git protocol is very near wire speed, so should be faster than the ssh in msysgit.

If you start the Windows slave agent from a command line (rather than from JNLP or from a service), you should be able to pass the property on the slave agent startup command line -Dorg.jenkinsci.plugins.gitclient.timeOut=30.

If you're starting the Windows slave agent from a service, there is probably a registry key somewhere which controls the startup of the Java process that is the slave agent.  If you can find that key, you may be able to modify it to include that command line argument to set the timeOut value.

Mark Waite

[Mark Waite]

Sorry, I gave bad information.  The timeOut property I listed is incorrect.

It should be -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=30

--

Thanks!

Mark Waite

[Curtis Kline]

Thanks Mark. I'll try running my Windows slave from the command line. So the timeout value is in minutes?

Curtis

[Mark Waite]

That's correct, the units are minutes.

[Dunnigan, Terrence J]

Hi everyone,

 

Thanks for the feedback & suggestions.

 

I have resolved the issue by setting the “Credentials” to “None” in my Jenkins Git plugin configuration. I have credentials on the slave PC along with PuTTY and GIT_SSH set to Putty’s plink.exe, and everything works as it should. Why would using my saved credentials cause the Jenkins GIT plugin to ignore the slave PC’s GIT_SSH’s setting? (As an aside, what am I giving up by not using those stored credentials?)

 

To clarify:

 

-          My exact configuration works just fine using the Jenkins GIT plugin for small repos (2 MB or so). But I am encountering my problem when cloning a repo of ~ 20 MB and larger using the Git plugin and credentials. My workgroup has seen issues with cloning failing on larger repos unless we are using PuTTY’s plink as our SSH client. When not using credentials, I have no problems.

-          I think there’s a problem with the Jenkins GIT plugins not invoking my GIT_SSH’s plink.exe when using Credentials. When the job is hung, and I view the current processes running on my slave machine, I see git’s ssh.exe, and not plink.exe, running. The Jenkins console output even says “using GIT_SSH to set credentials”.

-          I get the same error when I set the Jenkins GIT plugin timeout values to something obnoxiously large, like 6 hours. My clone operation is consistently stalling out in the same spot.

-          This is a clean install of the latest LTS release on a test PC. No prior Jenkins was running on it.

-          I have set GIT_TRACE = 2, which is showing those trace and run_command messages.

-          We have some repos that approach 1 GB. Based on other comments in this thread it sounds like JGit may not be the best choice for me at the moment.

-          I get the same error with either C:\Program Files (x86)\git\cmd\git.exe and C:\Program Files (x86)\git\bin\git.exe defined as the git installation.

 

I see a few GIT_SSH related errors on https://issues.jenkins-ci.org/browse/JENKINS/component/17423 .I run Jenkins as a service and I define my repos in the Jenkins GIT configuration page as git@server:repo.git. For what it’s worth, though, I also saw this error when running Jenkins via JNLP.

 

If there is anything that I can do to help troubleshoot the issue further, please let me know.

 

Thanks,

 

Terry