Multibranch Pipeline: Checkout hangs on git rev-list

[Sverre Moe]

Recently we upgraded Jenkins from 2.100 to 2.103, and a bunch of plugins at the same time.

After this we are frequently experiencing checkout scm to freeze at the last step in this process (git rev-list). It will hang there forever, way past the default 10 minutes timeout.

Seems we are not the only one experiencing this problem, but while others have had this problem for quite some time we have only gotten it recently.

https://issues.jenkins-ci.org/browse/JENKINS-46503

Anyone familiar with this problem?

I was thinking of reverting back to Jenkins 2.100 and the following plugins

git 3.6.4

git-client 2.6.0

I have downloaded the JAR files for these plugins.

[Mark Waite]

The JENKINS-45447 and JENKINS-47169 fix in git client plugin 2.7.0 and in git plugin 3.7.0 might be visible as a hang in `git rev-list`.  Does that repository contain a large number of tags?  If so, then be sure you're using the latest releases of git client plugin and git plugin.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0a6cdca7-1a08-42e9-8e39-7fb2a10e287a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sverre Moe]

The problem occurred with the latest version of git plugin and git client plugin.

We have gotten the problem on projects that does not have many tags, aprox 300. We even got the problem on a project that has no tags at all.

We are back at running with the latest version after I tried to revert back to the previous git-client-3.7.0 (was 3.7.1 and could not revert further back). On the server I have upgraded git to latest v2.16 (wanted to see if using a newer version of git would solve the problem).

[Sverre Moe]

Just now we got one Multibranch pipeline to hang on git rev-list on a project that has no git tags.
Running with latest Jenkins, git plugin, git-client plugin and git. 

[derrick....@imsar.com]

We have seeing the same problem.  For some reason it is calling 'git rev-list' on a commit that is 6 months old.  There doesn't seems to be anything particularly special about that commit (No release tag).

I did notice that if I skip the 'merge before build' it has a higher chance of working, but it makes the job much less useful.

I don't think the git command is actually what is hanging, because when I run 'ps' I don't see any processes running on the slave or the master.  It seems that the command finished, but the waiting process didn't get notified somehow?

Any ideas how to debug this?  I would like to see what is going on in the git plugin.  Why is it picking that ancient commit?  What does it want to do next?  I tried adding a debug logger for 'hudson.plugins.git' and Log Level 'All' but I don't see anything useful.

[derrick....@imsar.com]

This is the thread dump while it is hung:

Thread #10
    at DSL.checkout(running in thread: org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#3])
    at Script1.add_repo(Script1.groovy:309)
    at Script1.run_compile_tests(Script1.groovy:49)
    at DSL.stage(Native Method)
    at Script1.run_compile_tests(Script1.groovy:48)
    at DSL.node(running on merlin)
    at Script1.run_compile_tests(Script1.groovy:46)
    at Script1.grizzly_merge_request(Script1.groovy:336)
    at DSL.gitlabCommitStatus(Native Method)
    at Script1.grizzly_merge_request(Script1.groovy:335)
    at Script1.grizzly_generator(Script1.groovy:331)
    at WorkflowScript.run(WorkflowScript:37)

I have reverted to git plugin 3.6.3 and git client plugin 2.6.0, which is as far back as the GUI will let me go.

[Sverre Moe]

It seems the problem is the JIRA Plugin.

Downgrading JIRA Plugin from 2.5.1 to 2.5.0 solves the problem.

As discussed in https://issues.jenkins-ci.org/browse/JENKINS-46503