Multibranch Pipeline Scan Hangs

[Randall Becker]

Hi All,

I am trying to move over to Pipelines from Freestyle and am experiencing a hang during the initial scan for branches at ls-remote in the Multibranch Pipeline Scan. A few symptoms:

1. When using a freestyle project, with the same credentials and URL, there is no issue.
2. When using ls-remote from the command line with the same credentials and URL, there is no issue.
3. Git is available to Jenkins, but I tried providing a full path - no change.
4. The project is minimal with no actual actions defined yet.
5. I am on Jenkins 2.138.2 and the latest plugins.
   

On interrupt, here is the stack trace:

[Thu Nov 15 15:35:46 GMT 2018] Starting branch indexing...
 > /usr/bin/git --version # timeout=10
using GIT_SSH to set credentials Randall's Credentials
 > /usr/bin/git ls-remote --symref g...@bitbucket.org:XXXXXXXXXXXXX-my-private-url # timeout=10
ERROR: [Thu Nov 15 15:40:45 GMT 2018] Could not update folder level actions from source b9692b8b-7465-43bf-b59c-0103ba223697
java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)
	at hudson.Proc$LocalProc.join(Proc.java:324)
	at hudson.Proc.joinWithTimeout(Proc.java:170)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2012)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1640)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1631)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteSymbolicReferences(CliGitAPIImpl.java:2893)
	at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1093)
	at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:765)
	at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:590)
	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
	at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1024)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
[Thu Nov 15 15:40:45 GMT 2018] Finished branch indexing. Indexing took 4 min 59 sec
Aborted
Finished: ABORTED

When running from the CLI:

# GIT_SSH_COMMAND="ssh -i .ssh/id_rsa" /usr/bin/git ls-remote --symref g...@bitbucket.org:XXXXXXXXXXXXX-my-private-url
Enter passphrase for key '.ssh/id_rsa':
ref: refs/heads/master  HEAD
0c3b1737c3f43cbc1fa149d220d244d7c89c35b5        HEAD
3793b236a749f4a17f3e063b4a7581bcfb03b2a2        refs/heads/Pipeline_Implementation
0c3b1737c3f43cbc1fa149d220d244d7c89c35b5        refs/heads/development
0c3b1737c3f43cbc1fa149d220d244d7c89c35b5        refs/heads/master
0c3b1737c3f43cbc1fa149d220d244d7c89c35b5        refs/heads/production
96329f8a65c31728d1477a1ed36f3988b77cea08        refs/tags/v1.6.510
7abe49dce26caddf0fa7e052a19983802f27fa77        refs/tags/v1.6.510^{}
 
No idea how to resolve this, since outside the plugin, connectivity is otherwise fine.

TIA,
Randall

[Ivan Fernandez Calvo]

Are you using a shared network filesystem for your JENKINS_HOME? it seems like takes too long to save data, I guess your filesystem is too slow, check if your filesystem speed is lower than100MB/s, if so, it is too slow and you have performance issues on Jenkins.

```

java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)

```

[Randall Becker]

The interrupted exception resulted from my stopping the job instead of waiting 10 minutes for the timeout failure. I am not on a shared filesystem. Rather, this is inside a docker image running entirely on SSDs. Speed is way above 100Mbps. As I said, no issue when using a freestyle project and Jenkins is so fast on this machine (Antsle 8 core Avoton) that I don't even get a chance to get coffee anymore. The ls-remote using the git CLI outside of docker takes under 1sec, so that's not it.

[Ivan Fernandez Calvo]

also, check the value of the TIMEOUT property, execute this code in the Jenkins script console, the default value is 10 minutes, you can increase it by setting the property in your command line

System.getProperty(org.jenkinsci.plugins.gitclient.Git.class.getName() + ".timeOut")

If you have tons of tags on your repos try to not discover tags.

El jueves, 15 de noviembre de 2018, 16:51:53 (UTC+1), Randall Becker escribió:

[Randall Becker]

The original post showed the entire list of refs. No issue requiring a timeout above 3 seconds actually. 10 minutes will make no difference. When run from Jenkins from a shell command, the result comes back in seconds. I don't think this is a performance issue. It is seeming to be a problem in the plugin.

[Mark Waite]

I assume from your description that you're using a private key with a passphrase as the credential in the Pipeline job definition.  I'm also assuming it is a simple Pipeline job rather than a multibranch Pipeline.

Does the same behavior happen if you use a private key which does not have a passphrase?

Does the same behavior happen if you use a multibranch Pipeline job rather than a simple Pipeline job?

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/28c82b76-6b89-4347-8190-71f8dceb349b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Randall Becker]

Hi Mark,

Yes, using a private key. This is a bitbucket.org site, which by my own company's policy prevents me from using a passphrase-less credential. The same key-pair is used elsewhere in other jobs in the same Jenkins without problems. The job is a multibranch job, not a simple pipeline. A simple pipeline (as tiny as I can get, as follows), works with this credential set:

node('maven') {
    git branch: 'development', credentialsId: 'idalias', url: 'g...@bitbucket.org:XXXXMYURL.git'
}

So this is looking increasingly like an issue with Pipeline: Multibranch.

Cheers,

Randall

[Randall Becker]

There is also similar hang at fetch when the Jenkinsfile SCM is used. It does appear that the credentials may not be supplied properly but there is no indication that this is the case. I'm dropping back to Simple Pipelines without SCM, sadly. This rather makes the whole point rather moot. Where can I raise a defect?

[Mark Waite]

I created a multibranch pipeline with bitbucket.org as the source using the Bitbucket Branch Source plugin.  That plugin prompted for a username / password rather than a private key.  It won't accept a private key.  I believe it uses REST APII calls to reduce the overhead of checking for new branches and for changes in the existing branches.  Since the REST API is based on https, it needs username / password rather than a private key.

The Bitbucket Branch Source plugin provides additional capabilities because it is aware of Bitbucket, rather than being limited to those components which are generic git operations.

Is there a reason you're not using the Bitbucket Branch Source plugin, rather than using the Git branch source?

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8176d940-3f70-40dc-bc02-015f1813d440%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Mark Waite]

I created a multibranch Pipeline with bitbucket.org as the source using Git as the branch source with a passphrase protected ed25519 private key.  It scans and builds as expected.

I can't duplicate the problem that you're describing.  Before you spend the time to submit a bug report, you may want to duplicate the bug in a different environment, just in case there is something specific in your environment that is different than my attempt to duplicate the problem.

Bug reports for the project are submitted to https://issues.jenkins-ci.org/ .

--

Thanks!

Mark Waite

[Randall Becker]

I will continue to investigate. I am wondering whether there might be something associated with running Jenkins in docker with Controller/Agent structure.

[Meg Watson]

I’ll take a stab...How many executors do you have on the master and agent? You may be hanging waiting for an executor to be available to run the pipeline. I think I made this mistake once, had too few executors.

Meg

[Mark Waite]

My passing tests were run from a master running in Docker with agents running a mix of various operating systems including Windows, Linux (CentOS, Debian, Ubuntu), and FreeBSD.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/148ad79f-da30-4b46-ac8f-316269d1e01e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Randall Becker]

I have 2 idle executors on every node, so that's not it. The hang is definitely waiting on a response from git fetch or git ls-remote depending on the situation. Version of git on the Ubuntu VM is 2.13, so that's not particularly recent, but sufficient for the function being executed. I cannot get this to work on any instance of any VM I have. Of note, there are no specific instructions other than the basic setup of the job (monitor the repo, run ls-remote, and finish). Perhaps with some instructions it might behave differently. When I interrupt the job, the job name is "null", which is suspect and wrong.

I've gone to non-multibranch pipelines, as assume that this particular plugin is DOA - sorry about the opinion, but I cannot make it work, despite all efforts, so will not recommend it. If there is a working configuration, please describe it, and I'll try to make it work as specified. $DAYJOB forces me away from this approach.

[Mark Waite]

My working bitbucket.org multibranch Pipeline configurations include:

• Bitbucket branch source (username / password authentication)
• Git branch source using https (username / password authentication)
• Git branch source using ssh (ed25519 private key with passphrase)

Will your corporate standards allow use of a username and password over https?  If so, does that behave any better in your environment?

Mark Waite

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a2538d76-8460-40bf-861c-57bff009df27%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Randall Becker]

The answer to that is no. SSH only. I rebuilt the docker image badly (last time I listen to the Internet on this one) - so I'm starting from scratch.

[Randall Becker]

The good news is that after the rebuild and downloading a new docker image, things seem to be working fine now. I guess it's possible the image was not properly downloaded but there was no indication of problems at the time. Thanks for the shoulder to cry on.