[JIRA] (JENKINS-51421) limit the number of concurrent request to a workflow library repository

[fernando.rosado@gmail.com (JIRA)]

Fernando Rosado Altamirano created an issue

Jenkins / JENKINS-51421 limit the number of concurrent request to a workflow library repository Issue Type: Improvement Assignee: Unassigned Components: pipeline Created: 2018-05-18 11:45 Priority: Major Reporter: Fernando Rosado Altamirano We have a workflow-lib repository enabled that contains some groovy script utilities commons for all JenkinsFile. We have named it as "workflow-libs" an located on a git repository : ssh://bitbucket.server/prj/workflow-lib.git and we use it as usual : Unable to find source-code formatter for language: groovy. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml @Library ('jworkflow-libs') _  node \{      stage ("Test") \{      builder = new workflow.BuildHelper() // ....      echo "Hello world"    } } But we are suffering a problem with the communication between Jenkins and Bitbucket server, when you launch >5 jobs simultaneously, the 6^th^ job fails with an error message : hudson.plugins.git.GitException: Failed to fetch from ssh://bitbucket.server/prj/workflow-lib.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:112) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:84) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:153) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:134) at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:421) Caused by: hudson.plugins.git.GitException: Command "git fetch --no-tags --progress ssh://bitbucket.server/prj/workflow-lib.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: ssh_exchange_identification: read: Connection reset by peer fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. If you relaunch the job individually it works. We are trying to solve it ( it could be a security restriction between bitbucket - jenkins to avoid DoS ) but would be very useful a configuration parameter for worfklow libs that limit the number of simultaneous request to the same repository (like pooling configuration has). Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[fernando.rosado@gmail.com (JIRA)]

Fernando Rosado Altamirano updated an issue

Jenkins / JENKINS-51421 limit the number of concurrent request to a workflow library repository

Change By: Fernando Rosado Altamirano

We have a workflow-lib repository enabled that contains some groovy script utilities commons for all JenkinsFile. We have named it as "workflow-libs" an located on a git repository : ssh://bitbucket.server/prj/workflow-lib.git and we use it as usual :

{code: groovy java }

@Library ('jworkflow-libs') _  node {      stage ("Test") {      builder = new workflow.BuildHelper() //  ....      echo "Hello world"    } }

{code}

But we are suffering a problem with the communication between Jenkins and Bitbucket server, when you launch >5 jobs simultaneously, the 6^th^ job fails with an error message :

{noformat}

{noformat}

If you relaunch the job individually it works. We are trying to solve it ( it could be a security restriction between bitbucket - jenkins to avoid DoS ) but would be very useful a configuration parameter for worfklow libs that limit the number of simultaneous request to the same repository (like pooling configuration has).

Add Comment

[fernando.rosado@gmail.com (JIRA)]

Fernando Rosado Altamirano updated an issue

Jenkins / JENKINS-51421 limit the number of concurrent request to a workflow library repository Change By: Fernando Rosado Altamirano

We have a workflow-lib repository enabled that contains some groovy script utilities commons for all JenkinsFile. We have named it as "workflow-libs" an located on a git repository : ssh://bitbucket.server/prj/workflow-lib.git and we use it as usual : {code:groovy}

\ {      stage ("Test") \ {

[fernando.rosado@gmail.com (JIRA)]

Fernando Rosado Altamirano updated an issue

Jenkins / JENKINS-51421 limit the number of concurrent request to a workflow library repository Change By: Fernando Rosado Altamirano

We have a workflow-lib repository enabled that contains some groovy script utilities commons for all JenkinsFile. We have named it as "workflow-libs" an located on a git repository : ssh://bitbucket.server/prj/workflow-lib.git and we use it as usual :

{code:java}

@Library ('jworkflow-libs') _  node {      stage ("Test") {      builder = new workflow.BuildHelper() //  ....      echo "Hello world"    } } {code}

But we are suffering a problem with the communication between Jenkins and Bitbucket server, when you launch >5 jobs simultaneously, the 6^th^ job fails with an error message . This can be emulated using : { code:java} stage("Launch all") { parallel(     test1 : {build job: 'test1', wait: true},     test2 : {build job: 'test2', wait: true},     test3 : {build job: 'test3', wait: true},     test4 : {build job: 'test4', wait: true},     test5 : {build job: 'test5', wait: true},     test6 : {build job: 'test6', wait: true},     test7 : {build job: 'test7', wait: true},     test8 : {build job: 'test8', wait: true},     test9 : {build job: 'test9', wait: true},     ) {code} Where job 'testX' contains the previous one commented. { noformat}

[fernando.rosado@gmail.com (JIRA)]

{noformat}

[andrew.bayer@gmail.com (JIRA)]

Andrew Bayer updated an issue

Jenkins / JENKINS-51421 limit the number of concurrent request to a workflow library repository

Change By: Andrew Bayer Component/s: workflow-cps-global-lib-plugin Component/s: pipeline

Add Comment

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

[fernando.rosado@gmail.com (JIRA)]

Fernando Rosado Altamirano updated an issue

Jenkins / JENKINS-51421 limit the number of concurrent request to a workflow library repository

Change By: Fernando Rosado Altamirano Component/s: git-plugin

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[fernando.rosado@gmail.com (JIRA)]

Fernando Rosado Altamirano commented on JENKINS-51421

Re: limit the number of concurrent request to a workflow library repository I have detected similar issue if you are using the same repository on different jobs. If the git server limit the number of git requests it will fails with this error. It wil be great if we can limit the number of simultaneous requests to the same repository globally. And probably this is a configuration that should be available like the global configuration "Max # of concurrent polling"

Add Comment