[JIRA] (JENKINS-55071) Slow branch scan when using Bitbucket source

[fragpit@gmail.com (JIRA)]

Igor Tretyak created an issue

Jenkins / JENKINS-55071 Slow branch scan when using Bitbucket source Issue Type: Bug Assignee: Unassigned Components: bitbucket-branch-source-plugin Created: 2018-12-07 08:31 Environment: jenkins 2.153 bitbucket branch source plugin 2.2.15 Priority: Major Reporter: Igor Tretyak Branch scan is very slow when using Bitbucket source in multibranch pipeline jobs. For example: repo with ~20 branches,  when using Git source it took 6 seconds, when using Bitbucket source it took up to 10 minutes. Looks like it is Bitbucket API related issue.  According to Jenkins server log Issue begun  at 05 dec 2018. 03:00 AM (GMT+3) I tested it on different geographically separated environments.  Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[mimmerkaer@jabra.com (JIRA)]

Mathias Immerkær commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source I've seen similar issue due to API limit throttling. Check your Jenkins log. https://issues.jenkins-ci.org/browse/JENKINS-54983

Add Comment

[fragpit@gmail.com (JIRA)]

Igor Tretyak commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source

I've checked, but there is nothing else. Just a scan report after some time.

Add Comment

[david.kostal@gmail.com (JIRA)]

David Kostal commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source

I am experiencing the same issue for cca 2-3 months already. The branch scan is fast if jenkins is down for longer time and then started, but gets slow over time. We use https access to Bitbucket with same username for all projects (and all projects' scans get slow at the same time). There is not notification in the jenkins log about API limits, just the final report like 'branch indexing action completed: SUCCESS in 13 min'

Add Comment

[slobodan@miskovic.ca (JIRA)]

Slobodan Miskovic commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source

Here's a "Me Too". It starts off quick, but then we get to stuff like this: branch indexing action completed: SUCCESS in 19 min Any suggestion on where to look for the bottleneck would be greatly appreciated.

Add Comment

[david.kostal@gmail.com (JIRA)]

David Kostal edited a comment on JENKINS-55071

Re: Slow branch scan when using Bitbucket source

I am experiencing the same issue for cca 2-3 months already. The branch scan is fast if jenkins is down for longer time and then started, but gets slow over time. We use https access to Bitbucket with same username for all projects (and all projects' scans get slow at the same time). There is not notification in the jenkins log about API limits, just the final report like 'branch indexing action completed: SUCCESS in 13 min' In the extreme, project with 130 branches and 20 PRs can get as slow as 1 hour for scanning. Then jenkins restart and it "only" takes 18 minutes. When it runs as expected it needs cca 2 mins.

Add Comment

[david.kostal@gmail.com (JIRA)]

David Kostal commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source

at in my case this was really related to Bitbucket API throttling. However there was no report in Jenkins log. After switch of the biggest and most used repo to dedicated BB user, it's fast again.

Add Comment

[lukasz.wojciechowski@ngsoft.pl (JIRA)]

Łukasz Wojciechowski commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source

Not sure I'll bring anything to the topic but at least I can share my "solution" for the same problem. In my case, we have projects with crazy amount of branches backed by mercurial which results in 3+ minutes delays. In practice, we need Jenkins build kick in, only for a very small subset of branches so at some point I've ended up building my own version of bitbucket-branch-source-plugin. Note, that I have "zero" experience in Java dev world. So, with below change I got rid of that nasty delay before every build and in the end it was all I needed. https://github.com/lukaswoj/bitbucket-branch-source-plugin/commit/549899cf8d8b0229c37a3a155b588f36b98f2c78 This is how I build it, launched from repo root dir docker run -it -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:latest mvn -DskipTests package You should end up with HPI file generated under "target" subdirectory. HPI file can be installed under Upload Plugin section on "/pluginManager/advanced" page of your Jenkins installation.   Hope this will be useful for someone.

Add Comment

[elmhaidara@gmail.com (JIRA)]

Mohamed HAIDARA updated an issue

Jenkins / JENKINS-55071

Slow branch scan when using Bitbucket source

Change By: Mohamed HAIDARA Attachment: log_config.PNG

Add Comment

[elmhaidara@gmail.com (JIRA)]

[elmhaidara@gmail.com (JIRA)]

Mohamed HAIDARA updated an issue

Jenkins / JENKINS-55071 Slow branch scan when using Bitbucket source

Change By: Mohamed HAIDARA Attachment: rate_limit.PNG

Add Comment

[elmhaidara@gmail.com (JIRA)]

Mohamed HAIDARA updated an issue

Jenkins / JENKINS-55071 Slow branch scan when using Bitbucket source

Change By: Mohamed HAIDARA Attachment: log_config.PNG

Add Comment

[elmhaidara@gmail.com (JIRA)]

Mohamed HAIDARA commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source I can confirm this issue with Jenkins 2.172 and Bitbucket Branche Plugin 2.4.4 However, to see the logs you need to add a new logger in the Jenkins interface. Go to "Configure Jenkins" > "System logs" and add a new logger. See my attached files.

Add Comment

[josephp90@gmail.com (JIRA)]

Joseph Petersen commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source

if you go into Jenkins global settings, you can enable cache for bitbucket cloud endpoint. You should increase your periodic scans to 8 hours or more and rely on events from Bitbucket cloud. Łukasz Wojciechowski you can achieve the same effect by adding branch filters in the behavior sections. No need to hardcode it.

Add Comment

[lukasz.wojciechowski@ngsoft.pl (JIRA)]

Łukasz Wojciechowski commented on JENKINS-55071

Re: Slow branch scan when using Bitbucket source

Joseph Petersen - can't agree with you. I still use what you referred, to limit Jenkins actions to only selected set of branches. But even then, Jenkins "blindly" asks Bitbucket API about all branches for given repository. Even worse, I believe it does only 10 rows per request (while it could ask for 100). Check my comment again, please. I am about places like this one: https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java#L454 There is no code that modifies API calls for list of branches. In my case it takes 3-4 minutes between clicking on Build now and first Pipeline line being executed. This is why I needed a patch. Thanks!

Add Comment