[JIRA] (JENKINS-36029) multibranch-job deleted when bitbucket communication fails

75 views
Skip to first unread message

asgeirf@gmail.com (JIRA)

unread,
Jun 16, 2016, 11:24:01 PM6/16/16
to jenkinsc...@googlegroups.com
Asgeir Frimannsson created an issue
 
Jenkins / Bug JENKINS-36029
multibranch-job deleted when bitbucket communication fails
Issue Type: Bug Bug
Assignee: Antonio Muñiz
Components: bitbucket-branch-source-plugin
Created: 2016/Jun/17 3:23 AM
Environment: Jenkins 2.8
Priority: Major Major
Reporter: Asgeir Frimannsson

I'm trying out the "Bitbucket team/project" feature: I have an issue where some of my jobs keeps being deleted and recreated randomly, and it seems to be related to situations where communicating with bitbucket fails. I would not expect Jenkins to delete my multibranch-job when e.g. Bitbucket or some network link is down.

Relevant log:

Proposing ansible-docker
Connecting to https://bitbucket.org using hidden-org-name/****** (hidden-org-name bitbucket credentials)
Looking up hidden-org-name/ansible-docker for branches
Checking branch jenkins-test from hidden-org-name/ansible-docker
Met criteria
ERROR: Failed to create or update a subproject ansible-docker
com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: Communication error: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRequest(BitbucketCloudApiClient.java:421)
	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRepository(BitbucketCloudApiClient.java:173)
	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.getRepositoryType(BitbucketSCMSource.java:254)
	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.observe(BitbucketSCMSource.java:372)
	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieveBranches(BitbucketSCMSource.java:326)
	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:279)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:146)
	at jenkins.scm.api.SCMSource.retrieve(SCMSource.java:230)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:175)
	at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria$1.call(MultiBranchProjectFactory.java:157)
	at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria$1.call(MultiBranchProjectFactory.java:154)
	at jenkins.branch.OrganizationFolder.withSCMSourceCriteria(OrganizationFolder.java:255)
	at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:154)
	at jenkins.branch.OrganizationFolder$1$1.complete(OrganizationFolder.java:165)
	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.add(BitbucketSCMNavigator.java:198)
	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.visitSources(BitbucketSCMNavigator.java:175)
	at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:125)
	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:157)
	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:122)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
	at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRequest(BitbucketCloudApiClient.java:412)
	... 20 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at sun.security.ssl.InputRecord.read(InputRecord.java:505)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	... 33 more

  Evaluating orphaned items in hidden-org-name on Bitbucket
  Will remove ansible-docker as it is #1 in the list
  Finished: SUCCESS
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

amuniz@cloudbees.com (JIRA)

unread,
Jun 20, 2016, 3:52:01 AM6/20/16
to jenkinsc...@googlegroups.com

xpunkerx@gmail.com (JIRA)

unread,
Jul 11, 2016, 4:53:12 AM7/11/16
to jenkinsc...@googlegroups.com

I have a lot of repositories in our team account (kinda 30) and I'm very annoyed that I have to re-run folder computation 10 times to get all my repos configured.
Will some retry logic work here?

michaeldkfowler@gmail.com (JIRA)

unread,
Jul 11, 2016, 3:51:02 PM7/11/16
to jenkinsc...@googlegroups.com
Michael Fowler updated an issue
 
Change By: Michael Fowler
Priority: Major Critical

javier.martincaro@beeva.com (JIRA)

unread,
Jul 13, 2016, 2:37:01 AM7/13/16
to jenkinsc...@googlegroups.com
Javier Martín commented on Bug JENKINS-36029
 
Re: multibranch-job deleted when bitbucket communication fails

We are experiencing this issue as well. As we delegate our User Directory to JIRA, when this instance is not available an Unauthorized Error 401 appears rising the following error:

{{
com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: No Autorizado.
{"errors":[

{"context":null,"message":"Authentication failed. Please check your credentials and try again.","exceptionName":"com.atlassian.bitbucket.auth.AuthenticationSystemException"}

]}
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:382)
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getBranches(BitbucketServerAPIClient.java:258)
at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:384)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:245)
at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:75)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:206)


at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)

Finished: FAILURE
}}

Then we need to run the folder computation again and all build numbers are restarted. We use this parameter as a complement to version number so there is a way to retain build numbers when this happens? This is not a case when a Jenkinsfile is not present in a branch (I assume when this happens the multibranch job is deleted), this is a temporary issue with the scan credentials and I think it wouln't be that way.

javier.martincaro@beeva.com (JIRA)

unread,
Jul 13, 2016, 2:38:01 AM7/13/16
to jenkinsc...@googlegroups.com
Javier Martín edited a comment on Bug JENKINS-36029

javier.martincaro@beeva.com (JIRA)

unread,
Jul 13, 2016, 2:38:03 AM7/13/16
to jenkinsc...@googlegroups.com

xpunkerx@gmail.com (JIRA)

unread,
Jul 13, 2016, 4:35:01 AM7/13/16
to jenkinsc...@googlegroups.com

Javier Martin this is not the same issue. The issue is about ssl handshake failure and yours is about auth failure. Make sure you've setup credentials correctly.

Meanwhile, I'm working on PR to fix the SSL handshake issue.

javier.martincaro@beeva.com (JIRA)

unread,
Jul 13, 2016, 5:42:01 AM7/13/16
to jenkinsc...@googlegroups.com

The title of the issue is "multibranch-job deleted when bitbucket communication fails" and I think this is the same issue. The log says unauthorized request, but this is due to a communication problem between bitbucket and jira and therefore bitbucket-branch-source-plugin behaves in the same way as the open issue. The credentials are placed correctly and works fine almost always.

Maybe I neet to create another issue because this is only because SSL handshake, but seems pretty close to me, the problem is that when bitbucket communication fails jenkins deletes all multibranch-jobs and restarts build numbers when the computation folder runs again.

amuniz@cloudbees.com (JIRA)

unread,
Jul 13, 2016, 6:37:02 AM7/13/16
to jenkinsc...@googlegroups.com

Right, the fix is the same for any "connection issue".

xpunkerx@gmail.com (JIRA)

unread,
Jul 13, 2016, 7:05:02 AM7/13/16
to jenkinsc...@googlegroups.com

Javier Martín: sorry then for misunderstanding.
Does simple retry fix the issue, how do you think?

amuniz@cloudbees.com (JIRA)

unread,
Jul 13, 2016, 7:22:03 AM7/13/16
to jenkinsc...@googlegroups.com

amuniz@cloudbees.com (JIRA)

unread,
Jul 13, 2016, 7:22:03 AM7/13/16
to jenkinsc...@googlegroups.com
Antonio Muñiz edited a comment on Bug JENKINS-36029


The fix needed here is similar.

xpunkerx@gmail.com (JIRA)

unread,
Jul 13, 2016, 7:53:02 AM7/13/16
to jenkinsc...@googlegroups.com

Antonio Muñiz: this fix is about github availability, there is only one check – "github == unavailable ? abort job : proceed job". Our issue is different. Our issue is about single requests failing sporadically that's why I'm talking about retries.

javier.martincaro@beeva.com (JIRA)

unread,
Jul 13, 2016, 7:57:01 AM7/13/16
to jenkinsc...@googlegroups.com

That's right, if you rerun folder computation after a failed connection it works again, but with the build numbers restarted. As Antonio Muñiz said, I think the problem is similar as the one resolved for github-branch-source. Regards!

alik@kurdyukov.com (JIRA)

unread,
Jul 15, 2016, 1:36:02 AM7/15/16
to jenkinsc...@googlegroups.com

amuniz@cloudbees.com (JIRA)

unread,
Jul 18, 2016, 1:32:03 PM7/18/16
to jenkinsc...@googlegroups.com

No, that PR is not fixing this.

alik@kurdyukov.com (JIRA)

unread,
Jul 18, 2016, 3:30:01 PM7/18/16
to jenkinsc...@googlegroups.com

Well, for me it works. I've built and installed plugin with this patch and haven't got any SSL errors in a week or so.

jenkins-ci.org@cowsgomoo.org (JIRA)

unread,
Aug 12, 2016, 12:55:02 PM8/12/16
to jenkinsc...@googlegroups.com

Any ETA on fixing the delete issue? Deleting all the build history when there's a transient bitbucket error is impacting me as well.

sanjeevkumar7@gmail.com (JIRA)

unread,
Aug 23, 2016, 10:38:02 AM8/23/16
to jenkinsc...@googlegroups.com

Experiencing the same issue. Currently have indexing turned off in multibranch plugin so we constantly don't end up deleting jobs.

Waiting for the fix for this.

davidkarlsen@java.net (JIRA)

unread,
Sep 8, 2016, 11:36:06 AM9/8/16
to jenkinsc...@googlegroups.com

Any progress on this? We´re a paying support customer with CB and have an issue open there as well.

rene.pieter.kok@wxs.nl (JIRA)

unread,
Sep 20, 2016, 5:15:09 AM9/20/16
to jenkinsc...@googlegroups.com
Rene Kok commented on Bug JENKINS-36029

We are using BitBucket cloud. Our scan bitbucket over 100+repositories randomly fails because the API rate limit is reached (https://confluence.atlassian.com/bitbucket/rate-limits-668173227.html). With the current implementation of the plugin we can do nothing but wait one hour (rate limit is measured per hour) and then try again to recover the lost build jobs.

I.m.o. the request rate can be reduced a lot (preventing hitting the API rate limit) if the scan logic is changed to only add jobs or branches that do not exist yet. A separate scan could do nightly cleanups of removed branches/repositories.

rene.pieter.kok@wxs.nl (JIRA)

unread,
Sep 20, 2016, 5:15:11 AM9/20/16
to jenkinsc...@googlegroups.com
Rene Kok edited a comment on Bug JENKINS-36029

amuniz@cloudbees.com (JIRA)

unread,
Oct 3, 2016, 2:08:01 PM10/3/16
to jenkinsc...@googlegroups.com

A workaround for this issue is to set the "Orphaned Item Strategy -> Days to keep old items" to something different than 0, for example 1 day.

benfortuna@gmail.com (JIRA)

unread,
Oct 3, 2016, 6:53:01 PM10/3/16
to jenkinsc...@googlegroups.com

Antonio Muñiz Yes, this is a semi-workaround which I use - currently set to 7 day expiry. However for some projects the develop branch isn't built so frequently (i.e. greater than 7 days between builds) so the build history is lost (i.e. resets to build #1 which causes other issues such as versioning conflicts).

anderius@gmail.com (JIRA)

unread,
Nov 24, 2016, 4:41:01 AM11/24/16
to jenkinsc...@googlegroups.com

Same issue for my team: After Bitbucket Server was down, all jobs reset their build numbers. We used the build number as versioning, but due to this bug we have to use the timestamp instead, which is less human friendly.

It would be nice if the plugin could see the difference between a branch not being present, and the Bitbucket server not answering.

stephen.alan.connolly@gmail.com (JIRA)

unread,
Jan 11, 2017, 9:41:04 AM1/11/17
to jenkinsc...@googlegroups.com

OK this seems to be a case of the BitBucketBranchSource not actually propagating exceptions.

Basically the BitbucketCloudApiClient and BitbucketServerAPIClient methods that might invoke network operations do not declare throwing IOException or InterruptedException and instead opt for the "friendly" returning of "dummy" values or null

This defeats the intended behaviour of branch api (which was verified in JENKINS-40767 e.g. see this test https://github.com/jenkinsci/branch-api-plugin/blob/d55f2b4369e0fe9b3b0441654e632eaf8bb5920f/src/test/java/integration/EventsTest.java#L287-L321 as evidence that a propagated exception prevents the orphaned item strategy from kicking in)

stephen.alan.connolly@gmail.com (JIRA)

unread,
Feb 13, 2017, 12:48:05 PM2/13/17
to jenkinsc...@googlegroups.com

I have a fix for this under test. If anyone wants to assist in doing some early testing the code is on https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/35 (though at this point it doesn't build)

stephen.alan.connolly@gmail.com (JIRA)

unread,
Feb 13, 2017, 12:48:06 PM2/13/17
to jenkinsc...@googlegroups.com
Stephen Connolly started work on Bug JENKINS-36029
 
Change By: Stephen Connolly
Status: Open In Progress

benfortuna@gmail.com (JIRA)

unread,
Feb 13, 2017, 6:02:03 PM2/13/17
to jenkinsc...@googlegroups.com

Stephen Connolly Will this fix also apply for Git/Github repositories? I suspect most of these plugins will just swallow exceptions when they can't connect to a repository, so perhaps the design needs to be rethought.. (possibly require an exception/explicit return value before deleting the job?)

stephen.alan.connolly@gmail.com (JIRA)

unread,
Feb 13, 2017, 6:54:02 PM2/13/17
to jenkinsc...@googlegroups.com

Ben Fortuna to the best of my knowledge the Git, GitHub and Subversion implementations do the correct thing and propagate IO errors as exceptions. Certainly when I run out of rate limit the scan is aborted and no repositories or branches are deleted. If you have a reproducible test case or an example log where there was an API error and the scan / index completed as success (which would then cause the missing branches to be removed) please create an issue for it in JIRA as that kind of data loss is highest priority IMHO (i.e. this is why I am working on this issue ahead of all others at present - because it is a data loss issue)

benfortuna@gmail.com (JIRA)

unread,
Feb 13, 2017, 8:32:04 PM2/13/17
to jenkinsc...@googlegroups.com

Stephen Connolly new issue created for github: JENKINS-42000

stephen.alan.connolly@gmail.com (JIRA)

unread,
Feb 15, 2017, 2:27:05 PM2/15/17
to jenkinsc...@googlegroups.com
Change By: Stephen Connolly
Assignee: Antonio Muñiz Stephen Connolly

stephen.alan.connolly@gmail.com (JIRA)

unread,
Feb 15, 2017, 2:27:11 PM2/15/17
to jenkinsc...@googlegroups.com
Stephen Connolly resolved as Fixed
 

I claim fixed in 2.1.0 release
Change By: Stephen Connolly
Status: In Progress Resolved
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages