[JIRA] (JENKINS-61493) Lightweight checkout fails in Bitbucket Server 7.0; refs/pull-requests/*/merge does not exist

0 views
Skip to first unread message

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 16, 2020, 12:05:02 PM3/16/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo updated an issue
 
Jenkins / Improvement JENKINS-61493
Lightweight checkout fails in Bitbucket Server 7.0; refs/pull-requests/*/merge does not exist
Change By: Kalle Niemitalo
Summary: Lightweight checkout fails in Bitbucket Server 7.0; refs/pull-requests/*/ from merge does not exist
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 16, 2020, 12:10:08 PM3/16/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo updated an issue
After upgrading to Bitbucket Server 7.0.1, we get this kind of error for every pull request in multibranch projects in which Jenkins has been configured to build the result of merging the pull request:

{noformat}
ERROR: Could not do lightweight checkout, falling back to heavyweight
java.io.FileNotFoundException: URL: /rest/api/1.0/projects/ SISU REDACTED /repos/ rahti-hybrid redacted /browse/Jenkinsfile?at=pull-requests%2F771%2Fmerge&start=0&limit=500
  at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:831)
  at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getFileContent(BitbucketServerAPIClient.java:1123)
  at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:98)
  at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:335)
  at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:107)
  at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)
  at hudson.model.ResourceController.execute(ResourceController.java:97)
  at hudson.model.Executor.run(Executor.java:427)
{noformat}

{{git ls-remote origin}} shows that Bitbucket Server now publishes "refs/pull-requests/771/from" but not "refs/pull-requests/771/merge", even though the pull request has no merge conflicts (the target branch even has no new commits). This might be related to ["the switch from a 3-way diff to a 2-way diff in pull requests"|https://confluence.atlassian.com/bitbucketserver/bitbucket-server-7-0-release-notes-990546638.html#BitbucketServer7.0releasenotes-Amodifiedbehaviorfordiffs] in Bitbucket Server 7.0.0.

Jenkins then does the heavyweight checkout and successfully merges the pull request, so the project ultimately builds OK, but the heavyweight checkout consumes a lot of time and disk space.

I don't know whether there is any simple way to fix this in the bitbucket-branch-source plugin. If Bitbucket Server has deliberately ceased publishing "refs/pull-requests/*/merge", perhaps the plugin could do a three-way merge on Jenkinsfile only, rather than on the entire tree. That would require a REST API for getting the commit ID of a merge base though, and I don't know whether such a thing exists.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 16, 2020, 12:26:02 PM3/16/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo updated an issue
After upgrading to Bitbucket Server 7.0.1, we get this kind of error for every pull request in multibranch projects in which Jenkins has been configured to build the result of merging the pull request:

{noformat}
ERROR: Could not do lightweight checkout, falling back to heavyweight
java.io.FileNotFoundException: URL: /rest/api/1.0/projects/REDACTED/repos/redacted/browse/Jenkinsfile?at=pull-requests%2F771%2Fmerge&start=0&limit=500

  at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:831)
  at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getFileContent(BitbucketServerAPIClient.java:1123)
  at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:98)
  at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:335)
  at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:107)
  at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)
  at hudson.model.ResourceController.execute(ResourceController.java:97)
  at hudson.model.Executor.run(Executor.java:427)
{noformat}

{{git ls-remote origin}} shows that Bitbucket Server now publishes "refs/pull-requests/771/from" but not "refs/pull-requests/771/merge", even though the pull request has no merge conflicts (the target branch even has no new commits). This might be related to ["the switch from a 3-way diff to a 2-way diff in pull requests"|https://confluence.atlassian.com/bitbucketserver/bitbucket-server-7-0-release-notes-990546638.html#BitbucketServer7.0releasenotes-Amodifiedbehaviorfordiffs] in Bitbucket Server 7.0.0.

Jenkins then does the heavyweight checkout and successfully merges the pull request, so the project ultimately builds OK, but the heavyweight checkout consumes a lot of time and disk space.

I don't know whether there is There might not be any simple way to fix improve this in the bitbucket-branch-source plugin. If Bitbucket Server has deliberately ceased publishing "refs/pull-requests/*/merge", perhaps the plugin could do a three-way merge on Jenkinsfile only, rather than on the entire tree. That would require

- One way might be to use the REST API to get Jenkinsfile from the target branch ([/rest/api/1.0/projects/\{projectKey\}/repos/\{repositorySlug\}/browse/\{path:.*\}|https://docs.atlassian.com/bitbucket-server/rest/7.0.1/bitbucket-rest.html#idp208]) and any Jenkinsfile changes of the pull request as a patch ([/rest/api/1.0/projects/\{projectKey\}/repos/\{repositorySlug\}/pull-requests/\{pullRequestId\}.patch|https://docs.atlassian.com/bitbucket-server/rest/7.0.1/bitbucket-rest.html#idp289] gives the standard diff format but does not support limiting to a single file; [/rest/api/1.0/projects/\{projectKey}/repos/\{repositorySlug}/pull-requests/\{pullRequestId}/diff/\{path:.*}|https://docs.atlassian.com/bitbucket-server/rest/7.0.1/bitbucket-rest.html#idp229] supports that but the output is JSON) and then apply the patch to Jenkinsfile at the Jenkins master.
- Alternatively, if Bitbucket Server has a REST API for getting querying the commit ID of a merge base though of a pull request , then the plugin could first use that, then get Jenkinsfile from the three commits, and I don't know whether finally do the three-way merge. However, there does not seem to be such a thing exists an API .

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 16, 2020, 12:31:02 PM3/16/20
to jenkinsc...@googlegroups.com

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 16, 2020, 12:33:02 PM3/16/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo updated an issue
Change By: Kalle Niemitalo
Environment: Jenkins 2.204.5
Bitbucket Branch Source Plugin 2.7.0
Bitbucket Server 7.0.1

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 24, 2020, 7:43:03 AM3/24/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo updated an issue
After upgrading to Bitbucket Server 7.0.1, we get this kind of error for every pull request in multibranch projects in which Jenkins has been configured to build the result of merging the pull request:

{noformat}
ERROR: Could not do lightweight checkout, falling back to heavyweight
java.io.FileNotFoundException: URL: /rest/api/1.0/projects/REDACTED/repos/redacted/browse/Jenkinsfile?at=pull-requests%2F771%2Fmerge&start=0&limit=500
  at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:831)
  at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getFileContent(BitbucketServerAPIClient.java:1123)
  at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:98)
  at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:335)
  at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:107)
  at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)
  at hudson.model.ResourceController.execute(ResourceController.java:97)
  at hudson.model.Executor.run(Executor.java:427)
{noformat}

{{git ls-remote origin}} shows that Bitbucket Server now publishes "refs/pull-requests/771/from" but not "refs/pull-requests/771/merge", even though the pull request has no merge conflicts (the target branch even has no new commits). This might be related to ["the switch from a 3-way diff to a 2-way diff in pull requests"|https://confluence.atlassian.com/bitbucketserver/bitbucket-server-7-0-release-notes-990546638.html#BitbucketServer7.0releasenotes-Amodifiedbehaviorfordiffs] in Bitbucket Server 7.0.0.

Jenkins then does the heavyweight checkout and successfully merges the pull request, so the project ultimately builds OK, but the heavyweight checkout consumes a lot of time and disk space.

There might not be any simple way to improve this in the bitbucket-branch-source plugin. If Bitbucket Server has deliberately ceased publishing "refs/pull-requests/*/merge", perhaps the plugin could do a three-way merge on Jenkinsfile only, rather than on the entire tree.

- One way might be to use the REST API to get Jenkinsfile from the target branch ([/rest/api/1.0/projects/\{projectKey\}/repos/\{repositorySlug\}/browse/\{path:.*\}|https://docs.atlassian.com/bitbucket-server/rest/7.0.1/bitbucket-rest.html#idp208]) and any Jenkinsfile changes of the pull request as a patch ( , and then apply the patch to Jenkinsfile at the Jenkins master. Unfortunately, it does not seem easy to get a patch that can be applied:
** [/rest/api/1.0/projects/\{projectKey\}/repos/\{repositorySlug\}/pull-requests/\{pullRequestId\}.patch|https://docs.atlassian.com/bitbucket-server/rest/7.0.1/bitbucket-rest.html#idp289] gives the standard diff format but does not support limiting to a single file ; .
** [/rest/api/1.0/projects/\{projectKey}/repos/\{repositorySlug}/pull-requests/\{pullRequestId}/diff/\{path:.*}|https://docs.atlassian.com/bitbucket-server/rest/7.0.1/bitbucket-rest.html#idp229] supports that limiting to a single file, but the output is JSON ) and then apply the patch would thus be more difficult to Jenkinsfile at the Jenkins master apply .
- Alternatively, if Bitbucket Server has a REST API for querying the commit ID of a merge base of a pull request, then the plugin could first use that, then get Jenkinsfile from the three commits, and finally do the three-way merge. However, there does not seem to be such an API.

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 24, 2020, 4:02:16 PM3/24/20
to jenkinsc...@googlegroups.com
Alexey Pelykh commented on Improvement JENKINS-61493
 
Re: Lightweight checkout fails in Bitbucket Server 7.0; refs/pull-requests/*/merge does not exist

Seems that on 7.1.0 "/from" are not revealed via ls-remove as well

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 24, 2020, 4:24:03 PM3/24/20
to jenkinsc...@googlegroups.com

It seems that unless non-author user actually tries to view the PR, the "/from" ref is not created. So when I just use PRs on my own - "/from" is not created whatsoever, that's seems to be a severe case

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 24, 2020, 4:34:02 PM3/24/20
to jenkinsc...@googlegroups.com

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 24, 2020, 4:36:03 PM3/24/20
to jenkinsc...@googlegroups.com

Update: I can trigger appearance of the "/from" ref only by viewing the diff. Super-bad behaviour

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 24, 2020, 4:37:02 PM3/24/20
to jenkinsc...@googlegroups.com

The missing "/from" was filed two years ago as JENKINS-45997, and fixed in https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/116 by making the plugin poke the /rest/api/1.0/projects/{owner}/repos/{repo}/pull-requests/{id}/merge API, but that apparently stopped working in Bitbucket Server 7.

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 24, 2020, 4:39:02 PM3/24/20
to jenkinsc...@googlegroups.com

Kalle Niemitalo seems so, huge blocker  will take a look on various APIs maybe it's possible to trigger that somehow otherwise

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 24, 2020, 4:51:04 PM3/24/20
to jenkinsc...@googlegroups.com

I can think of two approaches that do not rely on Bitbucket Server updating the "/refs/pull-requests/{id}/from" ref:

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 24, 2020, 4:57:03 PM3/24/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo edited a comment on Improvement JENKINS-61493
I can think of two approaches that do not rely on Bitbucket Server updating the " / refs/pull-requests/\{id\}/from" ref:
* Take the commit ID from the [webhook event payload|https://confluence.atlassian.com/bitbucketserver070/event-payload-996644369.html]. In [BSERV-8268|https://jira.atlassian.com/browse/BSERV-8268], Atlassian enabled {{uploadpack.allowReachableSHA1InWant}}, so this should work even if that commit is no longer the tip of any branch, as long as it is still reachable. However, if a user of Jenkins manually triggers a build, then I guess the webhook payload will not be available.
* Query [/rest/api/1.0/projects/\{projectKey\}/repos/\{repositorySlug\}/pull-requests/\{pullRequestId\}|https://docs.atlassian.com/bitbucket-server/rest/7.0.1/bitbucket-rest.html#idp283] and get the name of the source branch from within the "fromRef" property.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 24, 2020, 5:03:03 PM3/24/20
to jenkinsc...@googlegroups.com

The newer Bitbucket Server Integration plugin does not even support pull requests yet, according to JENKINS-60342.

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 24, 2020, 6:40:02 PM3/24/20
to jenkinsc...@googlegroups.com

Second approach sounds really robust, yet I have no real idea what implications are there.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 24, 2020, 7:09:03 PM3/24/20
to jenkinsc...@googlegroups.com

It looks like that is what the Bitbucket Branch Source plugin does anyway, for Bitbucket Cloud. So, fixing the "refs/pull-requests/{id}/from" problem for pull requests within the same repository in Bitbucket Server could be just a matter of removing a few instanceof BitbucketCloudApiClient checks in BitbucketGitSCMBuilder.java and BitbucketSCMFileSystem.java. Fixing it for pull requests between forks would need more changes.

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 25, 2020, 11:11:02 AM3/25/20
to jenkinsc...@googlegroups.com

Call to `/rest/api/1.0/projects/{PROJECT}/repos/{REPO}/pull-requests/19/diff` causes `/from` to appear

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 25, 2020, 11:17:03 AM3/25/20
to jenkinsc...@googlegroups.com

Does Bitbucket Server ensure that the "refs/pull-requests/{id}/from" ref appears (or is updated) immediately after the diff request, or can there be a delay sometimes?

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 25, 2020, 11:19:02 AM3/25/20
to jenkinsc...@googlegroups.com

I'm checking manually, so no way to track the time between that. I've created a pull request, did ls-remove, made API call using Postman and re-did ls-remote to verify it appeared

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 25, 2020, 11:20:02 AM3/25/20
to jenkinsc...@googlegroups.com

I want to try HEAD instead of GET request next time

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Mar 25, 2020, 12:34:03 PM3/25/20
to jenkinsc...@googlegroups.com

I would recommend doing the "fromRef" thing instead of testing which REST requests update "refs/pull-requests/*/from" as an undocumented side effect.

alexey.pelykh@gmail.com (JIRA)

unread,
Mar 25, 2020, 12:36:02 PM3/25/20
to jenkinsc...@googlegroups.com

That sounds much better, yet I'm not sure I'm qualified enough to make such PR

rlandon@usa.net (JIRA)

unread,
Mar 30, 2020, 11:32:04 PM3/30/20
to jenkinsc...@googlegroups.com

While this is being sorted out, does anyone have a suggested workaround?

huber@dakosy.de (JIRA)

unread,
Mar 31, 2020, 8:19:04 AM3/31/20
to jenkinsc...@googlegroups.com

You can use the hpi published by this PR: https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/294
HPI: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/2.7.1-rc629.f242e60a14b4/

It falls back to heavyweight checkout, but at least PRs can be build with Bitbucket >= 7.x

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Apr 8, 2020, 2:18:22 PM4/8/20
to jenkinsc...@googlegroups.com

BSERV-12284 "Creating pull request does not reflect the stash-refs/pull-requests/<#> immediately" was moved to SHORT TERM BACKLOG on 2020-04-07. I wonder what Atlassian intends to do.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Apr 17, 2020, 7:09:02 AM4/17/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo edited a comment on Improvement JENKINS-61493
[BSERV-12284|https://jira.atlassian.com/browse/BSERV-12284] "Creating pull request does not reflect the stash-refs/pull-requests/<#> immediately" was moved to SHORT TERM BACKLOG on 2020-04-07. I wonder what Atlassian intends to do.

It was then moved down to GATHERING INTEREST on 2020-04-14, and the priority was removed. :(
Reply all
Reply to author
Forward
0 new messages