Accessing PR head commit sha from a pipeline using branch source plugin

3,772 views
Skip to first unread message

Julien HENRY

unread,
Jun 4, 2019, 5:16:48 PM6/4/19
to jenkin...@googlegroups.com
Hi,

We are trying to simplify integration of SonarQube with most popular pull request provider, like GitHub or Bitbucket.
Those ALM usually provide some API to report issues on PR (GitHub checks, Bitbucket Code Insight, ...). The APIs require to pass the PR HEAD commit sha1.

We are trying to get this commit ref automatically when the SonarQube scanner is executed in a Jenkins pipeline. The problem I'm facing is that Jenkins is creating a local merge of the PR head with the PR base branch, and the value of GIT_COMMIT is the ref of this local transcient commit, which is pretty useless for reporting back to the ALM.

Example:

For PR with HEAD at ab86fe3afa0b9473ea9da76c87872bf89b9b5f3b:
image.png

Using this pipeline:

node {
stage 'Checkout'
def scmVars = checkout scm
def commitHash = scmVars.GIT_COMMIT
println commitHash

}

Produces this log:

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/henryju/test-jenkins.git # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/henryju/test-jenkins.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --force --progress https://github.com/henryju/test-jenkins.git +refs/pull/1/head:refs/remotes/origin/PR-1 +refs/heads/master:refs/remotes/origin/master
Merging remotes/origin/master commit 57ff7166357a4939da69bc43db84961536557c21 into PR head commit ab86fe3afa0b9473ea9da76c87872bf89b9b5f3b
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ab86fe3afa0b9473ea9da76c87872bf89b9b5f3b
 > git merge 57ff7166357a4939da69bc43db84961536557c21 # timeout=10
 > git rev-parse HEAD^{commit} # timeout=10
Merge succeeded, producing 6a95262014cca41cf11e810401bd5c1bc0954515
Checking out Revision 6a95262014cca41cf11e810401bd5c1bc0954515 (PR-1)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6a95262014cca41cf11e810401bd5c1bc0954515
Commit message: "Merge commit '57ff7166357a4939da69bc43db84961536557c21' into HEAD"
 > git rev-list --no-walk ab86fe3afa0b9473ea9da76c87872bf89b9b5f3b # timeout=10


GIT_COMMIT will return 6a95262014cca41cf11e810401bd5c1bc0954515 while this is useless for reporting issues in GitHub. Is there any simple way (except some fragile local Git operation) to access the original Git sha1 that triggered the PR build? In my example I need to get ab86fe3afa0b9473ea9da76c87872bf89b9b5f3b

Note that I have the same need with the Bitbucket Branch Source Plugin.

Thanks,

Julien Henry | SonarSource

Developer

https://sonarsource.com

Jesse Glick

unread,
Jun 4, 2019, 9:07:31 PM6/4/19
to Jenkins Dev
On Tue, Jun 4, 2019 at 5:16 PM Julien HENRY
<julien...@sonarsource.com> wrote:
> Is there any simple way (except some fragile local Git operation) to access the original Git sha1 that triggered the PR build? In my example I need to get ab86fe3afa0b9473ea9da76c87872bf89b9b5f3b

There is not currently any environmental variable for that purpose
that I know of. You can simply try publishing a commit status for the
hash you are given, and if that gives a 422 response, try its first
parent instead, as I suggested in the thread leading to

https://jira.sonarsource.com/browse/SONAR-10794

Joseph P

unread,
Jun 5, 2019, 6:37:37 AM6/5/19
to Jenkins Developers
Jesse that is simply not true for branch source plugins:

branch source plugin uses branch API plugin which contributes the environment variables.

Hopefully, you find that helpful Julien

Joseph P

unread,
Jun 5, 2019, 6:39:19 AM6/5/19
to Jenkins Developers
To clarify from the SCMHead: you should be able to get the commit sha

Joseph P

unread,
Jun 5, 2019, 6:54:37 AM6/5/19
to Jenkins Developers
If SCMHead is not the thing you want:
You can retrieve from the git build data:

Julien HENRY

unread,
Jun 5, 2019, 8:21:01 AM6/5/19
to Jenkins Developers
Hi Jesse,

Publishing of PR status happens on SonarQube server side, asynchronously. Since the commit hash we are collecting currently in the pipeline is something transient that doesn't exists on GitHub, it is impossible to find the parent. We would have to collect the parent in advance directly in the scanner/pipeline, running some Git commands. I started to think about it, but I see many corner cases.

First issue is that Jenkins offers multiple PR strategies (with merge or not). So we would have to do some logic to find if current commit is a merge commit, or maybe look directly for symbolic ref refs/remotes/origin/${BRANCH_NAME} ?
Second issue is that I know some users are doing some Git operations in their pipeline. That could affect our heuristic since our scanner may be called after they already manipulated the repo.

In my opinion it would be much more reliable to have the branch source plugin exposing it as an env variable directly. I don't think this is something too SonarQube specific, since this commit hash is what is needed to report PR status/GitHub check/Bitbucket Code Insight.

@Joseph thanks for the tips, but we would like our scanner to be used in a Jenkins pipeline as a "normal" command line tool. So we don't have access to Jenkins API, only to the job env variables, and possibly doing some Git operations.

Jesse Glick

unread,
Jun 5, 2019, 9:58:37 AM6/5/19
to Jenkins Dev
On Wed, Jun 5, 2019 at 6:37 AM Joseph P <jose...@gmail.com> wrote:
> branch source plugin uses branch API plugin which contributes the environment variables.
> There is also a way for plugins to retrieve this see https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/BranchNameContributor.java

Yes `BranchNameContributor` adds certain environment variables, but
none which would indicate a PR head commit hash. That is a concept
which is specific to GitHub (or analogous systems), so it cannot be in
`branch-api`. For example, `github-branch-source` _could_ be enhanced
to define an `EnvironmentContributor` for
`PullRequestSCMRevision.pullHash`. Alternately,
`ChangeRequestSCMRevision` could be enhanced to allow subtypes to
define an opaque “commit ID” token, in which case
`BranchNameContributor` could bind it when set and
`PullRequestSCMRevision` and its analogues could implement the API.

Jesse Glick

unread,
Jun 5, 2019, 10:03:34 AM6/5/19
to Jenkins Dev
On Wed, Jun 5, 2019 at 8:21 AM Julien HENRY
<julien...@sonarsource.com> wrote:
> Since the commit hash we are collecting currently in the pipeline is something transient that doesn't exists on GitHub

Note that as of JENKINS-43194 it _may_ be visible from GitHub APIs. I
would not advise relying on that, though.

> Jenkins offers multiple PR strategies (with merge or not). So we would have to do some logic to find if current commit is a merge commit

Well my offhand suggestion was:

· Try publishing a status for the recorded commit as is. If that worked, great.
· If that gave a 422, and the commit was a merge commit, try
publishing a status for its parent parent.

> some users are doing some Git operations in their pipeline. That could affect our heuristic since our scanner may be called after they already manipulated the repo.

Yes, this is always a risk.

Julien HENRY

unread,
Jun 5, 2019, 11:06:03 AM6/5/19
to Jenkins Developers
FYI, the kind of trick I would like to avoid to maintain is what this project is doing:

Jesse Glick

unread,
Jun 5, 2019, 12:11:17 PM6/5/19
to Jenkins Dev
On Wed, Jun 5, 2019 at 11:06 AM Julien HENRY
<julien...@sonarsource.com> wrote:
> https://github.com/deis/workflow-cli/blob/a285f6914f10ce81d0afe5d53603aa36cda78ccd/Jenkinsfile#L67

That particular implementation is wrong anyway, since the PR head
commit might happen to be a merge commit, for example by a developer
resolving conflicts with the base branch.

Julien HENRY

unread,
Jun 12, 2019, 3:56:28 AM6/12/19
to Jenkins Developers
Hi Jesse,

Our users are strongly asking for a better support of PR status update by SonarQube/SonarCloud when build is triggered by Jenkins. We don't like the try-fail strategy (trying to annotate current commit, then fallback on parent).
Would you be ok to accept a contribution from SonarSource where we would add a new env variable (like CHANGE_COMMIT) to expose the PR head commit sha to the pipeline? We are interested to support both GitHub and Bitbucket, so we would ensure it works at least for those 2 plugins.

++

Julien

Jesse Glick

unread,
Jun 12, 2019, 10:54:44 AM6/12/19
to Jenkins Dev
On Wed, Jun 12, 2019 at 3:56 AM Julien HENRY
<julien...@sonarsource.com> wrote:
> Would you be ok to accept a contribution from SonarSource where we would add a new env variable (like CHANGE_COMMIT) to expose the PR head commit sha to the pipeline? We are interested to support both GitHub and Bitbucket, so we would ensure it works at least for those 2 plugins.

Sounds good, especially if it is done the right way using
`ChangeRequestSCMRevision` as previously noted.

Ivan Fernandez Calvo

unread,
Jun 13, 2019, 7:46:31 AM6/13/19
to Jenkins Developers
we faced the same problem a few months ago, we've resolved it by adding a calculated environment variable for more details see https://groups.google.com/forum/#!topic/jenkinsci-dev/Dus0VX6y04g 

Julien HENRY

unread,
Jun 13, 2019, 8:53:40 AM6/13/19
to jenkin...@googlegroups.com
Hi Ivan,

Thanks for the notice. Your solution is heavily relying on the presence of the GIT_COMMIT variable, which was not defined out of the box in my tests. I had to get it from the return of the checkout step, which is not a good solution for us, since we want our step to work without asking users to do extra logic in their pipeline.


Julien Henry | SonarSource

Developer

https://sonarsource.com


Le jeu. 13 juin 2019 à 13:46, Ivan Fernandez Calvo <kuisat...@gmail.com> a écrit :
we faced the same problem a few months ago, we've resolved it by adding a calculated environment variable for more details see https://groups.google.com/forum/#!topic/jenkinsci-dev/Dus0VX6y04g 

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/-ojqklsaKbw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ee8163ba-5f14-48f8-b888-0983d67e22f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jesse Glick

unread,
Jun 13, 2019, 9:44:42 AM6/13/19
to Jenkins Dev
On Thu, Jun 13, 2019 at 8:53 AM Julien HENRY
<julien...@sonarsource.com> wrote:
> the GIT_COMMIT variable […] was not defined out of the box in my tests. I had to get it from the return of the checkout step

Yes, each `checkout` step in the build, of which there could be zero
or more, can return its own metadata.

For a branch project (such as a build of a GH PR),
`BranchNameContributor` sets build-wide environment variables
corresponding to the revision associated with the build overall. This
is what you would get if you `checkout scm`.

Julien HENRY

unread,
Jul 2, 2019, 11:46:42 AM7/2/19
to jenkin...@googlegroups.com
Hi Jesse,

Here are 3 PRs to implement the proposed change. I did only Bitbucket so far, but if you confirm I'm on the good track, I will submit the same for GitHub (I have the changes locally, just haven't tested end-to-end).

I think having a JIRA ticket to refer would be great, should I create one?

I'm open to discussion to improve the code, either on this thread or on PRs themselves.

++
Julien Henry | SonarSource
--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/-ojqklsaKbw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

Liam Newman

unread,
Jul 2, 2019, 1:29:55 PM7/2/19
to Jenkins Developers
Yes, you should absolutely create a JIRA. 
To unsubscribe from this group and all its topics, send an email to jenkin...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages