Announcing Jenkins-support for the Checks-API
653 views
Skip to first unread message
Thomas Dräbing
Jan 2, 2023, 5:12:16 AM1/2/23
to Repo and Gerrit Discussion
Dear Gerrit community,
during the last Gerrit User Summit I presented a first version of the Checks API
implementation for Jenkins CI servers. In the meantime, the plugin on Gerrit and
Jenkins-side were released and can be used.
How does it work?
In your Jenkins server you will need to install the Gerrit-Checks-Api plugin [1].
This plugin uses a Lucene index provided by the Lucene-search plugin [2], which is
installed as a dependency, to provide a REST API endpoint allowing to query for
builds (Runs) working on a patchset.
If you haven't used the Lucene-search plugin yet, you will have to build the
index once following the procedure described in [3]. Afterwards, the plugin will
add new builds automatically to the index.
The Jenkins server also needs to support CORS-requests coming from Gerrit. Since
Jenkins does not support that itself, a proxy like Nginx has to be used to add
the necessary headers to the request. An example on how to do this can be found
in [4].
In Gerrit, install the checks-jenkins plugin (e.g. for Gerrit 3.7 [5]). To enable
the plugin for a project, add a file called `checks-jenkins.config` to the project's
`refs/meta/config`-branch, containing the URL of the Jenkins-server(s):
```
[jenkins "gerrit-ci"]
url = https://gerrit-ci.gerritforge.com
```
The Checks-tab should now be visible in the Change Screen of Gerrit's UI and
display the data for builds run on the current patchset.
Note, that if the Jenkins instance does not allow anonymous access, the user has
to be logged into Jenkins with the same browser (The Jenkins UI does not have to
be open in a different tab). Otherwise, the plugin will not have the authorization
to access the data in Jenkins.
Why do I need another plugin on Jenkins?
The need for further required plugins arises due to how the remote API of Jenkins
core works and the existing Gerrit plugins.
Jenkins core does not include an index. Thus, searching for builds with certain
parameters is not efficient. Further, the remote access API provided by Jenkins [6]
does not allow to query but exposes a tree-like structure that can be navigated.
Thus, to get all builds working on a given patchset, each request would have to
look at every build of relevant jobs. On large Jenkins instances this would cause
quite some load, especially with a lot of clients trying to fetch this data with
regular polls. Thus, a more efficient way of finding relevant builds was required.
The lucene-search plugin provided that by providing a Lucene-index that allows to
query for builds, but did not expose a REST API and of course did not contain
Gerrit-specific knowledge.
So why not extend existing Gerrit-plugins for Jenkins, i.e. gerrit-trigger and
gerrit-code-review? As there are two completely independent plugins which might
even both be used on the same server, this would not work. In Jenkins, two plugins
cannot provide the same REST API endpoint. Having two different REST API endpoints
for mostly the same use case would not be nice. Also this would have required
double maintenance of the REST-API code.
For these reasons, I decided to create a separate plugin instead.
What are the current limitations?
- Unfortunately, the lucene-search plugin does not yet support Java 11 and with
it newer Jenkins versions (> 2.357). I have started working on changing that,
but this requires a Lucene update over a lot of versions with a lot of breaking
changes, so it might take me some time to get this done.
- The lucene-search plugin currently enforces the use of its search-functionality
in the Gerrit UI, which might not be desired by users. I have already created
a PR [7] to make this optional, but did not receive any feedback yet.
What is planned next?
- Allow to proxy requests through the Gerrit backend to avoid CORS.
- Support jobs that neither use the gerrit-trigger plugin nor the gerrit-code-review
plugin, but still work on a patchset.
- Add more data to the report (e.g. test results)
I am looking forward to your feedback and as always contributions are more than
welcome!
Best Regards,
Thomas
[1] https://plugins.jenkins.io/gerrit-checks-api/
[2] https://plugins.jenkins.io/lucene-search/
[3] https://plugins.jenkins.io/lucene-search/#plugin-content-database-rebuild
[4] https://enable-cors.org/server_nginx.html
[5] https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.7/job/plugin-checks-jenkins-bazel-master-stable-3.7/
[6] https://www.jenkins.io/doc/book/using/remote-access-api/
[7] https://github.com/jenkinsci/lucene-search-plugin/pull/57
during the last Gerrit User Summit I presented a first version of the Checks API
implementation for Jenkins CI servers. In the meantime, the plugin on Gerrit and
Jenkins-side were released and can be used.
How does it work?
In your Jenkins server you will need to install the Gerrit-Checks-Api plugin [1].
This plugin uses a Lucene index provided by the Lucene-search plugin [2], which is
installed as a dependency, to provide a REST API endpoint allowing to query for
builds (Runs) working on a patchset.
If you haven't used the Lucene-search plugin yet, you will have to build the
index once following the procedure described in [3]. Afterwards, the plugin will
add new builds automatically to the index.
The Jenkins server also needs to support CORS-requests coming from Gerrit. Since
Jenkins does not support that itself, a proxy like Nginx has to be used to add
the necessary headers to the request. An example on how to do this can be found
in [4].
In Gerrit, install the checks-jenkins plugin (e.g. for Gerrit 3.7 [5]). To enable
the plugin for a project, add a file called `checks-jenkins.config` to the project's
`refs/meta/config`-branch, containing the URL of the Jenkins-server(s):
```
[jenkins "gerrit-ci"]
url = https://gerrit-ci.gerritforge.com
```
The Checks-tab should now be visible in the Change Screen of Gerrit's UI and
display the data for builds run on the current patchset.
Note, that if the Jenkins instance does not allow anonymous access, the user has
to be logged into Jenkins with the same browser (The Jenkins UI does not have to
be open in a different tab). Otherwise, the plugin will not have the authorization
to access the data in Jenkins.
Why do I need another plugin on Jenkins?
The need for further required plugins arises due to how the remote API of Jenkins
core works and the existing Gerrit plugins.
Jenkins core does not include an index. Thus, searching for builds with certain
parameters is not efficient. Further, the remote access API provided by Jenkins [6]
does not allow to query but exposes a tree-like structure that can be navigated.
Thus, to get all builds working on a given patchset, each request would have to
look at every build of relevant jobs. On large Jenkins instances this would cause
quite some load, especially with a lot of clients trying to fetch this data with
regular polls. Thus, a more efficient way of finding relevant builds was required.
The lucene-search plugin provided that by providing a Lucene-index that allows to
query for builds, but did not expose a REST API and of course did not contain
Gerrit-specific knowledge.
So why not extend existing Gerrit-plugins for Jenkins, i.e. gerrit-trigger and
gerrit-code-review? As there are two completely independent plugins which might
even both be used on the same server, this would not work. In Jenkins, two plugins
cannot provide the same REST API endpoint. Having two different REST API endpoints
for mostly the same use case would not be nice. Also this would have required
double maintenance of the REST-API code.
For these reasons, I decided to create a separate plugin instead.
What are the current limitations?
- Unfortunately, the lucene-search plugin does not yet support Java 11 and with
it newer Jenkins versions (> 2.357). I have started working on changing that,
but this requires a Lucene update over a lot of versions with a lot of breaking
changes, so it might take me some time to get this done.
- The lucene-search plugin currently enforces the use of its search-functionality
in the Gerrit UI, which might not be desired by users. I have already created
a PR [7] to make this optional, but did not receive any feedback yet.
What is planned next?
- Allow to proxy requests through the Gerrit backend to avoid CORS.
- Support jobs that neither use the gerrit-trigger plugin nor the gerrit-code-review
plugin, but still work on a patchset.
- Add more data to the report (e.g. test results)
I am looking forward to your feedback and as always contributions are more than
welcome!
Best Regards,
Thomas
[1] https://plugins.jenkins.io/gerrit-checks-api/
[2] https://plugins.jenkins.io/lucene-search/
[3] https://plugins.jenkins.io/lucene-search/#plugin-content-database-rebuild
[4] https://enable-cors.org/server_nginx.html
[5] https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.7/job/plugin-checks-jenkins-bazel-master-stable-3.7/
[6] https://www.jenkins.io/doc/book/using/remote-access-api/
[7] https://github.com/jenkinsci/lucene-search-plugin/pull/57
Sven Selberg
Jan 2, 2023, 5:20:40 AM1/2/23
to Repo and Gerrit Discussion
Awesome Thomas, a big thank you for getting this in place!
Wishes for a good continuation of 2023!
/Sven
Sven Selberg
Jan 4, 2023, 8:28:22 AM1/4/23
to Repo and Gerrit Discussion
On Monday, January 2, 2023 at 11:20:40 AM UTC+1 Sven Selberg wrote:
On Monday, January 2, 2023 at 11:12:16 AM UTC+1 thomas....@gmail.com wrote:Dear Gerrit community,
during the last Gerrit User Summit I presented a first version of the Checks API
implementation for Jenkins CI servers. In the meantime, the plugin on Gerrit and
Jenkins-side were released and can be used.
How does it work?
In your Jenkins server you will need to install the Gerrit-Checks-Api plugin [1].
This plugin uses a Lucene index provided by the Lucene-search plugin [2], which is
installed as a dependency, to provide a REST API endpoint allowing to query for
builds (Runs) working on a patchset.
There have been some concerns raised internally around the fact that lucene-search-plugin
by-default indexes the console logs.
Do you know if it's possible to opt-out of indexing console logs?
I will try to get the Jenkins team to air their concerns regarding indexing console-logs as I
Do you know if it's possible to opt-out of indexing console logs?
I will try to get the Jenkins team to air their concerns regarding indexing console-logs as I
am not Jenkins-savvy enough to quite understand them myself.
Thomas Dräbing
Jan 4, 2023, 9:23:44 AM1/4/23
to Sven Selberg, Repo and Gerrit Discussion
On Wed, 4 Jan 2023 at 14:28, Sven Selberg <sven.s...@axis.com> wrote:
On Monday, January 2, 2023 at 11:20:40 AM UTC+1 Sven Selberg wrote:On Monday, January 2, 2023 at 11:12:16 AM UTC+1 thomas....@gmail.com wrote:Dear Gerrit community,
during the last Gerrit User Summit I presented a first version of the Checks API
implementation for Jenkins CI servers. In the meantime, the plugin on Gerrit and
Jenkins-side were released and can be used.
How does it work?
In your Jenkins server you will need to install the Gerrit-Checks-Api plugin [1].
This plugin uses a Lucene index provided by the Lucene-search plugin [2], which is
installed as a dependency, to provide a REST API endpoint allowing to query for
builds (Runs) working on a patchset.There have been some concerns raised internally around the fact that lucene-search-pluginby-default indexes the console logs.
Do you know if it's possible to opt-out of indexing console logs?
This is not configurable so far, but should not be too hard to do. Since the gerrit-checks-api-plugin does not use the indexed console logs, it would not affect us (it only uses the build parameters). Being able to configure what is indexed would also definitely help optimizing the performance and time required to build the index, so this definitely makes sense.
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/ffce4ad6-5e83-4bc7-a0fb-70de7ac385bbn%40googlegroups.com.
Phil Longstaff
Mar 28, 2023, 7:48:12 AM3/28/23
to Repo and Gerrit Discussion
Thanks for your work on this. I'm just curious if there is any update. Do you consider this beta software or is it ready for production use?
Phil
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAG7bb4B-KZ%3DqkHam59ZBoVwSSV2QSvWnVB98BdAcn%2Bj479KstQ%40mail.gmail.com.
CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient and may contain material that is proprietary, confidential, privileged or otherwise legally protected or restricted under applicable government laws. Any review, disclosure, distributing or other use without expressed permission of the sender is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies without reading, printing, or saving.
Thomas Dräbing
Mar 29, 2023, 2:02:04 AM3/29/23
to Phil Longstaff, Repo and Gerrit Discussion
Hi Phil,
Unfortunately, I was not able to put a lot of work into this. The current state is, that it works as expected and the security issues brought in by the lucene-search plugin are fixed. But unfortunately the plugin does not work with newer Jenkins versions yet, since the lucene-search plugin does not support Java 11. I started to work on this and also have a PR: https://github.com/jenkinsci/lucene-search-plugin/pull/61. The PR does not build, since the CI continues to use the Java 8 build container and I haven't figured out how to fix this. Let me ask the Jenkins team.
Best,
Thomas
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CABMdshEcRWYFQ%3DPnhBd_aeyMuNUviuwHjCTXnMB44DLmp6KC5A%40mail.gmail.com.
Thomas Dräbing
Jun 7, 2023, 3:12:35 PM6/7/23
to Tibor Baranya, Repo and Gerrit Discussion
Hi Tibor,
On Wed, Jun 7, 2023, 17:15 Tibor Baranya <ti...@arista.com> wrote:
Hey Thomas,I'm not sure if I'm supposed to write here, but I didn't find a better forum to ask myquestion.I'm trying to set up gerrit-checks-api on Jenkins and checks-jenkins on Gerrit, andbeside the lucene-search hickup (my organization uses folders heavily), now itseems to work.Except that if I push the RERUN button in Gerrit, it happily announces in the lowerleft corner that RUN TRIGGERED, except it didn't. Upon closer inspection it seemsit is trying towhich results in 404.When I try to retrigger from Jenkins, it is usingAs I understand the /index is not the important part here, but that the methodshall be POST.
New Jenkins versions require to use post requests as a security feature for sensitive requests. I didn't know about it when I originally implemented the Gerrit plugin. I will fix it asap (might be only by next week).
I didn't want to raise a bug on the Jenkins plugin, because this may be an issuewith the Gerrit plugin. Also I'm not even sure if this is really a bug, or just amisconfiguration on my side. Of course if you point me to the right place, I cancreate a ticket from this.
You can open a ticket at https://bugs.chromium.org/p/gerrit/issues/list?can=2. where bugs for Gerrit and it's plugins are tracked. Please add me as an owner.
Thanks,
Thomas
Thanks in advance,Tibi
Reply all
Reply to author
Forward
0 new messages