Build Gerrit change-topic

372 views
Skip to first unread message

Nikita Pustovoi

unread,
Apr 5, 2021, 12:16:25 PM4/5/21
to Repo and Gerrit Discussion
Hi all,

I was wondering if anyone can share some light how one or android-review can build all changes in a topic as one?

The idea is when you develop AOSP, sometimes you have changes across multiple repos, and you want to validate & merge these changes in one go.

Gerrit solves this problem with 'topic'  ( and change.submitWholeTopic = true)

I was wondering if anyone can share how they do their CI based on the topic

My initial idea is, I can have a server listening for gerrit-events, and when a topic is submitted/updated - my server can generate a temporary manifest (based on develop/master branch, but include the changes in topic) & submit this to the CI to validate.

Has anyone else done something similar? Any hits how Google does it with android-review ?

Thanks. 

Christian Gagneraud

unread,
Apr 6, 2021, 5:26:48 AM4/6/21
to Repo and Gerrit Discussion
I would be interesting to know too. I find it weird, that one has to write lot of gluing code to make gerrit and jenkins act together.

Chris

Jonathan Nieder

unread,
Apr 15, 2021, 10:04:23 PM4/15/21
to Christian Gagneraud, Repo and Gerrit Discussion
Hi!

Android's presubmit tests use https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#submitted-together to find the changes that would be submitted together with a given change, so they can be tested together.

Another available tool is https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#submit-preview, which gives a preview of exactly what would happen if you were to submit a given change and all its dependencies. Android's CI does not use that because it needs more control over the baseline that tests run against: instead of running against HEAD, tests can run against a "last-known good revision".

A side note: for single-repository projects, it can be nice to be able to test a change in exactly the form and against exactly the same baseline that its author had locally, instead of pulling in other changes that have landed in parallel --- that way, any test failures are more meaningful to the change uploader since they match what they have been testing locally. That's not possible today with AOSP, because Gerrit does not *see* what commit the author is working against in repositories they did not modify. One idea the Android CI team and I have chatted about a while ago is increased use of submodules: if the normal review flow were to involve uploading a change in https://android.googlesource.com/platform/superproject along with the changes in its various submodules, then that baseline information would be available, opening up some possibilities. Having that work well involves some other changes to Gerrit (to be able to review a superproject change and submodule changes on the same screen) so it's not something to expect in the near future, but it's something I've been interested in. Let me know if you're interested, too.

Jonathan

Nasser Grainawi

unread,
Apr 20, 2021, 12:44:20 PM4/20/21
to Christian Gagneraud, Repo and Gerrit Discussion
We created the batch plugin https://gerrit.googlesource.com/plugins/batch for exactly this purpose. We haven’t provided an out-of-the-box integration with Jenkins yet, but the idea would be to 1) poll using gerrit query, then 2) use the batch merge-change command to add whichever changes you want to validate together to a single batch, 3) download the automatically created batch ref for each repo into your CI workspace, and finally 4) use batch submit (if you want) to update the destination branch for each project in the batch to be exactly the ref/SHA-1 you downloaded in 3 and close the changes in the batch.

The docs have a lot more detail on those commands and I’d be happy to help answer more questions if you have any.

As an aside, our CI teams use the task plugin https://gerrit.googlesource.com/plugins/task for finding the correct changes during that polling in step #1.

Nasser


Chris

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/e2abef34-7a20-47ee-b462-bddc52bb4e76n%40googlegroups.com.

Christian Gagneraud

unread,
Aug 13, 2021, 7:57:26 PM8/13/21
to Repo and Gerrit Discussion
On Wednesday, 21 April 2021 at 04:44:20 UTC+12 nas...@codeaurora.org wrote:
> We created the batch plugin https://gerrit.googlesource.com/plugins/batch for exactly this purpose. We haven’t provided an out-of-the-box integration with Jenkins yet, but the idea would be to 1) poll using gerrit query, then 2) use the batch merge-change command to add whichever changes you want to validate together to a single batch, 3) download the automatically created batch ref for each repo into your CI workspace, and finally 4) use batch submit (if you want) to update the destination branch for each project in the batch to be exactly the ref/SHA-1 you downloaded in 3 and close the changes in the batch.

Thanks for sharing, we're now moving forward and looking at setting up a gerrit server for initial real world trial (AOSP project).
 
> The docs have a lot more detail on those commands and I’d be happy to help answer more questions if you have any.

Would you have a link to the docs?
I've checked these and couldn't find any documentation:

> As an aside, our CI teams use the task plugin https://gerrit.googlesource.com/plugins/task for finding the correct changes during that polling in step #1.

Will look into that.

Chris

Nasser Grainawi

unread,
Aug 16, 2021, 10:50:43 AM8/16/21
to Christian Gagneraud, Repo and Gerrit Discussion
On Aug 13, 2021, at 5:57 PM, Christian Gagneraud <chg...@gmail.com> wrote:

On Wednesday, 21 April 2021 at 04:44:20 UTC+12 nas...@codeaurora.org wrote:
> We created the batch plugin https://gerrit.googlesource.com/plugins/batch for exactly this purpose. We haven’t provided an out-of-the-box integration with Jenkins yet, but the idea would be to 1) poll using gerrit query, then 2) use the batch merge-change command to add whichever changes you want to validate together to a single batch, 3) download the automatically created batch ref for each repo into your CI workspace, and finally 4) use batch submit (if you want) to update the destination branch for each project in the batch to be exactly the ref/SHA-1 you downloaded in 3 and close the changes in the batch.

Thanks for sharing, we're now moving forward and looking at setting up a gerrit server for initial real world trial (AOSP project).
 
> The docs have a lot more detail on those commands and I’d be happy to help answer more questions if you have any.

Would you have a link to the docs?
I've checked these and couldn't find any documentation:

Ah sorry, they’re hidden in the source tree where they get turned into the installed docs: https://gerrit.googlesource.com/plugins/batch/+/refs/heads/master/src/main/resources/Documentation

I’ll see if we can add a link from a top-level README or something.


> As an aside, our CI teams use the task plugin https://gerrit.googlesource.com/plugins/task for finding the correct changes during that polling in step #1.

Will look into that.


Chris

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Christian Gagneraud

unread,
Aug 16, 2021, 11:34:21 PM8/16/21
to Nasser Grainawi, Repo and Gerrit Discussion
On Tue, 17 Aug 2021 at 02:50, Nasser Grainawi <nas...@codeaurora.org> wrote:
> On Aug 13, 2021, at 5:57 PM, Christian Gagneraud <chg...@gmail.com> wrote:
> On Wednesday, 21 April 2021 at 04:44:20 UTC+12 nas...@codeaurora.org wrote:
> > We created the batch plugin https://gerrit.googlesource.com/plugins/batch for exactly this purpose. We haven’t provided an out-of-the-box integration with Jenkins yet, but the idea would be to 1) poll using gerrit query, then 2) use the batch merge-change command to add whichever changes you want to validate together to a single batch, 3) download the automatically created batch ref for each repo into your CI workspace, and finally 4) use batch submit (if you want) to update the destination branch for each project in the batch to be exactly the ref/SHA-1 you downloaded in 3 and close the changes in the batch.
>
> Thanks for sharing, we're now moving forward and looking at setting up a gerrit server for initial real world trial (AOSP project).
>
> > The docs have a lot more detail on those commands and I’d be happy to help answer more questions if you have any.
[...]
> Ah sorry, they’re hidden in the source tree where they get turned into the installed docs: https://gerrit.googlesource.com/plugins/batch/+/refs/heads/master/src/main/resources/Documentation
[...]
> > As an aside, our CI teams use the task plugin https://gerrit.googlesource.com/plugins/task for finding the correct changes during that polling in step #1.
[...]
> Similarly, docs for task are at https://gerrit.googlesource.com/plugins/task/+/refs/heads/master/src/main/resources/Documentation/task.md

Thanks for sharing, It's starting to make sense in my head! :)

At the same time i'm looking on the jenkins side, there seems to be 2
main plugins
- https://plugins.jenkins.io/gerrit-trigger/
This looks like an old style system to me (all conf is in jenkins, not
under source control)
Haven't explored more, b/c for me it's a big no-no straight away.
- https://plugins.jenkins.io/gerrit-code-review/
This one looks more interesting, it's more "pipeline as code" oriented.
It refers to a gerrit "Checks" plugin with a dead link.
By the look of it, this seems to correspond to the "tasks" plugin you mentioned.
The code might not be up to date:
https://github.com/jenkinsci/gerrit-code-review-plugin/blob/master/src/main/java/jenkins/plugins/gerrit/workflow/GerritCheckStep.java

So if I go to the AOSP status page
(https://android-review.googlesource.com/q/project:platform/frameworks/support+status:open+branch:androidx-master-dev),
all the columns on the right are tasks defined using the "Tasks"
plugin?
And following a given CR link, these translate into "Submit
requirements" and "Other labels".

I can see a "Checks" section, but I do not have permissions to access
them. Are these the "checks" referred to by the "gerrit-code-review"
plugin for Jenkins?

Thanks a lot for helping me get my head around.

Chris

Christian Gagneraud

unread,
Aug 17, 2021, 1:26:54 AM8/17/21
to Nasser Grainawi, Repo and Gerrit Discussion
On Wed, 21 Apr 2021 at 04:44, Nasser Grainawi <nas...@codeaurora.org> wrote:
> On Apr 6, 2021, at 3:26 AM, Christian Gagneraud <chg...@gmail.com> wrote:
> On Tuesday, 6 April 2021 at 04:16:25 UTC+12 deis...@gmail.com wrote:
>>
>> Hi all,
>>
>> I was wondering if anyone can share some light how one or android-review can build all changes in a topic as one?
>>
>> The idea is when you develop AOSP, sometimes you have changes across multiple repos, and you want to validate & merge these changes in one go.
>>
>> Gerrit solves this problem with 'topic' ( and change.submitWholeTopic = true)
>>
>> I was wondering if anyone can share how they do their CI based on the topic
>>
>> My initial idea is, I can have a server listening for gerrit-events, and when a topic is submitted/updated - my server can generate a temporary manifest (based on develop/master branch, but include the changes in topic) & submit this to the CI to validate.
>>
>> Has anyone else done something similar? Any hits how Google does it with android-review ?
>
>
> I would be interesting to know too. I find it weird, that one has to write lot of gluing code to make gerrit and jenkins act together.
>
>
> We created the batch plugin https://gerrit.googlesource.com/plugins/batch for exactly this purpose. We haven’t provided an out-of-the-box integration with Jenkins yet, but the idea would be to 1) poll using gerrit query, then 2) use the batch merge-change command to add whichever changes you want to validate together to a single batch, 3) download the automatically created batch ref for each repo into your CI workspace, and finally 4) use batch submit (if you want) to update the destination branch for each project in the batch to be exactly the ref/SHA-1 you downloaded in 3 and close the changes in the batch.
>
> The docs have a lot more detail on those commands and I’d be happy to help answer more questions if you have any.
>
> As an aside, our CI teams use the task plugin https://gerrit.googlesource.com/plugins/task for finding the correct changes during that polling in step #1.

Hmmm, I will have to downgrade gerrit to 3.3 as these plugins are not
available for 3.4, which seems to be the latest stable

Chris

Han-Wen Nienhuys

unread,
Aug 17, 2021, 5:28:28 AM8/17/21
to Christian Gagneraud, Nasser Grainawi, Repo and Gerrit Discussion
On Tue, Aug 17, 2021 at 5:34 AM Christian Gagneraud <chg...@gmail.com> wrote:

> So if I go to the AOSP status page
> (https://android-review.googlesource.com/q/project:platform/frameworks/support+status:open+branch:androidx-master-dev),
> all the columns on the right are tasks defined using the "Tasks"
> plugin?

The AOSP gerrit host does not use the 'tasks' plugin.

> I can see a "Checks" section, but I do not have permissions to access
> them. Are these the "checks" referred to by the "gerrit-code-review"
> plugin for Jenkins?

This is the new CI integration, which debuted in Gerrit 3.4. It runs
in the browser; the data is provided by a separate service run by the
CI team for Android. If you go to
https://chromium-review.googlesource.com you can see the same UI,
populated with publicly visible data.

We are planning a frontend plugin that will fetch data from Jenkins,
but I think work on this hasn't started yet.

--
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Luca Milanesio

unread,
Aug 17, 2021, 5:57:25 AM8/17/21
to Repo and Gerrit Discussion, Luca Milanesio, Christian Gagneraud, Nasser Grainawi, Han-Wen Nienhuys


> On 17 Aug 2021, at 10:28, 'Han-Wen Nienhuys' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>
> On Tue, Aug 17, 2021 at 5:34 AM Christian Gagneraud <chg...@gmail.com> wrote:
>
>> So if I go to the AOSP status page
>> (https://android-review.googlesource.com/q/project:platform/frameworks/support+status:open+branch:androidx-master-dev),
>> all the columns on the right are tasks defined using the "Tasks"
>> plugin?
>
> The AOSP gerrit host does not use the 'tasks' plugin.
>
>> I can see a "Checks" section, but I do not have permissions to access
>> them. Are these the "checks" referred to by the "gerrit-code-review"
>> plugin for Jenkins?
>
> This is the new CI integration, which debuted in Gerrit 3.4. It runs
> in the browser; the data is provided by a separate service run by the
> CI team for Android. If you go to
> https://chromium-review.googlesource.com you can see the same UI,
> populated with publicly visible data.
>
> We are planning a frontend plugin that will fetch data from Jenkins,
> but I think work on this hasn't started yet.

We are planning to work on it and release it for Gerrit v3.5. It will be integrated with the "Gerrit-Code-Review" plugin for Jenkins.

HTH

Luca.

>
> --
> Han-Wen Nienhuys - Google Munich
> I work 80%. Don't expect answers from me on Fridays.
> --
>
> Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
>
> Registergericht und -nummer: Hamburg, HRB 86891
>
> Sitz der Gesellschaft: Hamburg
>
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAFQ2z_PL_ZJZW6Rn8JbGPff%2B_pXZfXdePg3COO6%2B0J_B-RsBnA%40mail.gmail.com.

Nasser Grainawi

unread,
Aug 17, 2021, 10:59:40 AM8/17/21
to Christian Gagneraud, Repo and Gerrit Discussion
The master branch of task works with 3.4 (and tip, thus why it hasn’t branched off yet). Looks like batch is a little behind, let me check on that today.


Chris

Nasser Grainawi

unread,
Aug 18, 2021, 8:31:03 PM8/18/21
to Christian Gagneraud, Repo and Gerrit Discussion
315524: Merge branch 'stable-3.3' | https://gerrit-review.googlesource.com/c/plugins/batch/+/315524 updated batch’s master branch to work with 3.4.

Let me know if you have any issues with those.



Chris


-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages