Plugin CI split between Zuul and Jenkins still?

[Nasser Grainawi]

Hey folks,

I'm trying to better understand the current CI setup for plugins on gerrit-review. Seems like Zuul is configured to run CI on changes and Jenkins runs post-flight (branch tips)? And then https://www.gerritcodereview.com/plugins.html#compatibility-matrix and plugin-manager use artifacts from Jenkins?

Has anyone discussed consolidating those (probably by making Zuul produce the artifacts for downstream consumption)?

Thanks,

Nasser

[Luca Milanesio]

On 6 Jul 2021, at 19:39, Nasser Grainawi <nas...@codeaurora.org> wrote:

Hey folks,

I'm trying to better understand the current CI setup for plugins on gerrit-review. Seems like Zuul is configured to run CI on changes and Jenkins runs post-flight (branch tips)? And then https://www.gerritcodereview.com/plugins.html#compatibility-matrix and plugin-manager use artifacts from Jenkins?

Plugin-manager has a configurable URL from where to pull the plugins from: the default is from Jenkins because that’s what we’ve got at the moment.

Has anyone discussed consolidating those (probably by making Zuul produce the artifacts for downstream consumption)?

I believe that’s the plan.

Luca.

[Nasser Grainawi]

On Jul 6, 2021, at 1:25 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 6 Jul 2021, at 19:39, Nasser Grainawi <nas...@codeaurora.org> wrote:

Hey folks,

I'm trying to better understand the current CI setup for plugins on gerrit-review. Seems like Zuul is configured to run CI on changes and Jenkins runs post-flight (branch tips)? And then https://www.gerritcodereview.com/plugins.html#compatibility-matrix and plugin-manager use artifacts from Jenkins?

Plugin-manager has a configurable URL from where to pull the plugins from: the default is from Jenkins because that’s what we’ve got at the moment.

Make sense. Thanks for confirming.

Has anyone discussed consolidating those (probably by making Zuul produce the artifacts for downstream consumption)?

I believe that’s the plan.

Anyone working on that? Is there a design for how it should work?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2C058BE7-9D90-4190-804E-5303519B42E4%40gmail.com.

[Luca Milanesio]

On 6 Jul 2021, at 20:28, Nasser Grainawi <nas...@codeaurora.org> wrote:

On Jul 6, 2021, at 1:25 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 6 Jul 2021, at 19:39, Nasser Grainawi <nas...@codeaurora.org> wrote:

Hey folks,

I'm trying to better understand the current CI setup for plugins on gerrit-review. Seems like Zuul is configured to run CI on changes and Jenkins runs post-flight (branch tips)? And then https://www.gerritcodereview.com/plugins.html#compatibility-matrix and plugin-manager use artifacts from Jenkins?

Plugin-manager has a configurable URL from where to pull the plugins from: the default is from Jenkins because that’s what we’ve got at the moment.

Make sense. Thanks for confirming.

Has anyone discussed consolidating those (probably by making Zuul produce the artifacts for downstream consumption)?

I believe that’s the plan.

Anyone working on that? Is there a design for how it should work?

@James? Any plans on how and when to tackle this?

Luca.

Luca.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2C058BE7-9D90-4190-804E-5303519B42E4%40gmail.com.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/DF5917F8-F231-4E80-B003-2F4B6F76D5EA%40codeaurora.org.

[James E. Blair]

Luca Milanesio <luca.mi...@gmail.com> writes:

>> On 6 Jul 2021, at 20:28, Nasser Grainawi <nas...@codeaurora.org> wrote:
>>
>>
>>

>>> On Jul 6, 2021, at 1:25 PM, Luca Milanesio <luca.mi...@gmail.com <mailto:luca.mi...@gmail.com>> wrote:

>>>
>>>
>>>
>>>> On 6 Jul 2021, at 19:39, Nasser Grainawi <nas...@codeaurora.org <mailto:nas...@codeaurora.org>> wrote:
>>>>
>>>> Hey folks,
>>>>
>>>> I'm trying to better understand the current CI setup for plugins
>>>> on gerrit-review. Seems like Zuul is configured to run CI on
>>>> changes and Jenkins runs post-flight (branch tips)? And then
>>>> https://www.gerritcodereview.com/plugins.html#compatibility-matrix
>>>> <https://www.gerritcodereview.com/plugins.html#compatibility-matrix>
>>>> and plugin-manager use artifacts from Jenkins?
>>>
>>> Plugin-manager has a configurable URL from where to pull the
>>> plugins from: the default is from Jenkins because that’s what we’ve
>>> got at the moment.
>>
>> Make sense. Thanks for confirming.
>>
>>>
>>>>
>>>> Has anyone discussed consolidating those (probably by making Zuul produce the artifacts for downstream consumption)?
>>>
>>> I believe that’s the plan.
>>
>> Anyone working on that? Is there a design for how it should work?
>
> @James? Any plans on how and when to tackle this?

Building and making the artifacts available should be easy: we already
build and upload an artifact to log storage as part of the check job, so
we know that works. Example:

https://ci.gerritcodereview.com/t/gerrit/build/f2566378269146b8814d41979636457e/artifacts

click on "Build", and you get this URL:

https://storage.googleapis.com/gerrit_zuul_logs2/42/311342/1/check/gerrit-plugin-build/f256637/artifacts/healthcheck.jar

(The content-type is wrong, but the data are correct; that's a usable jar.)

That auto-deletes along with the logs, so it's not a suitable
publication location. But we can make a new GCS container for
publishing artifacts, and build them after changes merge and push them
there.

Here's what I propose:

* Create a new job that runs in the "post" pipeline (jobs run after each
change merges, and in the context of the newly-updated ref).

* It builds the artifact in the same way as the check job (but omits
testing).

* It uploads the artifact to a new GCS container, under the path:

/{reponame}/{branch}/{artifact.jar}

Example: /plugins/healthcheck/master/healthcheck.jar

If that sounds reasonable, I can do that relatively soon.

The next part, as I understand it, may be a little more complex and I
won't be able to do alone.

If I understand correctly, the plugin-manager uses the Jenkins API to
learn what plugins exist and for what branches[1]. We probably want to
implement a new repository class that either gets the data from Zuul's
API, or perhaps directly from GCS (maybe by crawling a predictable
directory structure).

I'd particularly like advice from folks familiar with the plugin-manager
to suggest a path there.

-Jim

[1] https://gerrit.googlesource.com/plugins/plugin-manager/+/refs/heads/master/src/main/java/com/googlesource/gerrit/plugins/manager/repository/JenkinsCiPluginsRepository.java

[James E. Blair]

After thinking on this a bit more, I think this may be a better plan for
the second phase (updating the plugin-manager plugin):

1) Publish a JSON file with all of the build information to GCS. An
example URL might be:

https://storage.googleapis.com/some_container/plugins.json

We'll have the Zuul plugin build jobs update that after each build.

2) Since that's not a great URL to encode into software, can we set up a
redirect on gerritcodereview.org? Something like:

https://gerritcodereview.org/plugins.json ->
https://storage.googleapis.com/some_container/plugins.json

3) Update plugin-manager to read the JSON file from that URL (and follow
redirects).

This scheme has the following advantages:

* Build-system agnostic (we're not tied to any particular build system
or its API).
* We can import the existing builds from Jenkins.
* Scalable (GCS carries the load).
* Easier re-use by other tools (can watch the json file for changes).

How does that sound?

-Jim

[Nasser Grainawi]

On Jul 7, 2021, at 2:30 PM, James E. Blair <j...@acmegating.com> wrote:

"James E. Blair" <j...@acmegating.com> writes:

Building and making the artifacts available should be easy: we already
build and upload an artifact to log storage as part of the check job, so
we know that works.  Example:

 https://ci.gerritcodereview.com/t/gerrit/build/f2566378269146b8814d41979636457e/artifacts

click on "Build", and you get this URL:

 https://storage.googleapis.com/gerrit_zuul_logs2/42/311342/1/check/gerrit-plugin-build/f256637/artifacts/healthcheck.jar

(The content-type is wrong, but the data are correct; that's a usable jar.)

That auto-deletes along with the logs, so it's not a suitable
publication location.  But we can make a new GCS container for
publishing artifacts, and build them after changes merge and push them
there.

Here's what I propose:

* Create a new job that runs in the "post" pipeline (jobs run after each
 change merges, and in the context of the newly-updated ref).

* It builds the artifact in the same way as the check job (but omits
 testing).

Should it really omit testing? I know we typically have changes going through verification before they submit, but two changes could independently work and then when both are submitted tests would fail. Seems like it’d be safer to test in the post pipeline too.

I think that sounds pretty good. In addition to the plugin-manager, we should plan to update https://www.gerritcodereview.com/plugins.html based on that json. That page is currently built using https://gerrit.googlesource.com/homepage/+/refs/heads/master/tools/plugins.py

Does that help to clarify requirements for the json content?

I don’t know much about how the homepage site is built or the plugin-manager code itself, so I’ll let others comment on the specifics in #2 and #3, but at a high level I like the plan.

-Jim

[James E. Blair]

Nasser Grainawi <nas...@codeaurora.org> writes:

>> On Jul 7, 2021, at 2:30 PM, James E. Blair <j...@acmegating.com> wrote:
>>
>> "James E. Blair" <j...@acmegating.com> writes:
>>
>>>
>>> Building and making the artifacts available should be easy: we already
>>> build and upload an artifact to log storage as part of the check job, so
>>> we know that works. Example:
>>>
>>> https://ci.gerritcodereview.com/t/gerrit/build/f2566378269146b8814d41979636457e/artifacts
>>>
>>> click on "Build", and you get this URL:
>>>
>>> https://storage.googleapis.com/gerrit_zuul_logs2/42/311342/1/check/gerrit-plugin-build/f256637/artifacts/healthcheck.jar
>>>
>>> (The content-type is wrong, but the data are correct; that's a usable jar.)
>>>
>>> That auto-deletes along with the logs, so it's not a suitable
>>> publication location. But we can make a new GCS container for
>>> publishing artifacts, and build them after changes merge and push them
>>> there.
>>>
>>> Here's what I propose:
>>>
>>> * Create a new job that runs in the "post" pipeline (jobs run after each
>>> change merges, and in the context of the newly-updated ref).
>>>
>>> * It builds the artifact in the same way as the check job (but omits
>>> testing).
>
> Should it really omit testing? I know we typically have changes going
> through verification before they submit, but two changes could
> independently work and then when both are submitted tests would
> fail. Seems like it’d be safer to test in the post pipeline too.

Of course you're right. This is the issue that Zuul was principally
created to solve. But I forgot we're not using a Gate pipeline here.

-Jim

[Antoine Musso]

Le 15/07/2021 à 17:54, Nasser Grainawi a écrit :
>>> * Create a new job that runs in the "post" pipeline (jobs run after each
>>>  change merges, and in the context of the newly-updated ref).
>>>
>>> * It builds the artifact in the same way as the check job (but omits
>>>  testing).
>
> Should it really omit testing? I know we typically have changes going
> through verification before they submit, but two changes could
> independently work and then when both are submitted tests would fail.
> Seems like it’d be safer to test in the post pipeline too.

Hello,

The whole promise of Zuul is that it "Stop Merging Broken Code": code
that has been merged has a guarantee that tests pass.

You rightfully points that two independent changes might conflict with
each others and lead to test suddenly failing after they have been
manually submitted. Even though independently they work fine.  The same
can happen when the target branch has moved forward and, even though
there is no merge conflict, some tests might fail once the change got
merged on top of the new commits. Zuul lets one catch those
incompatibilities.

Instead of submitting, humans approve the changes (ex: Code-Review +2)
but do not submit them. Zuul catches the approval requests, queue the
changes and tests them against the tip of the branch and as if all
changes ahead in the queue already have been merged.  Essentially it has
the ability to run tests against a speculative state.

Given changes A and B not having any file conflict and having tests
passing. Humans approve A then B which enter a queue.

When processing A, Zuul:

- updates its copy of the project repo and bring the target branch
up-to-date

- attempts to merge change A against the branch. If that fails: it
reports back

- creates a git reference for the merge of A into branch

- runs tests against that (updated branch + change A)

If that works, we know that A merged in the branch does indeed pass
tests and Zuul submit the change to Gerrit. The branch is known to have
tests passing, there is no need to run tests again.


When processing B while A is already in the queue, Zuul:

- updates its copy of the project repo and bring the target branch
up-to-date

- merges Change A, which has not yet been merged but is ahead in the queue

- merges Change B

- creates a git reference for the merge of A and B into the branch

- run tests against that (updated branch + change A + change B)

If that works, we know that B passed the tests if A gets merged
(speculative state). The change B is on hold until A got all tests
working and is merged.

If A and B are incompatible, B will fail since it is tested as if A got
in and thus B will not be merged.

If A fails for some reason, Zuul will remove A from the queue and report
the tests failure. It then cancels all jobs for B, move B ahead in the
queue then prepare a merge commit of B into the branch and retrigger
jobs. B is thus rested without A.  If everything pass, B get merged but
not A.

The concept is named "gating" and is definitely worth reading:
https://zuul-ci.org/docs/zuul/discussion/gating.html
<https://zuul-ci.org/docs/zuul/discussion/gating.html>

cheers,

--
Antoine "hashar" Musso
Release Engineering