Zuul for Gerrit

[James E. Blair]

Hi,

At the recent Gerrit User Summits in Gothenburg and Sunnyvale, we talked
about setting up Zuul as a CI system for the Gerrit project. This will
provide more scalable CI, easier collaboration on maintenance, and new
features for cross-project testing (which I personally think will be
important as we do more with plugins).

Since then we have made changes to Zuul to better support the checks
plugin, since stream-events is not available in the googlesource
environment. We have also been discussing how best to use the test
resources that Google is making available (thanks!).

I think we're ready to get started setting this up now. The rest of
this email is a proposed plan for doing so. I am happy to say that I
think that the set up and operation of this system can happen almost
entirely in public git repositories, where everyone will be welcome to
observe and help.

Functional System Components
----------------------------

Zuul consists of several mostly-scalable microservices which work
together to run the system. We will run these services in a dedicated
Kubernetes cluster in Google Cloud. The configuration of these services
will happen in a dedicated operations repo -- we'll probably start with
k8s YAML but we could move to helm charts or an operator as warranted.

One of the Zuul components is called "nodepool". It's job is to talk to
Google cloud and provide test nodes for jobs. It will create VMs as
needed for jobs to run on.

The jobs themselves are likely to produce artifacts and log files. Zuul
doesn't provide any internal storage for those, instead, we'll use
Google cloud storage for those.

A very small part of Zuul's configuration is in static config files.
They are mostly used to set up the connection to Gerrit and to indicate
what projects Zuul should operate on. Zuul uses that to bootstrap the
rest of its configuration from the contents of git repos directly, and
continually mutates its configuration as changes to those repos merge.

New Repositories
----------------

That's the really high-level overview. To implement that, I would like
to ask the maintainers to create the following repositories in
gerrit-review:

* zuul/ops
* zuul/config
* zuul/jobs

I expect these to contain:

* zuul/ops -- This will hold the configuration of the "control-plane"
services running in k8s. We will also place the "bootstrap"
configuration files for Zuul here, as they are likely to be
implemented as k8s configmaps or secrets.

(An eventual goal is to have Zuul deploy itself automatically as
changes are merged to this repository; we'll run 'kubectl apply' by
hand while we bootstrap this.)

* zuul/config -- This will contain the basic configuration of the Zuul
system. Zuul has two kinds of projects: config-projects and
untrusted-projects. This one will be a config-project, where we set
up the pipeline configuration for the system (inputs and outputs),
and configure base jobs that all other jobs inherit from.

* zuul-jobs -- This is an unstrusted-project, which means it has no
special access. We will use this to define jobs that can be used by
other projects in the system. Any repository can define jobs for use
by itself or any other, which can be convenient, but it makes sense to
have a central location for defining jobs that are widely used.

For example, imagine a job which builds a plugin. That could be
defined here, and all plugins in the system could use that job (or
inherit from it) to perform their builds and tests in the same way.

We may find that we want to use these repositories in the future, or we
want fewer or more of them. But I feel fairly certain that this group
of three are likely to be used in this way, and this is the best way to
get started.

Of course, I'm happy to answer any questions folks have, though I also
think that much of this will become clear as we set it up.

If this sounds good to the maintainers, would you please go ahead and
create those repos?

Thanks,

Jim

[Thomas Dräbing]

Hi James,

this sounds awesome. I am looking forward to seeing zuul in action. Are you planning to run it in parallel with the existing Jenkins-based CI for a while? How would you propose the switch between the CI-systems? With the checks plugin, we could for a while have duplicate checks, one worked on by Jenkins and one worked on by zuul. That would make it easy to test and also to compare metrics like job execution times.

Since I am involved in maintaining the current CI and also maintain our internal fork of the gerrit-ci-scripts project at SAP, I would be very interested in also looking closer into zuul. I will take some time to read through your documentation. Are your presentations at the User Summit available online?

I put some more questions/remarks below:

Are these jobs then run outside of Kubernetes on VMs? Or do you mean that just new nodes running Kubernetes worker pods will be deployed, if required?

 

The jobs themselves are likely to produce artifacts and log files.  Zuul
doesn't provide any internal storage for those, instead, we'll use
Google cloud storage for those.

Could we also index the logs instead of only saving the plain text? Depending on how the logs look like, this would make it easier to find relevant information or also create some statistics.

 

A very small part of Zuul's configuration is in static config files.
They are mostly used to set up the connection to Gerrit and to indicate
what projects Zuul should operate on.  Zuul uses that to bootstrap the
rest of its configuration from the contents of git repos directly, and
continually mutates its configuration as changes to those repos merge.

New Repositories
----------------

That's the really high-level overview.  To implement that, I would like
to ask the maintainers to create the following repositories in
gerrit-review:

* zuul/ops
* zuul/config
* zuul/jobs

I expect these to contain:

* zuul/ops -- This will hold the configuration of the "control-plane"
  services running in k8s.  We will also place the "bootstrap"
  configuration files for Zuul here, as they are likely to be
  implemented as k8s configmaps or secrets.

Will this then also contain the yaml-files describing the deployment etc. in addition to the configmaps and secrets? How do you plan to keep the secrets safe?

 

  (An eventual goal is to have Zuul deploy itself automatically as
  changes are merged to this repository; we'll run 'kubectl apply' by
  hand while we bootstrap this.) 

Are you planning to also do something like blue/green-deployments with some staging installation? I think Kubernetes should help a lot in doing this. My experience from our current CI setup tells me that this would make updates to the CI a lot easier and less stressful.

* zuul/config -- This will contain the basic configuration of the Zuul
  system.  Zuul has two kinds of projects: config-projects and
  untrusted-projects.  This one will be a config-project, where we set
  up the pipeline configuration for the system (inputs and outputs),
  and configure base jobs that all other jobs inherit from.

* zuul-jobs -- This is an unstrusted-project, which means it has no
  special access.  We will use this to define jobs that can be used by
  other projects in the system.  Any repository can define jobs for use
  by itself or any other, which can be convenient, but it makes sense to
  have a central location for defining jobs that are widely used.

  For example, imagine a job which builds a plugin.  That could be
  defined here, and all plugins in the system could use that job (or

  inherit from it) to perform their builds and tests in the same way.

 

For my understanding, are the jobs run by the CI completely managed in these two repositories or can the overall pipeline also be managed with the source code of the project, reusing the templates/building blocks provided in these repositories, e.g. like the Jenkinsfile for Jenkins?

 

We may find that we want to use these repositories in the future, or we
want fewer or more of them.  But I feel fairly certain that this group
of three are likely to be used in this way, and this is the best way to
get started.

Of course, I'm happy to answer any questions folks have, though I also
think that much of this will become clear as we set it up.

If this sounds good to the maintainers, would you please go ahead and
create those repos?

Thanks,

Jim

Thanks for explaining the overall setup and putting so much effort into it. Again, I am looking forward to it and will be glad to also give a helping hand, if I can.

Best,

Thomas

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/87d0clekpg.fsf%40meyer.lemoncheese.net.

[James E. Blair]

Thomas Dräbing <thomas....@gmail.com> writes:

> Hi James,
>
> this sounds awesome. I am looking forward to seeing zuul in action. Are you
> planning to run it in parallel with the existing Jenkins-based CI for a
> while? How would you propose the switch between the CI-systems? With the
> checks plugin, we could for a while have duplicate checks, one worked on by
> Jenkins and one worked on by zuul. That would make it easy to test and also
> to compare metrics like job execution times.

Yes, I believe there's no immediate need to retire the current system,
so we can run them both together as long as we want.

> Since I am involved in maintaining the current CI and also maintain our
> internal fork of the gerrit-ci-scripts project at SAP, I would be very
> interested in also looking closer into zuul. I will take some time to read
> through your documentation. Are your presentations at the User Summit
> available online?

Great! We have a lot of documentation, but we know that it is poorly
organized for introducing new users to how things work. It's very
strong in reference material though.

Nevertheless, the "Zuul Concepts" section of
https://zuul-ci.org/docs/zuul/ is still probably the best place to
start.

Monty Taylor gave a presentation at each of the GUS events, but I don't
see a video posted. We'll see if we can correct that and reply with an
update.

The jobs will run on VMs. Zuul and Nodepool do support running jobs on
k8s, and I would like for us to run Gerrit jobs on k8s as well.
However, our expertise and the bulk of our library of existing job
content is on VMs, and given that several aspects of this are already
novel (checks plugin; no stream-events; google cloud) I wanted to reduce
the risk of encountering problems running workloads in k8s. As soon as
we are comfortable with basic functionality, I think we can expand to
workloads-in-k8s. But also, if we find that some tasks are better
suited to VMs (building images, simulating entire k8s deployments, etc),
we should have both available to us and we can choose the right one for
the job.

>> The jobs themselves are likely to produce artifacts and log files. Zuul
>> doesn't provide any internal storage for those, instead, we'll use
>> Google cloud storage for those.
>>
> Could we also index the logs instead of only saving the plain text?
> Depending on how the logs look like, this would make it easier to find
> relevant information or also create some statistics.

Assuming we have resources, yes. In OpenDev (the Gerrit+Zuul
environment used for developing OpenStack, Zuul, and other projects) we
load many of the log files into an ELK stack for that purpose. Running
it is a bit of work, though since we set that up, the Elasticsearch
operator for k8s has been invented, so maybe that'll be easier. Also,
it takes a large amount of disk. Or folks may prefer other options.
Regardless, I think we can get basic hosting of log files up first, and
add indexing later. None of this needs to be "built-in" to Zuul; both
log storage and indexing are simply implemented as post-run Ansible
playbooks, so it's easy to support any similar system.

>> A very small part of Zuul's configuration is in static config files.
>> They are mostly used to set up the connection to Gerrit and to indicate
>> what projects Zuul should operate on. Zuul uses that to bootstrap the
>> rest of its configuration from the contents of git repos directly, and
>> continually mutates its configuration as changes to those repos merge.
>>
>> New Repositories
>> ----------------
>>
>> That's the really high-level overview. To implement that, I would like
>> to ask the maintainers to create the following repositories in
>> gerrit-review:
>>
>> * zuul/ops
>> * zuul/config
>> * zuul/jobs
>>
>> I expect these to contain:
>>
>> * zuul/ops -- This will hold the configuration of the "control-plane"
>> services running in k8s. We will also place the "bootstrap"
>> configuration files for Zuul here, as they are likely to be
>> implemented as k8s configmaps or secrets.
>>
> Will this then also contain the yaml-files describing the deployment etc.
> in addition to the configmaps and secrets? How do you plan to keep the
> secrets safe?

I believe they can all be placed into this repository. Because the
authors of Zuul are also interested in the idea of collaborative
management of systems supporting open source projects, we added a
feature to Zuul to allow storing private data directly in a public git
repo.

A Zuul secret[1] uses asymmetric (public/private key) encryption to
store credential data in the git repo. It is encrypted using a keypair
unique to a repository in Zuul. By the end of this process, we should
be able to have a Zuul job check out the zuul/ops repo, write out Zuul
secrets with any credential information necessary (such as the
credentials to access Gerrit), and then run kubectl apply or helm or
similar to update the running system.

If you're curious what that looks like, here's the username and password
Zuul uses to uploads its container images to Docker Hub[2].

To bootstrap, I imagine that we will manually add some secrets to the
k8s we deploy, until we can self-deploy.

[1] https://zuul-ci.org/docs/zuul/user/config.html#secret
[2] https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L110-L124

>> (An eventual goal is to have Zuul deploy itself automatically as
>> changes are merged to this repository; we'll run 'kubectl apply' by
>> hand while we bootstrap this.)
>
>
> Are you planning to also do something like blue/green-deployments with some
> staging installation? I think Kubernetes should help a lot in doing this.
> My experience from our current CI setup tells me that this would make
> updates to the CI a lot easier and less stressful.

We're big believers in project gating, and every commit to Zuul is so
thoroughly tested that we think master is always production-ready, and
that's what the :latest container images represent. Since
gerrit-review.googlesource.com is similarly deploying frequently from
master, I'm actually more worried about the interaction of two
"bleeding-edge" pieces of software there. We're really close to being
able to have meaningful cross-project testing between the two though, so
we could see if a patch to the checks plugin would break Zuul, or the
other way around. But in short, I think both pieces of software are
relatively stable and well tested enough to run from master.

As far as the jobs themselves, we wanted for years to more easily test
changes to jobs, so with Zuul v3, we were able to make most jobs
self-testing. Zuul's configuration is in-repo, and if you modify
that configuration in a change, Zuul will test that change with the
modified configuration. For an example of what that looks like, here's
a change[3] which added some simple golang jobs to the Zuul standard job
library, along with tests of those jobs:

[3] https://review.opendev.org/#/c/691114/10/zuul.d/go-jobs.yaml

On the change screen[4], you can see it ran the
"zuul-jobs-test-golang-go" job, which did not exist prior to that change.

[4] https://review.opendev.org/#/c/691114/10

>> * zuul/config -- This will contain the basic configuration of the Zuul
>> system. Zuul has two kinds of projects: config-projects and
>> untrusted-projects. This one will be a config-project, where we set
>> up the pipeline configuration for the system (inputs and outputs),
>> and configure base jobs that all other jobs inherit from.
>>
>> * zuul-jobs -- This is an unstrusted-project, which means it has no
>> special access. We will use this to define jobs that can be used by
>> other projects in the system. Any repository can define jobs for use
>> by itself or any other, which can be convenient, but it makes sense to
>> have a central location for defining jobs that are widely used.
>>
>> For example, imagine a job which builds a plugin. That could be
>> defined here, and all plugins in the system could use that job (or
>> inherit from it) to perform their builds and tests in the same way.
>
>
> For my understanding, are the jobs run by the CI completely managed in
> these two repositories or can the overall pipeline also be managed with the
> source code of the project, reusing the templates/building blocks provided
> in these repositories, e.g. like the Jenkinsfile for Jenkins?

The configuration can be in any repo. It is universal within a Zuul
tenant. Zuul is a multi-tenant system, but for Gerrit, I expect us to
have just a single tenant. I think what you describe is best -- use
zuul/jobs (and sorry, that was a typo above; it should be "zuul/jobs"
not "zuul-jobs") to hold job definitions, templates, reusable building
blocks, etc. Then in individual projects have smaller in-repo
configuration files which direct how those should be applied to that
particular repo.

But it's worth noting that any repo can define a job that any other repo
can use, so any organizational structure is possible.

>> We may find that we want to use these repositories in the future, or we
>> want fewer or more of them. But I feel fairly certain that this group
>> of three are likely to be used in this way, and this is the best way to
>> get started.
>>
>> Of course, I'm happy to answer any questions folks have, though I also
>> think that much of this will become clear as we set it up.
>>
>> If this sounds good to the maintainers, would you please go ahead and
>> create those repos?
>>
>> Thanks,
>>
>> Jim
>>
> Thanks for explaining the overall setup and putting so much effort into
> it. Again, I am looking forward to it and will be glad to also give a
> helping hand, if I can.

Thank you for the interest and questions; I'm looking forward to it!

-Jim

[Patrick Hiesel]

Hi James,

I created the repos for you: https://gerrit-review.googlesource.com/admin/repos/q/filter:zuul

The owner group is zuul-maintainers which your are a member of (next to gerritcodereview-maintainers). As owner, you can change the permissions if needed.

I am really looking forward to having Zuul in Gerrit :-)

Han-Wen said that he can try to help with the GCP setup. He will be back on Jan 7.

Thanks,

Patrick

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/87sglhbdpa.fsf%40meyer.lemoncheese.net.

[Luca Milanesio]

On 20 Dec 2019, at 11:59, 'Patrick Hiesel' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Hi James,

I created the repos for you: https://gerrit-review.googlesource.com/admin/repos/q/filter:zuul

The owner group is zuul-maintainers which your are a member of (next to gerritcodereview-maintainers). As owner, you can change the permissions if needed.

I am really looking forward to having Zuul in Gerrit :-)

+1 

I am publishing Monty’s talk to the GerritForge TV YouTube channel, so that people can have an overview of what is Zuul and how it integrates with Gerrit.

Han-Wen said that he can try to help with the GCP setup. He will be back on Jan 7.

Thanks,

Patrick

On Wed, Dec 18, 2019 at 9:07 PM James E. Blair <cor...@inaugust.com> wrote:

Thomas Dräbing <thomas....@gmail.com> writes:

> Hi James,
>
> this sounds awesome. I am looking forward to seeing zuul in action. Are you
> planning to run it in parallel with the existing Jenkins-based CI for a
> while? How would you propose the switch between the CI-systems? With the
> checks plugin, we could for a while have duplicate checks, one worked on by
> Jenkins and one worked on by zuul. That would make it easy to test and also
> to compare metrics like job execution times.

Yes, I believe there's no immediate need to retire the current system,
so we can run them both together as long as we want.

I believe it will be done in three stages:

1) Gerrit-CI (Jenkins) doing validation/voting, Zuul CI doing only validation but just informative voting

2) Gerrit-CI (Jenkins) + Zuul CI voting at the same level, so that we can identify any inconsistency

3) Zuul CI voting and being the unique source of validation

We are approaching stage 1) and looking forward to get to stage 3) very soon.

> Since I am involved in maintaining the current CI and also maintain our
> internal fork of the gerrit-ci-scripts project at SAP, I would be very
> interested in also looking closer into zuul. I will take some time to read
> through your documentation. Are your presentations at the User Summit
> available online?

Great!  We have a lot of documentation, but we know that it is poorly
organized for introducing new users to how things work.  It's very
strong in reference material though.

Nevertheless, the "Zuul Concepts" section of
https://zuul-ci.org/docs/zuul/ is still probably the best place to
start.

Monty Taylor gave a presentation at each of the GUS events, but I don't
see a video posted.  We'll see if we can correct that and reply with an
update.

Working on it … it tried last night but failed. Trying again now … it was a long one and I’m stress-testing YouTube with Monty’s talk :-)

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAM7sg%3D0uOq9U%2Bm3L2dbtypRmzsw7Dr8KbpGKebnsByOwi8TZTA%40mail.gmail.com.

[Luca Milanesio]

On 20 Dec 2019, at 12:58, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 20 Dec 2019, at 11:59, 'Patrick Hiesel' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Hi James,

I created the repos for you: https://gerrit-review.googlesource.com/admin/repos/q/filter:zuul

The owner group is zuul-maintainers which your are a member of (next to gerritcodereview-maintainers). As owner, you can change the permissions if needed.

I am really looking forward to having Zuul in Gerrit :-)

+1 

I am publishing Monty’s talk to the GerritForge TV YouTube channel, so that people can have an overview of what is Zuul and how it integrates with Gerrit.

Done:

https://youtu.be/QiJzK4lC-2k

@Monty: we made it, and thanks again for presenting it at the Summit.

Luca.

[Monty Taylor]

> On Dec 20, 2019, at 8:35 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>
>
>> On 20 Dec 2019, at 12:58, Luca Milanesio <luca.mi...@gmail.com> wrote:
>>
>>
>>
>>> On 20 Dec 2019, at 11:59, 'Patrick Hiesel' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>>>
>>> Hi James,
>>>
>>> I created the repos for you: https://gerrit-review.googlesource.com/admin/repos/q/filter:zuul
>>>
>>> The owner group is zuul-maintainers which your are a member of (next to gerritcodereview-maintainers). As owner, you can change the permissions if needed.
>>>
>>> I am really looking forward to having Zuul in Gerrit :-)
>>
>> +1
>>
>> I am publishing Monty’s talk to the GerritForge TV YouTube channel, so that people can have an overview of what is Zuul and how it integrates with Gerrit.
>
> Done:
> https://youtu.be/QiJzK4lC-2k
>
> @Monty: we made it, and thanks again for presenting it at the Summit.
>
> Luca.

Sweet - thanks! Glad I could help stress-test YouTube. :)

[James E. Blair]

Hi,

I'd like to provide a quick update, and make one more request of the
Maintainers.

I've been working with Han-Wen (thanks!) on structuring the service
accounts used for the Zuul installation. The Google authentication for
Gerrit along with the desire to use short-lived access tokens rather
than exporting service account keys provided some challenges. But I'm
happy to say that it looks like we have addressed those (and being able
to minimize the distribution of credentials is a big security win).

I'll write the structure and process up before this is all finished so
everyone can see how it's set up.

I'm still finishing up the process of bootstrapping the system, so it's
not quite available for use yet, but I think we're very close.


The request I have for the Maintainers is regarding DNS:

Can we add a DNS A record for Zuul's web dashboard? The usual pattern
would be "zuul.", so perhaps zuul.gerritcodereview.com? I've reserved a
global IPv4 address in Google Cloud, so the entry should look like

zuul.gerritcodereview.com. IN A 34.107.128.34

Thanks,

Jim

(I would like to add IPv6 as well, and I tested that, but I think the
bug at https://github.com/kubernetes/ingress-gce/issues/87 needs to be
fixed first in order for cert-manager to provide the correct certs to
both https frontends.)

[Han-Wen Nienhuys]

On Wed, Jan 29, 2020 at 10:59 PM James E. Blair <cor...@inaugust.com> wrote:
> I'm still finishing up the process of bootstrapping the system, so it's
> not quite available for use yet, but I think we're very close.
>
>
> The request I have for the Maintainers is regarding DNS:
>
> Can we add a DNS A record for Zuul's web dashboard? The usual pattern
> would be "zuul.", so perhaps zuul.gerritcodereview.com? I've reserved a
> global IPv4 address in Google Cloud, so the entry should look like

Thanks for all your hard work James!

> zuul.gerritcodereview.com. IN A 34.107.128.34

Unfortunately, I still need to do some work to get the project
released from the clutches of the firewall. (I think you are still not
able to make HTTP requests from external networks, correct?)

--
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[James E. Blair]

Han-Wen Nienhuys <han...@google.com> writes:

> On Wed, Jan 29, 2020 at 10:59 PM James E. Blair <cor...@inaugust.com> wrote:
>> I'm still finishing up the process of bootstrapping the system, so it's
>> not quite available for use yet, but I think we're very close.
>>
>>
>> The request I have for the Maintainers is regarding DNS:
>>
>> Can we add a DNS A record for Zuul's web dashboard? The usual pattern
>> would be "zuul.", so perhaps zuul.gerritcodereview.com? I've reserved a
>> global IPv4 address in Google Cloud, so the entry should look like
>
> Thanks for all your hard work James!
>
>> zuul.gerritcodereview.com. IN A 34.107.128.34
>
> Unfortunately, I still need to do some work to get the project
> released from the clutches of the firewall. (I think you are still not
> able to make HTTP requests from external networks, correct?)

HTTP(S) does seem to be working (ssh is blocked; but that's not necessary).

That's via a load-balancer though, so perhaps that's excluded from the
firewall policy?

-Jim

[Han-Wen Nienhuys]

Curious; maybe the change was effected without me noticing. It's odd
though. Let me look into it.

>
> -Jim

[James E. Blair]

Hi,

I think we're very close to declaring this production ready. I think
the most significant remaining task is getting the name set up in DNS,
and I believe that is in progress.

In the mean time, with Han-Wen's help, I believe we've worked out all of
the challenges in running in this particular environment, and the
installation is now self-deploying!

Here is a job which updates the k8s resource definitions for the system
as well as Zuul's own configuration:

https://gerrit-zuul.inaugust.com/t/gerrit/build/867c84e630244ec4ab06b6126764c9aa/console

(The domain name in that URL is obviously temporary.)

There's more to be done in the operations department, but none of these
need to block use of the system:

* We made several changes to Zuul to support the Google environment; a
few of these have yet to be merged and therefore Gerrit's Zuul is
running from locally-built container images. As the changes merge, I
will update the config to use the upstream images. Basically, we're
running a local change anywhere "jeblair" shows up in a container
namespace. I believe only two changes are outstanding at this point.

* We need a better database setup -- I just ran the click-to-deploy
mariadb image because it was simple, but we should really move to one
of the HA operators. This will be easy to swap out later.

* I need to write some documentation for the ops repo.

I have been self-approving changes during the bootstrapping process, but
I'd like to stop doing that and utilize real code review starting now.
I'd appreciate anyone interested to start watching changes on these
repos:

* zuul/ops -- System administration for Zuul. The files in k8s/ are k8s
resource definitions, and the files in nodepool/ and zuul/ are the
software configuration of the nodepool and zuul applications.
Whenever a change is merged to this repo, Zuul will run
playbooks/deploy.yaml to update the running config.

https://gerrit.googlesource.com/zuul/ops/+/refs/heads/master

* zuul/config -- This is where the pipelines and base jobs for the
system are defined. This should be mostly settled at this point.
Only playbooks and jobs which require special permissions (like
uploading artifacts to Google cloud storage) need to be located here.

https://gerrit.googlesource.com/zuul/config/+/refs/heads/master/

* zuul/jobs -- This repo is currently empty, but is where most of the
next phase of activity will happen. This is where we can define jobs
that any project in the system can use.

https://gerrit.googlesource.com/zuul/jobs/+/refs/heads/master/

Finally, I think we can begin work on creating real jobs now. This is
definitely not something I'm suited to working on alone. What job
should we tackle first? A basic built+unit test job of Gerrit itself?
A job to build Gerrit plugins? Let's decide what needs to happen and
I'll work with you to make it so.

-Jim

[Thomas Dräbing]

Hi Jim,

thanks for the update :-)!

I started to watch all projects and are happy to help out with reviews as good as I can without knowing a lot about zuul (yet).

 

Finally, I think we can begin work on creating real jobs now.  This is
definitely not something I'm suited to working on alone.  What job
should we tackle first?  A basic built+unit test job of Gerrit itself?
A job to build Gerrit plugins?  Let's decide what needs to happen and
I'll work with you to make it so.

I think plugins might be a lower hanging fruit, since the builds are less complex (at least so far). In the Jenkins-based CI, we are using so far, the plugin-jobs are just running, when the branch is updated and not used to verify changes. The current build script for plugins that are built against the same Gerrit branch can be found in [1] and the job-configuration is defined in [2]. In our fork at SAP, we use the same job in combination with the Gerrit-trigger-plugin to also verify changes to the plugins. Adding verification jobs for plugins using zuul (and the checks plugin?) would also already provide additional value and since most plugins are built in the same way, would already cover a lot of space quickly. Also most plugins do not have a lot of development going on at a time, so in case there are some issues in the beginning, it would be of limited impact. On the other hand however, it would also mean less (real) changes coming in to test the jobs. WDYT?

[1] https://gerrit.googlesource.com/gerrit-ci-scripts/+/refs/heads/master/jenkins/gerrit-bazel-build-plugin.sh

[2] https://gerrit.googlesource.com/gerrit-ci-scripts/+/refs/heads/master/jenkins/gerrit-bazel-plugin-template.yaml#54

Best,

Thomas

 

-Jim

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/87v9ocwvme.fsf%40meyer.lemoncheese.net.

[Luca Milanesio]

Same here !

I am really keen to learn to use Zuul and help out with the CI/CD.

P.S. I would like to use Zuul and Checks also with GerritHub.io as well :-)

 

Finally, I think we can begin work on creating real jobs now.  This is
definitely not something I'm suited to working on alone.  What job
should we tackle first?  A basic built+unit test job of Gerrit itself?
A job to build Gerrit plugins?  Let's decide what needs to happen and
I'll work with you to make it so.

I think plugins might be a lower hanging fruit, since the builds are less complex (at least so far). In the Jenkins-based CI, we are using so far, the plugin-jobs are just running, when the branch is updated and not used to verify changes. The current build script for plugins that are built against the same Gerrit branch can be found in [1] and the job-configuration is defined in [2]. In our fork at SAP, we use the same job in combination with the Gerrit-trigger-plugin to also verify changes to the plugins. Adding verification jobs for plugins using zuul (and the checks plugin?) would also already provide additional value and since most plugins are built in the same way, would already cover a lot of space quickly. Also most plugins do not have a lot of development going on at a time, so in case there are some issues in the beginning, it would be of limited impact. On the other hand however, it would also mean less (real) changes coming in to test the jobs. WDYT?

[1] https://gerrit.googlesource.com/gerrit-ci-scripts/+/refs/heads/master/jenkins/gerrit-bazel-build-plugin.sh

[2] https://gerrit.googlesource.com/gerrit-ci-scripts/+/refs/heads/master/jenkins/gerrit-bazel-plugin-template.yaml#54

Yes, that’s a good idea. Also, because we don’t have Change validations at the moment in Jenkins, haven them in Zuul would be an immediate benefit and won’t clash with the existing validation workflow currently in Jenkins.

@James thanks again for working on that, I consider the synergies with Zuul and OpenStack the most successful outcome of the User Summits last year, in terms of community engagement and collaboration.

Luca.

[James E. Blair]

Luca Milanesio <luca.mi...@gmail.com> writes:

>> On 12 Feb 2020, at 10:18, Thomas Dräbing <thomas....@gmail.com> wrote:
>> Hi Jim,
>> thanks for the update :-)!
>> I started to watch all projects and are happy to help out with
>> reviews as good as I can without knowing a lot about zuul (yet).
>
> Same here !
> I am really keen to learn to use Zuul and help out with the CI/CD.
>

> P.S. I would like to use Zuul and Checks also with GerritHub.io <http://gerrithub.io/> as well :-)

That looks like a great place to start, thanks! I think I understand
most of what needs to happen there. I'll put together a series of
changes that work toward this and come back here with a summary (and
perhaps further questions). I'm hoping this will be a useful way for
folks to become familiar with this as we go. I should be able to show
most of this working (or not) without even merging any changes.

One of the exceptions to that is this change, which I have gone ahead
and merged:

https://gerrit-review.googlesource.com/c/zuul/ops/+/254715

And you can see Zuul updated the deployed config here:

https://gerrit-zuul.inaugust.com/t/gerrit/build/7dc1d142bdd94b3facb2beeec4bacf80/console

The change adds a new node type ("label") to the system, therefore it's
something that needs to actually be merged before it takes effect. Now
that it has been, the next changes I write can use that label to run the
new jobs.

-Jim

[Han-Wen Nienhuys]

On Wed, Feb 12, 2020 at 7:59 PM James E. Blair <cor...@inaugust.com> wrote:

  https://gerrit-zuul.inaugust.com/t/gerrit/build/7dc1d142bdd94b3facb2beeec4bacf80/console

The change adds a new node type ("label") to the system, therefore it's
something that needs to actually be merged before it takes effect.  Now
that it has been, the next changes I write can use that label to run the
new jobs.

You can now see this instance running at 

  https://ci.gerritcodereview.com/

[Matthias Sohn]

On Fri, Feb 14, 2020 at 10:47 AM 'Han-Wen Nienhuys' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

On Wed, Feb 12, 2020 at 7:59 PM James E. Blair <cor...@inaugust.com> wrote:

  https://gerrit-zuul.inaugust.com/t/gerrit/build/7dc1d142bdd94b3facb2beeec4bacf80/console

The change adds a new node type ("label") to the system, therefore it's
something that needs to actually be merged before it takes effect.  Now
that it has been, the next changes I write can use that label to run the
new jobs.

You can now see this instance running at 

  https://ci.gerritcodereview.com/

nice :-)

thanks James for all this work

what's the purpose of the "noop" Jobs ?

https://ci.gerritcodereview.com/t/gerrit/builds

-Matthias 

[Luca Milanesio]

On 14 Feb 2020, at 10:07, Matthias Sohn <matthi...@gmail.com> wrote:

On Fri, Feb 14, 2020 at 10:47 AM 'Han-Wen Nienhuys' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

On Wed, Feb 12, 2020 at 7:59 PM James E. Blair <cor...@inaugust.com> wrote:

  https://gerrit-zuul.inaugust.com/t/gerrit/build/7dc1d142bdd94b3facb2beeec4bacf80/console

The change adds a new node type ("label") to the system, therefore it's
something that needs to actually be merged before it takes effect.  Now
that it has been, the next changes I write can use that label to run the
new jobs.

You can now see this instance running at 

  https://ci.gerritcodereview.com/

nice :-)

thanks James for all this work

+2 this is amazing.

Thanks, James.

Luca.

[James E. Blair]

Matthias Sohn <matthi...@gmail.com> writes:

> what's the purpose of the "noop" Jobs ?
> https://ci.gerritcodereview.com/t/gerrit/builds

Short answer: it's sometimes helpful to have a "noop" (always succeeds
immediately) job when bootstrapping a new repo. We can probably drop
them now.

Long answer:

When modifying Zuul's in-repo configuration, if there is an error, Zuul
will let you know. However, there's one exception to this: if a project
has no jobs at all configured in a pipeline, Zuul will avoid reporting
errors to that project. The safest way to start using a new project
with Zuul is to merge a change which runs a "noop" job, then add real
jobs in subsequent changes and remove the "noop" job. I just didn't do
that last part yet.

I'm still working one some initial changes to build and test a Gerrit
plugin. When I finish those up, I'll reply here with a summary.

-Jim

[James E. Blair]

cor...@inaugust.com (James E. Blair) writes:

> That looks like a great place to start, thanks! I think I understand
> most of what needs to happen there. I'll put together a series of
> changes that work toward this and come back here with a summary (and
> perhaps further questions). I'm hoping this will be a useful way for
> folks to become familiar with this as we go. I should be able to show
> most of this working (or not) without even merging any changes.
>
> One of the exceptions to that is this change, which I have gone ahead
> and merged:
>
> https://gerrit-review.googlesource.com/c/zuul/ops/+/254715
>
> And you can see Zuul updated the deployed config here:
>
> https://gerrit-zuul.inaugust.com/t/gerrit/build/7dc1d142bdd94b3facb2beeec4bacf80/console
>
> The change adds a new node type ("label") to the system, therefore it's
> something that needs to actually be merged before it takes effect. Now
> that it has been, the next changes I write can use that label to run the
> new jobs.

Hi, I think I have a basic job for building plugins working!

First, let me jump to the end and show you what it will look like to add
this job to a plugin repo:

https://gerrit-review.googlesource.com/c/plugins/checks/+/255215

There's a lot of stuff I'm going to talk about below, and some of it is
a little complicated, but it's all in service of making it easy for a
plugin to add this job, and to build other jobs like it.

I'd also like to emphasize that the entire process for building and
adding this job to a repo is contained in the changes below (all but one
of which haven't even been merged) -- the configuration is entirely
dynamic and git-driven. Okay, let's dive in!

0) Add the required projects for Gerrit

https://gerrit-review.googlesource.com/c/zuul/ops/+/255012

This is the one change I had to merge in order for the rest of the
series to work. Zuul needs a list of all the projects it works with,
and that can't be dynamically updated; it's a configuration change that
has to merge and be deployed before it takes effect (it could be
dangerous to let someone add a project to Zuul without the
administrator's permission). This change adds all of the projects
needed to build Gerrit, plus one extra optional plugin.

Zuul doesn't automatically do anything with these projects, but since it
knows about them, it can check out their git repos, and they have the
option to add jobs if they want.

1) Add ensure-bazelisk role

https://gerrit-review.googlesource.com/c/zuul/jobs/+/254772

This might seem like a lot of boilerplate for what's essentially a curl
invocation. But I wrote it this way for two reasons. First, because
I don't intend to leave this role in this repo in the long term.
Installing bazelisk is in no way tied to Gerrit -- it's something that
any Zuul user might benefit from. So I'm going to propose this exact
change to the zuul-jobs project:

https://zuul-ci.org/docs/zuul-jobs/

That's a standard library of Zuul jobs and Ansible roles that can be
used in any Zuul in the world. Because Zuul is git-driven, any Zuul
system can have immediate access to those just by adding the zuul-jobs
git repo to the system. We're using it in our instance too; more on
that later.

Even if I didn't have a global audience in mind for this role, I think
it's still a good practice to factor out tasks like this into roles, and
document their parameters. In the same way that a good programming
language lets you break down problems into constituent parts and then
build larger structures out of them, Ansible and Zuul do the same for
CI/CD systems. Roles are essentially the unit of re-usability in
Ansible, so that's why this unit of work gets its own role.

The role gets its own "unit test" job too. Any future changes to the
role will at least run a simple job that executes the role, so we can
catch errors without having to run a heavyweight job. And it's handy
right now since we haven't built up any more complex jobs that use it
yet. With this, we can verify that the first step of our future plugin
build job works before we proceed.

The way all of these connect together is:

* The contents of zuul.d/project.yaml tell Zuul what jobs to run on
changes to this project. It says to run "test-ensure-bazelisk".
* The job "test-ensure-bazelisk" in zuul.d/test.jobs.yaml is our "unit
test" job.
* It calls the playbook at playbooks/test-ensure-bazelisk.yaml.
* The playbook invokes the role "ensure-bazelisk".
* The contents of roles/ensure-bazelisk comprise the role. The
entrypoint for the role is at roles/ensure-bazelisk/tasks/main.yaml.
That's where the actual work starts.

And yes, we absolutely could have just implemented this as a one-liner
shell script. Nothing in Zuul prohibits that kind of simplicity. But
this boilerplate will appear more useful later as we start to build up.

2) Add prepare-gerrit-repos role and gerrit-base job

https://gerrit-review.googlesource.com/c/zuul/jobs/+/254994

The bulk of this change is about setting up the gerrit repos, since
there are submodules involved. There's about as much documentation in
this change as there is code, and for the details, I'll refer you to the
README. The comments on the change are also enlightening. We will
probably change this in the future, but the current patch is functional,
and in most cases, functionally equivalent to other options under
discussion. Because it's modular, we can easily change this in the
future without disrupting the jobs that we build on it.

This change adds something new: our first "real" job. The previous job
we added was narrow in scope: unit testing an Ansible role. Here we've
added a job in zuul.d/jobs.yaml called "gerrit-base". As its name
implies, it will form the base of every future job in the system which
builds Gerrit (if we add linters or documentation build jobs, they might
not use gerrit-base).

Zuul achieves modularity not only by relying on Ansible roles, but also
its own internal inheritance system for jobs. The jobs we build later
will declare that gerrit-base is their parent, and they will inherit
everything you see here. Notably they will inherit the list of projects
that Zuul needs to check out in order to build Gerrit (Gerrit and its
required submodules). They will also inherit the pre-run playbook at
playbooks/gerrit-base/pre.yaml. That playbook is run before the main
part of the job. By separating them like this, we can put all of the
"setup" logic in the base job, and reserve the main playbook for the
interesting work.

In this same change, you can see that we add a job named
"test-gerrit-base" in test-jobs.yaml that inherits from "gerrit-base"
(I'm putting jobs that test roles in this repo in "test-jobs.yaml" and
real jobs in "jobs.yaml"). The line "parent: gerrit-base" is the magic
bit there. Note that it also adds a "run" playbook. When this job
runs, it will run the "pre-run" playbook from gerrit-base, and then the
"run" playbook here. The run playbook simply performs some independent
verification that the repos were set up correctly. This is our "unit
test" for this role.

Finally, take a look at patchset one of this change. I wrote this
before I added the projects in step 0, which means most of this change
was invalid Zuul configuration. In these cases, Zuul leaves robot
comments on the change pointing directly at the problem:

https://gerrit-review.googlesource.com/c/zuul/jobs/+/254994/1/zuul.d/jobs.yaml#22

After merging the change in step 0, Zuul accepted the config.

3) Add ensure-java role

https://gerrit-review.googlesource.com/c/zuul/jobs/+/255212

We need to install java. You probably get the pattern at this point.
This role can probably go to zuul-jobs too.

4) Add install-build-essential

https://gerrit-review.googlesource.com/c/zuul/jobs/+/255213

We need to install autotools, and zip. And also node. Let's take a
look at the gerrit-base pre-run playbook:

https://gerrit-review.googlesource.com/c/zuul/jobs/+/255213/7/playbooks/gerrit-base/pre.yaml

You'll notice that's starting to look like a concise list of things that
the job needs to do. It makes it pretty easy to see the overall
structure of that job. Most of the time we won't have to look at that,
we can just inherit from it and not worry about it. But if we need to
do some of those things, but in a different order, or omit one of them,
then because we've modularized them in this way, it's easy to do. We
could create a new job that does not inherit from gerrit-base and give
it a playbook with just the roles that we need for that job. That's our
escape valve if inheritance isn't the right solution for a problem. We
retain a huge amount of flexibility with minimal code duplication.

If you've spent any time with jenkins-job-builder (which I know some
folks here have -- the gerrit-ci scripts use it extensively) you may
think "hey that looks like a list of builders". The primary authors of
Zuul are also the original authors of JJB (hi!). That's one of the
things we found people really liked in JJB, so we've kept the habit of
building our Ansible playbooks in a similar way. You might also notice
that the "project" and even to some extent, "job" stanzas look like JJB.
That's no accident. One thing we did not borrow from JJB: template
substitution. At larger scales, we found the substitution rules to be
difficult to follow and maintain. Instead, we added inheritance. It
doesn't do everything that the template expansion did (on purpose!).
And as any honest object-oriented programmer will tell you, it can be
complex too. But how complex we make it is our choice. In my
experience, just a few levels deep is enough to solve most problems and
humans can still hold the inheritance hierarchy in their heads. If you
ever get lost, we have a browser:
https://ci.gerritcodereview.com/t/gerrit/jobs

Finally, note that we're also adding the install-nodejs role to the
playbook. We didn't define that. That's from the zuul-jobs repo:

https://zuul-ci.org/docs/zuul-jobs/js-roles.html#role-install-nodejs

If you were wondering what all the weird sphinx syntax in the role
README was about -- there's a sphinx extension called "zuul-sphinx"
which is used to render the documentation you see in the link above.

5) Add plugin build roles

https://gerrit-review.googlesource.com/c/zuul/jobs/+/255092

Two new roles that follow the pattern. And a new "real" job:
"gerrit-plugin-build". As its name implies, it is a job to build any
Gerrit plugin.

You should be able to add this job to any plugin repo and it will
automatically build that plugin. That's our goal in sight. :)

We also add an interesting new "unit test" job,
"test-gerrit-plugin-build". I wanted to verify that this would work for
any plugin before I wrote any changes to add it to plugin repos. This
job inherits from gerrit-plugin-build, but does two things: it adds the
"plugins/checks" repo to the list of required projects (so that Zuul
will check it out). And it sets a variable that tells the roles we just
wrote which plugin it should build. This variable normally just
defaults to the project under test, but in the case of this change, that
project is "zuul/jobs" which is obviously not a Gerrit plugin. So we
set the variable explicitly to "checks", and the job will build the
checks plugin.

5.1) DNM: test bazelisk test

https://gerrit-review.googlesource.com/c/zuul/jobs/+/255214

This is a throwaway change -- the checks plugin doesn't have junit
tests, but I wanted to make sure that the "bazelisk test" code in the
previous change works before I went any further, so I wrote a quick
change that builds the delete-project plugin instead of checks and runs
its tests. It works, I'm happy, so I abandoned the change. But this is
a useful process sometimes, and I wanted you to see it.

6) Add a Zuul job to build the plugin

https://gerrit-review.googlesource.com/c/plugins/checks/+/255215

This is the culmination of our work. We add the job that we made in
step 5 to the checks repo, and it runs. Note that in this case, we
don't have to override that variable -- it builds the checks plugin
because that's the project under test.

Up to this point all of the changes have been a git patch series. The
git parent of change 5 is change 4, and so forth up to 1. Naturally,
when Zuul looks at each of those changes, it checks out their parents as
well. This is true not only for the git repos on disk, but for Zuul's
own configuration. As of this moment, none of those changes have
merged, so the "running" configuration of Zuul doesn't have any of those
jobs we defined. If I visit

https://ci.gerritcodereview.com/t/gerrit/jobs

and search for "gerrit" no jobs appear, and they won't until those
changes merge. But because the central idea of Zuul is speculative
execution ("what would happen if this change did land?") Zuul
dynamically updates its configuration when it evaluates each of these
changes, so for that moment, in that context, the jobs do exist.

This feature doesn't stop at the git repository boundary though. One of
Zuul's best features is buried in the footer of the commit message of
this change:

Depends-On: https://gerrit-review.googlesource.com/c/zuul/jobs/+/255092

This is a cross-repo dependency. It tells Zuul that not only should it
check out change 255215 in the plugins/checks repo, it should also check out
change 255092 from the zuul/jobs repo, and any git parents of that
change. And not only should it check out that change, it should also
use the Zuul configuration specified there.

What we have here, is a series of 6 changes spanning two repositories
that build a complete CI setup that should work for any Gerrit plugin,
and we've demonstrated that it works end-to-end before merging any of
the implementing changes.

In other Zuul installations, we use this all the time to make changes
that span projects. I expect this to be very useful with Gerrit plugins
in particular -- we can test a change to Gerrit with any plugin (or
vice-versa) with a simple Depends-On in the commit message.

Not to dive too far down the rabbit hole -- but it does keep going.
Zuul supports cross-source dependencies -- we regularly have changes in
Zuul's own Gerrit that "Depends-On" changes in Github.

I think this is a good stopping point. I think all these changes are
ready for further review and merging (just don't merge the change to the
checks repo before the others -- if Zuul were gating, it would prohibit
that because of the Depends-On line, but in its current configuration,
it's just advisory). I think once these changes merge, we can start to
add more jobs and repos.

I'm happy to answer further questions here, or in review. And while
email and Gerrit are probably the best channels for discussion like
this, Monty, Paladox, and I are all available in #zuul (and some of us
in #gerrit too) on Freenode IRC if that happens to be convenient.

-Jim

[David Pursehouse]

On Sat, Feb 15, 2020 at 9:08 AM James E. Blair <cor...@inaugust.com> wrote:

cor...@inaugust.com (James E. Blair) writes:

> That looks like a great place to start, thanks!  I think I understand
> most of what needs to happen there.  I'll put together a series of
> changes that work toward this and come back here with a summary (and
> perhaps further questions).  I'm hoping this will be a useful way for
> folks to become familiar with this as we go.  I should be able to show
> most of this working (or not) without even merging any changes.
>
> One of the exceptions to that is this change, which I have gone ahead
> and merged:
>
>   https://gerrit-review.googlesource.com/c/zuul/ops/+/254715
>
> And you can see Zuul updated the deployed config here:
>
>   https://gerrit-zuul.inaugust.com/t/gerrit/build/7dc1d142bdd94b3facb2beeec4bacf80/console
>
> The change adds a new node type ("label") to the system, therefore it's
> something that needs to actually be merged before it takes effect.  Now
> that it has been, the next changes I write can use that label to run the
> new jobs.

Hi, I think I have a basic job for building plugins working!

This is awesome.  Thanks for working on this.  I'm really looking forward to being able to have plugin changes verified on CI, along with some other projects that currently still need to be manually verified, for example gitiles. 

Just one minor comment inline below.

The checks plugin has quite extensive tests...

https://gerrit.googlesource.com/plugins/checks/+/refs/heads/master/javatests/com/google/gerrit/plugins/checks

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/87y2t4vhiv.fsf%40meyer.lemoncheese.net.

[James E. Blair]

David Pursehouse <david.pu...@gmail.com> writes:

> This is awesome. Thanks for working on this. I'm really looking forward
> to being able to have plugin changes verified on CI, along with some other
> projects that currently still need to be manually verified, for example
> gitiles.
>
> Just one minor comment inline below.
>

>> This is a throwaway change -- the checks plugin doesn't have junit
>> tests,
>
>
> The checks plugin has quite extensive tests...
>
> https://gerrit.googlesource.com/plugins/checks/+/refs/heads/master/javatests/com/google/gerrit/plugins/checks

Indeed I thought that was the case, but I didn't see the "junit_tests"
string that the grep looks for in:

https://gerrit.googlesource.com/gerrit-ci-scripts/+/refs/heads/master/jenkins/gerrit-bazel-build-plugin.sh#15

which is the script I modeled the job after. What is the right way to
run the checks plugin tests? We can update the job (or make a new one)
to handle that.

-Jim

[David Ostrovsky]

Am Samstag, 15. Februar 2020 16:00:42 UTC+1 schrieb James E. Blair:

David Pursehouse <david.p...@gmail.com> writes:

> This is awesome.  Thanks for working on this.  I'm really looking forward
> to being able to have plugin changes verified on CI, along with some other
> projects that currently still need to be manually verified, for example
> gitiles.
>
> Just one minor comment inline below.
>
>> This is a throwaway change -- the checks plugin doesn't have junit
>> tests,
>
>
> The checks plugin has quite extensive tests...
>
> https://gerrit.googlesource.com/plugins/checks/+/refs/heads/master/javatests/com/google/gerrit/plugins/checks

Indeed I thought that was the case, but I didn't see the "junit_tests"
string that the grep looks for in:

  https://gerrit.googlesource.com/gerrit-ci-scripts/+/refs/heads/master/jenkins/gerrit-bazel-build-plugin.sh#15

which is the script I modeled the job after.

Yes, you are right. That's because checks plugin is an exception from

all other plugins, where there is only one single junit_tests rule for the

entire plugin exists, located in the root plugin BUILD file.

In checks plugin the authors decided to split the tests into different

packages (like in gerrit core).

  $ cd checks

  $ find javatest -name BUILD | wc -l

  9

 What is the right way to
run the checks plugin tests?

Target patterns generalize labels to include wildcards over packages

and targets. For example, checks/...:all (or just checks/...) is a target

pattern that evaluates to a set containing all rules in every package

recursively beneath the check directory:

  $ bazel test plugins/checks/...

[Han-Wen Nienhuys]

Are you aware of topic submits? These are widely used on our Android hosts to manage cross-repository changes. Does Zuul support those too?

-- 

[James E. Blair]

"'Han-Wen Nienhuys' via Repo and Gerrit Discussion"

<repo-d...@googlegroups.com> writes:

> On Sat, Feb 15, 2020 at 1:08 AM James E. Blair <cor...@inaugust.com> wrote:
>
>> cor...@inaugust.com (James E. Blair) writes:
>> This feature doesn't stop at the git repository boundary though. One of
>> Zuul's best features is buried in the footer of the commit message of
>> this change:
>>
>> Depends-On: https://gerrit-review.googlesource.com/c/zuul/jobs/+/255092
>>
>> This is a cross-repo dependency. It tells Zuul that not only should it
>> check out change 255215 in the plugins/checks repo, it should also check
>> out
>> change 255092 from the zuul/jobs repo, and any git parents of that
>> change. And not only should it check out that change, it should also
>> use the Zuul configuration specified there.
>>
>>
> Are you aware of topic submits? These are widely used on our Android hosts
> to manage cross-repository changes. Does Zuul support those too?

Zuul doesn't currently support those directly (so if there's a
dependency relationship you want Zuul to test, you would still need to
add Depends-On footer). But why is an interesting story...

The Depends-On footer was directly modeled after the original
cross-project-dependency plan for Gerrit (that was the name of the
footer that was to be used). Originally the plan with Gerrit was to
specify the dependency with the change-id. So the footer looked like:

Depends-On: Ib30e409044ccc48b4c19beae36f1bcd453ebef8e

Zuul implemented that in order to be compatible with Gerrit. But the
Gerrit project to implement this was abandoned, and later
submit-whole-topic was proposed instead.

In the mean time, we had begun using Depends-On heavily in OpenStack,
and one of the things we really liked about it was that it forced a
one-way dependency relationship on changes. OpenStack is based on (what
is now called) a microservice architecture and we wanted people to be
able to deploy it continuously, without downtime. This meant that we
could not allow a change to merge that did not support a
backwards-compatible upgrade (so that we could be sure that someone
could update one component without needing to update another at exactly
the same time). For that reason, we didn't want to take advantage of
submit-whole-topic (and are still unlikely to want to do so in
OpenStack).

Meanwhile, new folks have joined the Zuul community with systems other
than Gerrit. Since they don't have change-ids and Gerrit has abandoned
future use of the Depends-On footer, we updated it to use URLs instead
of change ids, so that Zuul supports changes in Gerrit that depend on
changes in Github.

We also have folks in our community who don't share our constraints
about merging dependent changes in sequence, and so work has begun on
supporting simultaneous (or circular) dependencies. Once that work is
in place, Zuul will have the ability to test a whole topic as a single
submission rather than a series of changes. Once that is there, it
probably wouldn't be too hard to have the Zuul Gerrit driver derive that
input from the Gerrit topic setting rather than footers (though
obviously footers would be needed for cross-source dependencies).

-Jim

[James E. Blair]

David Ostrovsky <david.o...@gmail.com> writes:

> Yes, you are right. That's because checks plugin is an exception from
> all other plugins, where there is only one single junit_tests rule for the
> entire plugin exists, located in the root plugin BUILD file.
>
> In checks plugin the authors decided to split the tests into different
> packages (like in gerrit core).
>
> $ cd checks
> $ find javatest -name BUILD | wc -l
> 9
>
> What is the right way to
>> run the checks plugin tests?
>
>
> Target patterns generalize labels to include wildcards over packages
> and targets. For example, checks/...:all (or just checks/...) is a target
> pattern that evaluates to a set containing all rules in every package
> recursively beneath the check directory:
>
> $ bazel test plugins/checks/...

Thanks! This is the perfect opportunity to show how we can customize
how a job is run in a single repo without making a new job:

https://gerrit-review.googlesource.com/c/plugins/checks/+/255472/1

-Jim

[James E. Blair]

Hi,

I anticipate that we're getting close to a point where we might want to
add Zuul jobs to a number of plugin repositories. That will necessitate
creating checkers in Gerrit for each of those repositories (currently, a
checker may only be attached to a single repo).

I like having system configuration in Git repos where anyone can
contribute and it is subject to code review, but unlike other Gerrit
configuration, we can't edit a project.config file to configure
checkers.

To address this, I created a small Ansible module which can be run as
part of the Zuul self-deployment job which uses Zuul's credentials to
create or update checkers in Gerrit. It takes a simple declarative YAML
file as input, so anyone can see or propose changes to modify checker
configuration, and the appropriate people can review these changes.

I have proposed the change here, and it includes a starter config with
the current Zuul-related checkers:

https://gerrit-review.googlesource.com/256652

Note that the job will output the complete configuration of all checkers
in the system which are then available to anyone via the build logs. I
wrote it that way intentionally to aid contributors who may not have
access to the list checkers API endpoint (since it requires the
Administrate Checkers permission). I don't believe there is any
sensitive information there, but it would be good for the Gerrit
Maintainers to confirm this is okay before we merge this change.

-Jim

[Luca Milanesio]

Hi James,
Thanks for the update.

> On 25 Feb 2020, at 00:40, James E. Blair <cor...@inaugust.com> wrote:
>
> Hi,
>
> I anticipate that we're getting close to a point where we might want to
> add Zuul jobs to a number of plugin repositories.

Would this be inside each of the plugin repositories? Or is there a repo with the collection of all the jobs for building the plugins?

> That will necessitate
> creating checkers in Gerrit for each of those repositories (currently, a
> checker may only be attached to a single repo).

Could this phase be implicitly defined by the build job itself?
If you define a job to build the plugin X, it is obvious that you would need to create a Checker for it.

>
> I like having system configuration in Git repos where anyone can
> contribute and it is subject to code review, but unlike other Gerrit
> configuration, we can't edit a project.config file to configure
> checkers.
>
> To address this, I created a small Ansible module which can be run as
> part of the Zuul self-deployment job which uses Zuul's credentials to
> create or update checkers in Gerrit. It takes a simple declarative YAML
> file as input, so anyone can see or propose changes to modify checker
> configuration, and the appropriate people can review these changes.

That would increase the maintenance burden, as we have one more place to remember to edit whenever a new project is created.
Is there a way to create a Checker automatically and implicitly for every repo?

>
> I have proposed the change here, and it includes a starter config with
> the current Zuul-related checkers:
>
> https://gerrit-review.googlesource.com/256652
>
> Note that the job will output the complete configuration of all checkers
> in the system which are then available to anyone via the build logs. I
> wrote it that way intentionally to aid contributors who may not have
> access to the list checkers API endpoint (since it requires the
> Administrate Checkers permission). I don't believe there is any
> sensitive information there, but it would be good for the Gerrit
> Maintainers to confirm this is okay before we merge this change.
>
> -Jim
>

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/87mu97sdmr.fsf%40meyer.lemoncheese.net.

[James E. Blair]

Luca Milanesio <luca.mi...@gmail.com> writes:

> Hi James,
> Thanks for the update.
>
>> On 25 Feb 2020, at 00:40, James E. Blair <cor...@inaugust.com> wrote:
>>
>> Hi,
>>
>> I anticipate that we're getting close to a point where we might want to
>> add Zuul jobs to a number of plugin repositories.
>
> Would this be inside each of the plugin repositories? Or is there a
> repo with the collection of all the jobs for building the plugins?

This is one of the intentional changes we made with Zuul v3 as compared
to jenkins-job-builder. Rather than having a job for each plugin, we'll
have a single job that can build any plugin (or maybe two jobs for
in-tree and out-of-tree plugins), and then just attach the appropriate
job to each repo that should use it.

The main job definitions are central, and we add just a small change to
each plugin repo to tell Zuul to run that job for that repo (with any
localized customization if necessary). Here's what it looks like for
the checks repo:

https://gerrit-review.googlesource.com/255215

And here's a follow-up patch which adds a little bit of customization
specific for the checks plugin repo:

https://gerrit-review.googlesource.com/255472

Note that Zuul ran the jobs that those changes are adding -- you can see
the results in the Checks tab, even though the changes have not merged.

>> That will necessitate
>> creating checkers in Gerrit for each of those repositories (currently, a
>> checker may only be attached to a single repo).
>
> Could this phase be implicitly defined by the build job itself?
> If you define a job to build the plugin X, it is obvious that you
> would need to create a Checker for it.

Not in the job itself -- there's a catch-22. The checks plugin is
designed around a polling system; since that's what we're using here,
Zuul won't notice any changes to any projects that don't already have a
checker.

>> I like having system configuration in Git repos where anyone can
>> contribute and it is subject to code review, but unlike other Gerrit
>> configuration, we can't edit a project.config file to configure
>> checkers.
>>
>> To address this, I created a small Ansible module which can be run as
>> part of the Zuul self-deployment job which uses Zuul's credentials to
>> create or update checkers in Gerrit. It takes a simple declarative YAML
>> file as input, so anyone can see or propose changes to modify checker
>> configuration, and the appropriate people can review these changes.
>
> That would increase the maintenance burden, as we have one more place
> to remember to edit whenever a new project is created.
> Is there a way to create a Checker automatically and implicitly for every repo?

Zuul already needs to be told about each repo added to it. The current
list is here:

https://gerrit.googlesource.com/zuul/ops/+/refs/heads/master/zuul/main.yaml

We're already going to need to edit that when we add repos to Zuul. I'm
expecting that once we're happy with the first few repos, we'll
batch-add all the rest.

This is in the same repo where I'm proposing 'checkers.yaml', so adding
a project to both can be done in the same change. I figured at least
for now, the additional flexibility would be nice. For instance, we
only have 4 checkers defined right now, though Zuul knows about 18
repositories. I'm nervous about adding checkers since they can not be
deleted. This system also lets us easily enable and disable checkers.
But if we'd like to reduce the double accounting, we could automatically
generate the checkers configuration from the zuul project list.

Eventually, if and when we're sure we want the same configuration for
every repo in Gerrit, we could probably script the whole thing and have
Zuul's tenant config and the checkers for Zuul created automatically
from the project list.

-Jim

[James E. Blair]

Hi,

We've started a new thread of work related to Zuul for Gerrit. We
started by creating a job to build a plugin. We've now also created a
job to build Gerrit itself. Here's an overview of all the outstanding
changes and where the work stands.

Operations
==========

First, the most important blocker at this point is this change:

Update checkers in Gerrit after deployment
https://gerrit-review.googlesource.com/256652

This is my proposal for how to collaboratively manage the configuration
of checkers through code review in Gerrit, previously discussed on this
list. We need sign-off from the Gerrit Maintainers for this. Until
this merges, we will be unable to add any more checker definitions to
new projects.

Gerrit Plugin Jobs
==================

We do have a checker configured for the "Checks" plugin, and the job to
build and run tests for a plugin is ready to go. These two changes tell
Zuul to run the job on the plugin/checks repo, and further, to customize
the test command for the checks plugin (which, based on the template
from which I was working, uses a different test command than other
plugins).

Add a Zuul job to build the plugin
https://gerrit-review.googlesource.com/255215
Zuul: run the tests after build
https://gerrit-review.googlesource.com/255472

Gerrit Build Jobs
=================

This is where the bulk of the new work has been happening. Once again,
there are several changes here, and due to the speculative execution
approach in Zuul, we can see all of them work before any have merged.

I've attached a graph of the dependencies between these changes for the
benefit of those who prefer a more visual mental model. I'll work from
the bottom to the top of the graph.

Drop temp-debian nodeset
------------------------

https://gerrit-review.googlesource.com/256632

This one is straightforward. A nodeset defines a grouping of test
resources used by a job (in this case, we're running jobs on a single
debian node). In earlier changes I added a temporary nodeset definition
to zuul/jobs while we worked out what the default nodeset would be in
the zuul/config repo. Now that that change has merged, we can drop our
use of the temporary nodeset, and these jobs will revert to the default.

Fix branch detection in prepare-gerrit-repos
--------------------------------------------

https://gerrit-review.googlesource.com/257080

The role that we previously created to set up the multiple repos needed
to build Gerrit (and/or plugins) had some errors which didn't manifest
when running on a single branch, but do show up with stable branches.
The stable branch jobs that you'll see later in the stack uncovered
these errors, so this change earlier in the stack fixes them.

Correct some submodule edge cases
---------------------------------

https://gerrit-review.googlesource.com/257272

Similarly, this is a change to the same role which corrects errors that
we didn't notice until we started using it in different submodule
configurations (i.e., on stable branches without jgit, or on a plugin
which did not have a stable branch).

We're starting to use this role in OpenDev as well, so that we can
collaborate on making this more robust. Some of the edge cases came
from that environment, and so Monty has created a change[1] in OpenDev
which depends on this in order to continue his work there.

[1] https://review.opendev.org/709602

Rename gerrit-base to gerrit-setup
----------------------------------

https://gerrit-review.googlesource.com/255652

There was a long discussion on a previous change about where to put the
stable branch variants of the Gerrit build job. I think we've hit the
point where it makes sense to put them inside the Gerrit repo, so this
change, and the next several, accomplish that move.

The reason to put at least part of this configuration in the Gerrit repo
itself is to locate the branch-specific configuration as close to the
branch in question as possible. This way, the 2.16 version of the job
automatically runs when someone uploads a change to stable-2.16, and the
master version when a change is uploaded to master.

One approach would be to simply put the entirely of the Gerrit build job
into the gerrit repo (not only the job definition, but also the
associated playbooks and roles). But those might still change
frequently as we build out this system, and the code churn in the Gerrit
repo may not be appreciated.

Instead, the approach of this and the following changes is to keep the
bulk of the job in the zuul/jobs repo, and put the minimal amount of
per-branch configuration in the gerrit repo. This change sets that up.

This change defines a "gerrit-setup" job which prepares the git
submodules (whatever they may be).

A later change will add a "gerrit-base" job to the gerrit repo which
adds the repos needed for all gerrit-related builds (ie, it adds jgit,
but only on the master and stable-3.1 branches).

This change also defines a "gerrit-build-base" job, which temporarily
inherits from gerrit-setup, but will inherit from gerrit-base once it
exists. This job will execute the bazelisk build steps.

Finally, a later change will add "gerrit-build" which inherits from
gerrit-build-base and adds the appropriate core plugins.

In short, the inheritance chain moves back-and-forth between zuul/jobs
and gerrit so that we can define all the build steps in zuul-jobs, but
all the branch-specific information in gerrit.

gerrit-build (in gerrit repo) |
gerrit-build-base (in zuul/jobs repo) |
gerrit-base (in gerrit repo) |
gerrit-setup (in zuul/jobs repo) v

Let's move on to the changes in Gerrit.

Add Zuul config
---------------

https://gerrit-review.googlesource.com/257073 (master)
https://gerrit-review.googlesource.com/257075 (stable-3.1)
https://gerrit-review.googlesource.com/257076 (stable-3.0)
https://gerrit-review.googlesource.com/257077 (stable-2.16)

The goal is to have a single job ("gerrit-build") which builds gerrit.
When someone uploads a change to gerrit master, that job should run and
work. When someone uploads a change to stable-2.16, the same job should
run and work. But master and stable-2.16 have important differences,
particularly with respect to which submodules are required (but we can
easily imagine other changes accumulating as well -- bazel or java
versions, etc). The easiest and most natural way to accomplish this is
to define the job in the gerrit repo, and to do so on each branch. That
means that Zuul will automatically use the configuration of the job from
the appropriate branch when it runs the job.

Merging changes like this to each branch is a one-time cost. I'm not
going to promise that we won't make changes like this again, but I hope
not, at least not often. Once this is set up, the workflow generally
just stays out of the way.

Imagine that all of these changes are merged. Then, at some point, we
add a new core plugin, the foobar plugin. We'll need to add a commit to
add it to .gitmodules. In that same commit, we'll need to update the
Zuul configuration for the gerrit build job to add the plugins/foobar
repo to required-projects. Already you can see one benefit of this
approach -- since the list of required-projects for the gerrit-build job
is in the gerrit repo, when we add a new one, we can do so in a single
change (if this were maintained in the zuul/jobs repo, we would need two
changes to add a new core plugin).

This change would be proposed to the master branch only, since we're
only going to add the foobar plugin to new versions of Gerrit. Later
on, when we create the stable-3.2 branch, we will branch from master,
and the .zuul.yaml file will already contain the foobar plugin, so no
additional changes are required. Again, if this were defined in
zuul/jobs, we would need further changes to configure the stable-3.2
branch.

Note: these changes also tell Zuul to run the gerrit-build job on
changes to each of these branches, however, we can not see it running
yet because there is no checker for Zuul defined on the gerrit project.
Once we define one, they will start running.

Add gerrit build test jobs
--------------------------

https://gerrit-review.googlesource.com/257078

We're back in the zuul/jobs repo now. Notice that this change depends
on all four changes above. That means that when this job runs, the
gerrit-base and gerrit-build jobs will be fully defined for all four
branches.

We had some jobs temporarily parented to gerrit-setup since gerrit-base
did not exist yet. Now that it does, we can reparent them.

And since we don't have a checker defined on the gerrit repo, the only
way to actually see the gerrit build jobs run for each branch is to do
so here, so we create 4 new jobs which are intended only to run on
changes to this repo which run the gerrit builds from all 4 branches.
This way, when we make a change to one of the roles used by
gerrit-build, we can confirm that it doesn't break on any of the
supported branches.

Conclusion
==========

I tried to write useful commit messages, but hopefully this walkthrough
adds some extra context to the effort and ties all the changes
together. I think all of these are ready for review and (hopefully)
merging at this point.

-Jim