[Announce] New Gerrit continuous integration server (w/ review)

160 views
Skip to first unread message

lucamilanesio

unread,
May 13, 2015, 9:42:17 AM5/13/15
to repo-d...@googlegroups.com
I have spinned off a new Gerrit Continous Integration server, now 100% managed and configured by the Gerrit community :-)

The new URL is:

The YAML files for its configuration are on:

As soon a a change on the YAML files is merged, the corresponding Jobs on gerrit.gerritforge.com are automatically updated (or created). This automation is based on the OpenStack's Jenkins Job Builder [1].

Documentation of the YAML format can be found at [2] and the format is based on templating so allows to reduce duplication and automatically creating jobs on a per branch-basis.

The old CI (ci.gerritforge.com) will remain active as it contains the build of other GerritForge's specific packages, however moving forward more builds and plugins will move to the new YAML-based continuous integration.

Feel free to create the YAML files related to your plugins / jobs so that they can be built on gerrit.gerritforge.com.

lucamilanesio

unread,
May 13, 2015, 9:45:12 AM5/13/15
to repo-d...@googlegroups.com
I forgot to mention that I would like to ask other Gerrit contributors to co-manage the instance with us so that we can have more coverage on the day-to-day system maintenance.

@Gerrit contributors / maintainers: Feel free to login via GitHub OAuth to gerrit.gerritforge.com and request permissions to admin the Jenkins instance.

P.S. One idea could be as well to define the physical machine as well as Dockerfile and allow anybody to propose changes and review them, so that should any disaster happen we could be able to spin another Docker image somewhere else in the world, without loosing anything in terms of configuration.

Feedback / proposals are more than welcome (as usual) :-)

Luca.

Khai Do

unread,
May 14, 2015, 1:06:00 PM5/14/15
to repo-d...@googlegroups.com
Neat Luca, thanks for setting this up.  We have actually setup the same thing for Openstack a while ago and of course we use Jenkins job builder as well :)  We'll still need to build our own Gerrit since it's a fork but might serve as a nice repository to grab specific versions of the plugins.  We currently build the plugins we use against our forked Gerrit so my concern is that getting one from a stable-2.x branch may not work.  

I was wondering why the name 'gerrit.gerritforge.org'?  It's a jenkins not a Gerrit, right?  Also was wondering whether you plan on setup a proper repository for the artifacts or even pushing directly to maven central?    

-Khai

Luca Milanesio

unread,
May 14, 2015, 3:12:43 PM5/14/15
to Khai Do, repo-discuss
Hi Khai,
I remember you actually proposed / inspired the Jenkins Job Builder workstream for Gerrit … and here we are :-)

Now more YAML files need to be defined to have all the plugins as well built for every commit and with the correct graph dependencies.
Everyone can contribute their YAML files and make them live on the CI.

On 14 May 2015, at 18:06, Khai Do <zaro...@gmail.com> wrote:

Neat Luca, thanks for setting this up.  We have actually setup the same thing for Openstack a while ago and of course we use Jenkins job builder as well :)  We'll still need to build our own Gerrit since it's a fork but might serve as a nice repository to grab specific versions of the plugins.  We currently build the plugins we use against our forked Gerrit so my concern is that getting one from a stable-2.x branch may not work.  

In theory they should work, as far as you haven’t changed the plugin API in your forked Gerrit.


I was wondering why the name 'gerrit.gerritforge.org'?  It's a jenkins not a Gerrit, right?

Yes, the name could be misleading. What about gerrit-ci.gerritforge.com?

 Also was wondering whether you plan on setup a proper repository for the artifacts or even pushing directly to maven central?    

We already have one for the master versions of the plugins at it is at:

The problem with plugins is really ownership and release management: who owns them? who releases them?
What we *could* do is an automatic build and “smoke test” just to check that a plugin:
- is buildable
- inits correctly
- loads correctly

When those checks are OK then we could automatically create a “stable-XXX” version of the plugin and release it to Maven Central as we do for Gerrit.
However it requires a bit of work do automate it.


Luca Milanesio

unread,
May 14, 2015, 4:21:44 PM5/14/15
to Christian Aistleitner, repo-d...@googlegroups.com
Hi Christian,
very nice work, well done :-)

Are you willing to contribute them to the gerrit-ci-scripts project?
We could have:
/jenkins => for the CI yaml files
/nightly => for your set of scripts

> On 14 May 2015, at 17:14, Christian Aistleitner <chri...@quelltextlich.at> wrote:
>
> Hi Luca,
>
> On Wed, May 13, 2015 at 06:45:12AM -0700, lucamilanesio wrote:
>> I forgot to mention that I would like to ask other Gerrit contributors to
>> co-manage the instance with us so that we can have more coverage on the
>> day-to-day system maintenance.
>>
>> [...]
>>
>> Feedback / proposals are more than welcome (as usual) :-)
>
> As people more often than not asked me for its-* plugin jars, and
> since Jenkins did not nicely expose what I was looking for, I am
> running a custom script to build nightly artifacts since not quite
> 2 months. The output (and artifacts :-) ) can be found at
>
> http://builds.quelltextlich.at/gerrit/nightly/index.html

One question: why running them every night even when there are no new changes?
We had in a project years ago a “nightly build + deployment” in a QA environment … and you know what? It was mostly broken :-(

The reason was very simple: people were hurrying up in the evening to catch the train / bus or avoid traffic jams … and they were taking less attentions on the “last commit of the day”.
The best time to “cut a build” was then around 11-12 AM :-) … and went from nightly build to the “permalink” concept. We started deploying the “most stable build” of the day before rather than the one at midnight.

>
> Builds for master branch are at:
>
> http://builds.quelltextlich.at/gerrit/nightly/master/index.html

See for instance:

2015-04-04 v2.11-rc1-248-gbf37566
2015-04-03 v2.11-rc1-248-gbf37566

Those two builds are exactly the same code: do you build them separately or they are just links to the same artifact?

>
> Today's build for the master branch is at:
>
> http://builds.quelltextlich.at/gerrit/nightly/master/2015-05-14/index.html
>
> I wanted all branches to go all green [1] before I'd announce it
> ... but since you asked for feedback, I guess I should at least
> mention why Jenkins did not work for me.

Yes, that is going to be really useful :-)

>
> Jenkins did not fit nicely, as I wanted to better expose the grouping
> of artifacts. So already on the overview page for a branch, I wanted
> to see when which group caused issues, and also how severe it is. Also
> I wanted to make the repo description and schema number visible on the
> overview pages.

OK, so it was a UX issue. We could easily have a different landing page and just point to the Jenkins artifacts using their permalinks.

>
> In the page for a build, I wanted to bring the needed log files closer
> to the artifacts. And I wanted to decorate the artifacts with links to
> the repo browser, and the gerrit dashboards. So one can check with
> only a single click if there is already a fix for that issue on the
> dashboard / open changes.

Nice: why don’t we try to build that GUI together using your current page as example?

>
> Finally, I wanted to make it possible to isolate the building of
> plugins. So an error in the BUCK file of a plugin should not prohibit
> building core's WAR files, etc.
>
> I wanted it to be extensible, so I can easily add/remove plugins from
> the build pipeline.

That should be easy with the Jenkins YAML files, using the templates and multi-branch builds.

>
> And I wanted it to be possible to add system tests [2]. So I needed a
> scripting environment anyways.

That is really cool indeed!! A full integration test environment with plugins is currently missing and will be more than welcome :-)

>
> Sure, I could have hacked my whishlist into Jenkins. But I wanted some
> jars right away, as people asked me for them. Hence, I went for the
> quick and dirty approach, and prototyped a small bash script to
> experiment and find out what I'd like it to be.

You could do exactly the same thing with Jenkins using the Build Flow plugin, using a powerful Groovy-based DSL.
Shall we try this together?

>
> The script is available at
>
> git://git.quelltextlich.at/gerrit/gerrit-builder
>
> (no online browsing :-/ ... you have clone it). Maybe some parts of
> that are useful for you or others.

Why don’t you push the code for review to https://gerrit.googlesource.com/gerrit-ci-scripts ?

>
>
>
> Have fun,
> Christian
>
> P.S.: If someone wants me to add her/his plugin there, just shoot me a
> line.

If you push to gerrit-ci-scripts everyone can put their plugin script by themselves, more scalable :-)

>
>
> [1] its-storyboard is still failing a test on Java 8. The fix is at
>
> https://gerrit-review.googlesource.com/#/c/66735/
>
> since a few weeks, but it is still waiting for a merge.

Gerrit + Java 8 is a very wanted combination … but we’re still a bit far I’m afraid … or not?

>
> [2] Currently, each WAR is tested whether or not it allows to
> bootstrap a site, and for each plugin it is at least checked whether
> or not it loads correctly.

Yes, that’s exactly what I was looking for, well done :-)

Luca.

lucamilanesio

unread,
May 15, 2015, 3:01:13 AM5/15/15
to repo-d...@googlegroups.com, chri...@quelltextlich.at
@Christian: there you go [1].

Once the change is merged, feel free to contribute your scripts to the Gerrit Community :-)
I've included you as well in the list of gerrit-ci-scripts maintainers.

Christian Aistleitner

unread,
May 15, 2015, 5:40:56 AM5/15/15
to lucamilanesio, repo-d...@googlegroups.com
Hi Luca,

On Wed, May 13, 2015 at 06:45:12AM -0700, lucamilanesio wrote:
> I forgot to mention that I would like to ask other Gerrit contributors to
> co-manage the instance with us so that we can have more coverage on the
> day-to-day system maintenance.
>
> [...]
>
> Feedback / proposals are more than welcome (as usual) :-)

As people more often than not asked me for its-* plugin jars, and
since Jenkins did not nicely expose what I was looking for, I am
running a custom script to build nightly artifacts since not quite
2 months. The output (and artifacts :-) ) can be found at

http://builds.quelltextlich.at/gerrit/nightly/index.html

Today's build for the master branch is at:

http://builds.quelltextlich.at/gerrit/nightly/master/2015-05-14/index.html

I wanted all branches to go all green [1] before I'd announce it
... but since you asked for feedback, I guess I should at least
mention why Jenkins did not work for me.

Jenkins did not fit nicely, as I wanted to better expose the grouping
of artifacts. So already on the overview page for a branch, I wanted
to see when which group caused issues, and also how severe it is. Also
I wanted to make the repo description and schema number visible on the
overview pages.

In the page for a build, I wanted to bring the needed log files closer
to the artifacts. And I wanted to decorate the artifacts with links to
the repo browser, and the gerrit dashboards. So one can check with
only a single click if there is already a fix for that issue on the
dashboard / open changes.

Finally, I wanted to make it possible to isolate the building of
plugins. So an error in the BUCK file of a plugin should not prohibit
building core's WAR files, etc.

I wanted it to be extensible, so I can easily add/remove plugins from
the build pipeline.

And I wanted it to be possible to add system tests [2]. So I needed a
scripting environment anyways.

Sure, I could have hacked my whishlist into Jenkins. But I wanted some
jars right away, as people asked me for them. Hence, I went for the
quick and dirty approach, and prototyped a small bash script to
experiment and find out what I'd like it to be.

The script is available at

git://git.quelltextlich.at/gerrit/gerrit-builder

(no online browsing :-/ ... you have clone it). Maybe some parts of
that are useful for you or others.



Have fun,
Christian

P.S.: If someone wants me to add her/his plugin there, just shoot me a
line.



[1] its-storyboard is still failing a test on Java 8. The fix is at

https://gerrit-review.googlesource.com/#/c/66735/

since a few weeks, but it is still waiting for a merge.

[2] Currently, each WAR is tested whether or not it allows to
bootstrap a site, and for each plugin it is at least checked whether
or not it loads correctly.



--
---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ----
Companies' registry: 360296y in Linz
Christian Aistleitner
Kefermarkterstrasze 6a/3 Email: chri...@quelltextlich.at
4293 Gutau, Austria Phone: +43 7946 / 20 5 81
Fax: +43 7946 / 20 5 81
Homepage: http://quelltextlich.at/
---------------------------------------------------------------
signature.asc

Christian Aistleitner

unread,
May 17, 2015, 11:19:40 AM5/17/15
to Luca Milanesio, repo-d...@googlegroups.com
Hi Luca,

On Thu, May 14, 2015 at 09:21:37PM +0100, Luca Milanesio wrote:
> Are you willing to contribute them to the gerrit-ci-scripts project?

Sure, see

https://gerrit-review.googlesource.com/#/c/67924/1

We can discuss on the change, if you prefer a proper full import with
history instead of a submodule.



> One question: why running them every night even when there are no
> new changes?

Because that way it is easier to communicate what's going on.

If on

http://builds.quelltextlich.at/gerrit/nightly/stable-2.10/index.html

you'd only see

| [...] |
| ok | 2015-04-28 | v2.10.3.1-5-ge678284 | [...] |
| ok | 2015-05-01 | v2.10.3.1-8-gfa7880a | [...] |
| [...] |

with no 2015-04-29, and no 2015-04-30, people would start wondering
what was up with those missing dates.
Did v2.10.3.1-6 already exist for some date and build machine was
down? Or did things break really badly for those dates?

By having a line for each and every day, progress is clear and visible
to people, and one can avoid such questions.



> We had in a project years ago a “nightly build + deployment” in a QA
> environment … and you know what? It was mostly broken :-(

That's understandable. But when looking gerrit's build history, that's
clearly not the case:

http://builds.quelltextlich.at/gerrit/nightly/master/index.html

Kudos gerrit maintainers! :-)



> The best time to “cut a build” was then around 11-12 AM :-) [...]

With gerrit maintainers being spread out over at least three very
different time-zones, I guess there is no real, common “11-12 AM” :-)



> > Builds for master branch are at:
> >
> > http://builds.quelltextlich.at/gerrit/nightly/master/index.html
>
> See for instance:
>
> 2015-04-04 v2.11-rc1-248-gbf37566
> 2015-04-03 v2.11-rc1-248-gbf37566
>
> Those two builds are exactly the same code: do you build them
> separately or they are just links to the same artifact?

I build them separately.
It's a bit redundant, if none on the plugins change.
Agreed.

But a build is currently taking <10 minutes/day, so there is not much
win in implementing logic to filter out duplicate builds and linking
them accordingly.

The intention of my scripts is really to have a daily snapshot of
everything, and not to build for each commit ... although the latter
would be easy to achieve actually ... if someone wanted that.


> Gerrit + Java 8 is a very wanted combination … but we’re still a bit
> far I’m afraid … or not?

For building with Java 8, there were a few issues initially. But fixes
for them got merged. Core and most plugins are buildable on Java 8,
and their tests pass just fine.
I only did some limited testing in running sites with Java 8, but up
to now, it seemed to work just fine.



Have fun,
Christian
signature.asc

chri...@quelltextlich.at

unread,
May 17, 2015, 11:21:26 AM5/17/15
to lucamilanesio, repo-d...@googlegroups.com
Hi,

On Fri, May 15, 2015 at 12:01:13AM -0700, lucamilanesio wrote:
> Once the change is merged, feel free to contribute your scripts to the
> Gerrit Community :-)

just to keep archives connected:

https://gerrit-review.googlesource.com/#/c/67924

Have fun,
Christian
signature.asc

Doug Kelly

unread,
May 18, 2015, 2:33:21 PM5/18/15
to repo-d...@googlegroups.com
I'll be happy to help contribute scripts for the plugins I've had involvement with.  I was actually curious how to best test these changes, though, before unleashing them on the world. :) Docker seems like a perfectly sane way to do this.  Looking at the current YAML configuration, it doesn't look like there's much to it, but I would suppose we will eventually want to settle on some standard naming to keep it all logically separated.

Of course, if there's anything else I can do to help, please let me know!  This looks exciting to try... and I've appreciated your plugin-building service from ci.gerritforge.com, so I'm glad to help see this continues!

--Doug

Doug Kelly

unread,
May 18, 2015, 9:27:22 PM5/18/15
to repo-d...@googlegroups.com
OK, it's not perfect yet, but here's a start: https://github.com/vangdfang/gerrit-ci

I've found a few issues currently... one, if multiple builds trigger simultaneously (possibly on different branches/buck revisions), there's a good chance one or both builds will fail as a result.  The second issue is the piping of yes to the build script may be either throwing off error messages or otherwise causing issues (logs have "yes: standard output: Broken pipe" just before the build fails)... a cleaner way of achieving the same result that I *think* is intended is using:

export BUCK_CLEAN_REPO_IF_DIRTY=y

in the job configuration.  That's what I did locally here... and it seems to do okay (although I have other issues with that build environment being on too old of a JDK now, I think).

Anyway, that's enough for now.  Time for a dinner break!

--Doug

Doug Kelly

unread,
May 19, 2015, 11:01:05 PM5/19/15
to repo-d...@googlegroups.com
Good news! The Docker image seems to work fully at this point. There are some caveats when using boot2docker, but otherwise, it seems to run and build as expected!

Doug
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/gnxUdzyzbIY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Luca Milanesio

unread,
May 20, 2015, 2:17:57 AM5/20/15
to Doug Kelly, repo-d...@googlegroups.com
Hey Doug,
really cool :-)

Can we push the Dockerfile to a /docker subdir onto gerrit-ci-script? It will be easier for the people to get all the needed stuff from one place and will allow code reviews.

Thanks for contributing to it :-)

Luca


Sent from my iPhone
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Doug Kelly

unread,
May 20, 2015, 8:35:13 AM5/20/15
to repo-d...@googlegroups.com, doug...@gmail.com
Agreed! I just dumped it on Github for some scratch space as I was working; I've submitted the review here:

Expect some new jobs and minor tweaks in the near future!
Reply all
Reply to author
Forward
0 new messages