Bazel's new CI has arrived. Jenkins is gone.

[Philipp Wollermann]

Hi,

Bazel's new CI has arrived! It is based on Buildkite (www.buildkite.com) and completely replaces our old Jenkins setup.

It brings a modern, much more user-friendly UI and runs on completely redesigned and improved worker infrastructure, which makes everything very fast, reliable and gets rid of flaky test failures. This will allow us to test Bazel more often and consistently against downstream projects to ensure that we never accidentally break your use case with a new release. If you'd like to get your project tested on our CI, reach out to me and we can see how we can make it happen.

You can access the new CI by simply logging into https://buildkite.com/bazel. Currently, this requires creating an account via an invite, which we'll happily provide for all rules maintainers and interested contributors. :) Just send me an e-mail, and I'll send you an invite. We're also working with Buildkite to enable public anonymous read-only access to the CI.

We also integrated a shiny new test results viewer, which is active for GitHub pull requests - this should make it easier to see if tests failed.

Check out an example report here: https://source.cloud.google.com/results/invocations/799a7068-7597-4cc3-b69a-3978b7edec92/targets

You can find some first documentation and screenshots for the new setup here:

https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md

We're going to add a rules maintainer's and contributor's guide, too and hope to have it ready in the next days.

Maintainers: In the meantime, if you have any questions, complaints, stuff doesn't work for your projects in the github.com/bazelbuild org... let me know and we'll figure it out together.

Thanks to everyone on the Bazel team and community who helped with this undertaking!

And also a big thanks to the awesome people at Buildkite, who not only created a great CI platform, but make it available for free for open-source projects.

Cheers,
Philipp & Jakob

[Fedor Korotkov]

Hey Philipp,

I have a question about workflow for contributors. Seems there is no way to run a CI for my fork before creating an actual PR and waiting for an approval from a project member. Do I understand it correctly?

https://buildkite.com/bazel doesn't look like to be publicly available as you mentioned. But will the build page for my PR be available publicly? Or I will just see failing GitHub status checks without a way to see the logs until someone will invite me to Buildkite?

Best,

Fedor

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CA%2BAhZogFT9eO5zvsdwpsLzDjS3RFmy%3DU41Zc%2BwwxcPEKULBLVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Philipp Wollermann]

Hi Fedor,

On Tue, Feb 27, 2018 at 6:50 PM Fedor Korotkov <fedor.k...@gmail.com> wrote:

I have a question about workflow for contributors. Seems there is no way to run a CI for my fork before creating an actual PR and waiting for an approval from a project member. Do I understand it correctly?

Yes - we would love to simply allow this and are thinking about how it could be made possible, but it's also clearly a security issue: If we allow this, anyone with a GitHub account could run arbitrary code on our CI machines. :|

Note that this is not a change to our older system based on Jenkins - even then project members had to write the magic "Jenkins, test this please" phrase in order to trigger a CI run.

https://buildkite.com/bazel doesn't look like to be publicly available as you mentioned. But will the build page for my PR be available publicly? Or I will just see failing GitHub status checks without a way to see the logs until someone will invite me to Buildkite?

Yes, the build and test results for each platform are publicly and anonymously available for everyone to see.

You don't need a Buildkite account for that. They are linked from the GitHub status checks for a pull request:

If you run into any trouble, please let me know. :)

Cheers,

Philipp

--

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

[Hyrum Wright]

Philipp & Jakob,

Could you point me to the email thread or design document where this was publicly discussed with the broader community before it was implemented?  I have no doubt it will be an improvement, but I'm interested in how the open source community was involved in making this decision.

Thanks,

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CA%2BAhZogFT9eO5zvsdwpsLzDjS3RFmy%3DU41Zc%2BwwxcPEKULBLVQ%40mail.gmail.com.

[Oleg Tsarev]

Hello,

I can not open  https://buildkite.com/bazel or test result for specific PR like this: https://buildkite.com/bazel/bazel-bazel-presubmit/builds/36

I see login page for both links.

I created account on buildkite, and after login I see "Page not found" for both links.

How to get access to build information?

[Philipp Wollermann]

Hi Hyrum,

I'm sorry - we didn't discuss this plan much with others (neither internal broader audience, nor external community) before we implemented it, due to time constraints and because it mostly concerns Bazel's supporting infrastructure and not the actual product. I agree that we should have sent at least a heads-up mail that we're working on this to bazel-discuss@! I'll make sure that our team will communicate plans better in the future and discuss them with the community before implementing them.

Here's what happened so far to give everyone interested some insight into this project:

Jakob, Florian and me are a newly established team ("Bazel EngProd") inside the larger Bazel team at Google. About one month ago, we were asked to fix Bazel's testing and release process as our first project. When we started, our internal presubmit CI system that should ensure that changes don't break Bazel, was very flaky and often prevented people from getting their code submitted. In addition, the sole author and maintainer of Bazel's Jenkins infrastructure transferred to another team, leaving behind a working CI infrastructure, but unfortunately without doing a handoff. We clearly had some work to do to get everything working reliably and performant.

Thus we took ownership of a couple of critical systems around Bazel's testing and release process (Bazel's testing framework, our internal presubmit CI, our external Jenkins, Bazel's infrastructure running on Google Cloud and the Mac Pros in our office) and investigated possible solutions. We went through the following ideas for Bazel's CI:

- Reverse engineer the existing Jenkins setup and try to build upon it. => State: Canceled after some work was done. I started with this beginning of this year and made ~50 commits, but it soon became clear, that this is not going to help us much. The problem wasn't so much our configuration, but Jenkins itself. Bad: Jenkins just seemed too slow and buggy and people hated the UX.

- Create a new Jenkins-powered CI system from scratch. => State: Not even tried. Bad: We believe that Jenkins's bad UX is inherently unfixable and that many problems on our CI actually arose from Jenkins bugs.

- CircleCI => State: Canceled after initial research. Good: Nice UI, Angular people use it and are happy, seemed to support our pipeline workflows, hosted solution reduces maintenance burden. Bad: Hosted workers with unknown performance (we prefer to run our own), no Windows support, unclear Mac support, Linux only in Docker (but we need to test against a variety of kernels, too).

- Jetbrains TeamCity => State: Canceled after proof-of-concept. Good: Felt like high-quality software, easy to setup, self-hosted workers. Excellent documentation. Potential cooperation with Jetbrains might have been possible. Performance was good. Bad: Too complex UI, Kotlin DSL (we didn't feel like learning yet another programming language just to configure our CI), somehow the UI and concepts didn't "click" with us nor a few people that we asked to have a look. Job logs of tests looked a bit weird due to TeamCity interleaving  Bazel's output and test logs in the same text. Would have required implementing plug-ins in Java to fully support Bazel and the build event stream.

- Google Container Builder with some additional UI and automation. => State: Canceled after research. Good: Managed orchestration and UI with self-hosted workers. Bad: Lack of Windows / Mac support, lack of documentation, Linux required use of Docker, UI unsatisfactory.

- Build our own CI. State => Canceled after initial design complete. Good: Does exactly what we want. Bad: Would take too long until we have something ready, didn't want colleagues and community to have to endure Jenkins for so long.

- Buildkite. State => Implemented. Good: Feature set is a perfect match for our requirements. Hosted and managed orchestration with self-hosted workers is what we want. Open-source agent. Free for open-source projects. Nice, minimalistic UI (extra points for the color and curses aware terminal in their real-time build logs). Easy to use. Bad: Better support for triggering downstream projects as part of a pipeline would have been nice, but we worked around this in our pipeline creation script. Anonymous read-only access still missing, but this is being worked on and we provide a link to Bazel's Build Results UI reports for each job, which are publicly accessible.

Hope this gives everyone some insight into our thoughts as we worked on this for the past month.

If you have questions, concerns or ideas on how to further improve our work, let's discuss!

Cheers,

Philipp

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CA%2BAhZogFT9eO5zvsdwpsLzDjS3RFmy%3DU41Zc%2BwwxcPEKULBLVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Austin Schuh]

Looking back 3 months later, are you happy with working with Bazel + Buildkite?  Is it a path you would recommend to others?

Austin

--

You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CA%2BAhZogFT9eO5zvsdwpsLzDjS3RFmy%3DU41Zc%2BwwxcPEKULBLVQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Reply all

To

Philipp Wollermann

[László Csomor]

+Jakob Buchgraber 

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CABsbf%3DG%2BWKu2QUvW%2B5Cum0K4FbJGZ-zeareqBMSCf0rc2FW3nw%40mail.gmail.com.

[Jakob Buchgraber]

Hi Austin,

we couldn't be more pleased with Buildkite. I would definitely recommend

going with it.

Best,

Jakob

Jakob Buchgraber

Software Engineer

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

On Thu, Jun 7, 2018 at 1:51 AM Austin Schuh <austin...@gmail.com> wrote:

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CABsbf%3DG%2BWKu2QUvW%2B5Cum0K4FbJGZ-zeareqBMSCf0rc2FW3nw%40mail.gmail.com.

[dagj...@mazemap.com]

Hi!

I would like invite access to the buildkite CI to test with bazel here :) Would that be possible?

Regards,

Dag

[Philipp Wollermann]

Hi Dag,

no problem. I sent you an invite. :)

Cheers,

Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/6e7addd5-890b-4735-9e5e-6b0689c13426%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Philipp Wollermann
Software Engineer
phi...@google.com

[gustavo...@gmail.com]

Hello!

I work with ppc64le and I'd like to build and make available a version for this architecture. I already have Bazel and Builkdite built.

Now, I'd like to also keep an CI on ppc64le. So can I get access too?

Thanks!

[Philipp Wollermann]

Hi,

that sounds cool! Unfortunately we can't connect third-party machines to our Buildkite CI at the moment, but maybe signing up yourself for a Buildkite account (or running a smaller CI using Jenkins) might work for you?

Cheers,

Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/9330f9ed-5dde-4887-a981-767596d3eff6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Clint Harrison]

Hi!

I'd also appreciate an invite to Buildkite!

Thanks,

Clint

[Gustavo Storti Salibi]

Hi, Philipp!

Thanks for the last reply.

We already have a CI on Jenkins. But we'd also like to contribute to Bazel with our CI.

How shall we proceed?

Thanks!

[looro...@gmail.com]

Hi Philipp,

I have been contributing on Windows support in Bazel. A few Github issues and commits like [0] link to BuildKite build log and I couldn't access them. Can I get an invitation?

[0]: https://github.com/bazelbuild/bazel/commit/e6a2f609375312c4fb78507340644b2f75bf6c9e

Best regards,

Rong Jie

[Philipp Wollermann]

Hi,

Sure - no problem! I sent you an invite now :)

Cheers,

Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/171be85e-67a7-4439-a256-704f8a1698b7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Philipp Wollermann]

Hi Gustavo, Rafael,

I really appreciate that you want to help us out with testing Bazel on that platform.

Until now it was difficult for us to accept contributed machines for CI due to Buildkite lacking agent isolation features that we’d need to ensure that testing could run on one set of machines, while release builds would only happen on special “trusted” machines.

The good news is that I found a workaround together with the Buildkite team and we can accept contributed machines in the future. :)

I’m currently at BazelCon, but am happy to look into this in the next week or so.

By the way, would it be possible for me and a colleague to get SSH root access to these PowerPC machines in order to set things up? Our setup scripts and notes are all open in the “continuous-integration” repo, but we’d like access to quickly fix things if we need to, e.g. when rolling out config changes to CI that require matching changes on the workers.

Cheers,

Philipp

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/914a175e-4c3a-4d84-b788-26703a6d259e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gustavo Storti Salibi]

Hi Philipp,

That's great!

Of course, we can provide SSH access for you. We'll talk about it. Then you tell me what features will be needed (such as disk space and operating system).

Many thanks for your reply,

Gustavo

[Gustavo Storti Salibi]

Hi Philipp,

Do we have any news about Buildkite?

Is there anything else we can do to help with CI on Power?

Thank you!

Em terça-feira, 27 de fevereiro de 2018 14:29:13 UTC-3, Philipp Wollermann escreveu: