Bazel's new CI has arrived. Jenkins is gone.
3,026 views
Skip to first unread message
Philipp Wollermann
Feb 27, 2018, 12:29:13 PM2/27/18
to bazel-dev, bazel-discuss
Hi,
Bazel's new CI has arrived! It is based on Buildkite (www.buildkite.com) and completely replaces our old Jenkins setup.
Cheers,
Philipp & Jakob
Bazel's new CI has arrived! It is based on Buildkite (www.buildkite.com) and completely replaces our old Jenkins setup.
It brings a modern, much more user-friendly UI and runs on completely redesigned and improved worker infrastructure, which makes everything very fast, reliable and gets rid of flaky test failures. This will allow us to test Bazel more often and consistently against downstream projects to ensure that we never accidentally break your use case with a new release. If you'd like to get your project tested on our CI, reach out to me and we can see how we can make it happen.
You can access the new CI by simply logging into https://buildkite.com/bazel. Currently, this requires creating an account via an invite, which we'll happily provide for all rules maintainers and interested contributors. :) Just send me an e-mail, and I'll send you an invite. We're also working with Buildkite to enable public anonymous read-only access to the CI.
We also integrated a shiny new test results viewer, which is active for GitHub pull requests - this should make it easier to see if tests failed.
Check out an example report here: https://source.cloud.google.com/results/invocations/799a7068-7597-4cc3-b69a-3978b7edec92/targets
You can find some first documentation and screenshots for the new setup here:
We're going to add a rules maintainer's and contributor's guide, too and hope to have it ready in the next days.
Maintainers: In the meantime, if you have any questions, complaints, stuff doesn't work for your projects in the github.com/bazelbuild org... let me know and we'll figure it out together.
Thanks to everyone on the Bazel team and community who helped with this undertaking!
And also a big thanks to the awesome people at Buildkite, who not only created a great CI platform, but make it available for free for open-source projects.
Cheers,
Philipp & Jakob
Fedor Korotkov
Feb 27, 2018, 12:50:41 PM2/27/18
to Philipp Wollermann, bazel-dev, bazel-discuss
Hey Philipp,
I have a question about workflow for contributors. Seems there is no way to run a CI for my fork before creating an actual PR and waiting for an approval from a project member. Do I understand it correctly?
https://buildkite.com/bazel doesn't look like to be publicly available as you mentioned. But will the build page for my PR be available publicly? Or I will just see failing GitHub status checks without a way to see the logs until someone will invite me to Buildkite?
Best,
Fedor
--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CA%2BAhZogFT9eO5zvsdwpsLzDjS3RFmy%3DU41Zc%2BwwxcPEKULBLVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Philipp Wollermann
Feb 27, 2018, 1:02:46 PM2/27/18
to Fedor Korotkov, bazel-dev, bazel-discuss
Hi Fedor,
On Tue, Feb 27, 2018 at 6:50 PM Fedor Korotkov <fedor.k...@gmail.com> wrote:
I have a question about workflow for contributors. Seems there is no way to run a CI for my fork before creating an actual PR and waiting for an approval from a project member. Do I understand it correctly?
Yes - we would love to simply allow this and are thinking about how it could be made possible, but it's also clearly a security issue: If we allow this, anyone with a GitHub account could run arbitrary code on our CI machines. :|
Note that this is not a change to our older system based on Jenkins - even then project members had to write the magic "Jenkins, test this please" phrase in order to trigger a CI run.
https://buildkite.com/bazel doesn't look like to be publicly available as you mentioned. But will the build page for my PR be available publicly? Or I will just see failing GitHub status checks without a way to see the logs until someone will invite me to Buildkite?
Yes, the build and test results for each platform are publicly and anonymously available for everyone to see.
You don't need a Buildkite account for that. They are linked from the GitHub status checks for a pull request:
If you run into any trouble, please let me know. :)
Cheers,
Philipp
Google Germany GmbH
Erika-Mann-Straße 33
80636 München
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Erika-Mann-Straße 33
80636 München
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Hyrum Wright
Feb 27, 2018, 1:26:03 PM2/27/18
to Philipp Wollermann, bazel-dev, bazel-discuss
Philipp & Jakob,
Could you point me to the email thread or design document where this was publicly discussed with the broader community before it was implemented? I have no doubt it will be an improvement, but I'm interested in how the open source community was involved in making this decision.
Thanks,
-Hyrum
--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CA%2BAhZogFT9eO5zvsdwpsLzDjS3RFmy%3DU41Zc%2BwwxcPEKULBLVQ%40mail.gmail.com.
Philipp Wollermann
Mar 4, 2018, 8:40:18 AM3/4/18
to hwr...@duolingo.com, bazel-dev, bazel-discuss
Hi Hyrum,
I'm sorry - we didn't discuss this plan much with others (neither internal broader audience, nor external community) before we implemented it, due to time constraints and because it mostly concerns Bazel's supporting infrastructure and not the actual product. I agree that we should have sent at least a heads-up mail that we're working on this to bazel-discuss@! I'll make sure that our team will communicate plans better in the future and discuss them with the community before implementing them.
Here's what happened so far to give everyone interested some insight into this project:
Jakob, Florian and me are a newly established team ("Bazel EngProd") inside the larger Bazel team at Google. About one month ago, we were asked to fix Bazel's testing and release process as our first project. When we started, our internal presubmit CI system that should ensure that changes don't break Bazel, was very flaky and often prevented people from getting their code submitted. In addition, the sole author and maintainer of Bazel's Jenkins infrastructure transferred to another team, leaving behind a working CI infrastructure, but unfortunately without doing a handoff. We clearly had some work to do to get everything working reliably and performant.
Thus we took ownership of a couple of critical systems around Bazel's testing and release process (Bazel's testing framework, our internal presubmit CI, our external Jenkins, Bazel's infrastructure running on Google Cloud and the Mac Pros in our office) and investigated possible solutions. We went through the following ideas for Bazel's CI:
- Reverse engineer the existing Jenkins setup and try to build upon it. => State: Canceled after some work was done. I started with this beginning of this year and made ~50 commits, but it soon became clear, that this is not going to help us much. The problem wasn't so much our configuration, but Jenkins itself. Bad: Jenkins just seemed too slow and buggy and people hated the UX.
- Create a new Jenkins-powered CI system from scratch. => State: Not even tried. Bad: We believe that Jenkins's bad UX is inherently unfixable and that many problems on our CI actually arose from Jenkins bugs.
- CircleCI => State: Canceled after initial research. Good: Nice UI, Angular people use it and are happy, seemed to support our pipeline workflows, hosted solution reduces maintenance burden. Bad: Hosted workers with unknown performance (we prefer to run our own), no Windows support, unclear Mac support, Linux only in Docker (but we need to test against a variety of kernels, too).
- Jetbrains TeamCity => State: Canceled after proof-of-concept. Good: Felt like high-quality software, easy to setup, self-hosted workers. Excellent documentation. Potential cooperation with Jetbrains might have been possible. Performance was good. Bad: Too complex UI, Kotlin DSL (we didn't feel like learning yet another programming language just to configure our CI), somehow the UI and concepts didn't "click" with us nor a few people that we asked to have a look. Job logs of tests looked a bit weird due to TeamCity interleaving Bazel's output and test logs in the same text. Would have required implementing plug-ins in Java to fully support Bazel and the build event stream.
- Google Container Builder with some additional UI and automation. => State: Canceled after research. Good: Managed orchestration and UI with self-hosted workers. Bad: Lack of Windows / Mac support, lack of documentation, Linux required use of Docker, UI unsatisfactory.
- Build our own CI. State => Canceled after initial design complete. Good: Does exactly what we want. Bad: Would take too long until we have something ready, didn't want colleagues and community to have to endure Jenkins for so long.- Buildkite. State => Implemented. Good: Feature set is a perfect match for our requirements. Hosted and managed orchestration with self-hosted workers is what we want. Open-source agent. Free for open-source projects. Nice, minimalistic UI (extra points for the color and curses aware terminal in their real-time build logs). Easy to use. Bad: Better support for triggering downstream projects as part of a pipeline would have been nice, but we worked around this in our pipeline creation script. Anonymous read-only access still missing, but this is being worked on and we provide a link to Bazel's Build Results UI reports for each job, which are publicly accessible.
Hope this gives everyone some insight into our thoughts as we worked on this for the past month.
If you have questions, concerns or ideas on how to further improve our work, let's discuss!
Cheers,
Philipp
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CA%2BAhZogFT9eO5zvsdwpsLzDjS3RFmy%3DU41Zc%2BwwxcPEKULBLVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
itt...@wix.com
Mar 14, 2018, 3:10:03 AM3/14/18
to bazel-discuss
Hi,
Is there a way for a rule maintainer to "verify" a PR (and so kick off the actual build) via a PR comment (like we had "test this")?
I now have to click the PR link and then click the buildkite link even when I know the contents of the PR. Additionally I think some maintainers might be confused by the fact that before you verify the buildkite build seems to pass (though without listing of environments)
Is there a way for a rule maintainer to "verify" a PR (and so kick off the actual build) via a PR comment (like we had "test this")?
I now have to click the PR link and then click the buildkite link even when I know the contents of the PR. Additionally I think some maintainers might be confused by the fact that before you verify the buildkite build seems to pass (though without listing of environments)
On Sunday, March 4, 2018 at 3:40:18 PM UTC+2, Philipp Wollermann wrote:
> Hi Hyrum,
>
>
> I'm sorry - we didn't discuss this plan much with others (neither internal broader audience, nor external community) before we implemented it, due to time constraints and because it mostly concerns Bazel's supporting infrastructure and not the actual product. I agree that we should have sent at least a heads-up mail that we're working on this to bazel-discuss@! I'll make sure that our team will communicate plans better in the future and discuss them with the community before implementing them.
>
>
> Here's what happened so far to give everyone interested some insight into this project:
>
>
> Jakob, Florian and me are a newly established team ("Bazel EngProd") inside the larger Bazel team at Google. About one month ago, we were asked to fix Bazel's testing and release process as our first project. When we started, our internal presubmit CI system that should ensure that changes don't break Bazel, was very flaky and often prevented people from getting their code submitted. In addition, the sole author and maintainer of Bazel's Jenkins infrastructure transferred to another team, leaving behind a working CI infrastructure, but unfortunately without doing a handoff. We clearly had some work to do to get everything working reliably and performant.
>
>
>
> Thus we took ownership of a couple of critical systems around Bazel's testing and release process (Bazel's testing framework, our internal presubmit CI, our external Jenkins, Bazel's infrastructure running on Google Cloud and the Mac Pros in our office) and investigated possible solutions. We went through the following ideas for Bazel's CI:
>
>
> - Reverse engineer the existing Jenkins setup and try to build upon it. => State: Canceled after some work was done. I started with this beginning of this year and made ~50 commits, but it soon became clear, that this is not going to help us much. The problem wasn't so much our configuration, but Jenkins itself. Bad: Jenkins just seemed too slow and buggy and people hated the UX.
> - Create a new Jenkins-powered CI system from scratch. => State: Not even tried. Bad: We believe that Jenkins's bad UX is inherently unfixable and that many problems on our CI actually arose from Jenkins bugs.
> - CircleCI => State: Canceled after initial research. Good: Nice UI, Angular people use it and are happy, seemed to support our pipeline workflows, hosted solution reduces maintenance burden. Bad: Hosted workers with unknown performance (we prefer to run our own), no Windows support, unclear Mac support, Linux only in Docker (but we need to test against a variety of kernels, too).
> - Jetbrains TeamCity => State: Canceled after proof-of-concept. Good: Felt like high-quality software, easy to setup, self-hosted workers. Excellent documentation. Potential cooperation with Jetbrains might have been possible. Performance was good. Bad: Too complex UI, Kotlin DSL (we didn't feel like learning yet another programming language just to configure our CI), somehow the UI and concepts didn't "click" with us nor a few people that we asked to have a look. Job logs of tests looked a bit weird due to TeamCity interleaving Bazel's output and test logs in the same text. Would have required implementing plug-ins in Java to fully support Bazel and the build event stream.
> - Google Container Builder with some additional UI and automation. => State: Canceled after research. Good: Managed orchestration and UI with self-hosted workers. Bad: Lack of Windows / Mac support, lack of documentation, Linux required use of Docker, UI unsatisfactory.- Build our own CI. State => Canceled after initial design complete. Good: Does exactly what we want. Bad: Would take too long until we have something ready, didn't want colleagues and community to have to endure Jenkins for so long.
> Hi Hyrum,
>
>
> I'm sorry - we didn't discuss this plan much with others (neither internal broader audience, nor external community) before we implemented it, due to time constraints and because it mostly concerns Bazel's supporting infrastructure and not the actual product. I agree that we should have sent at least a heads-up mail that we're working on this to bazel-discuss@! I'll make sure that our team will communicate plans better in the future and discuss them with the community before implementing them.
>
>
> Here's what happened so far to give everyone interested some insight into this project:
>
>
> Jakob, Florian and me are a newly established team ("Bazel EngProd") inside the larger Bazel team at Google. About one month ago, we were asked to fix Bazel's testing and release process as our first project. When we started, our internal presubmit CI system that should ensure that changes don't break Bazel, was very flaky and often prevented people from getting their code submitted. In addition, the sole author and maintainer of Bazel's Jenkins infrastructure transferred to another team, leaving behind a working CI infrastructure, but unfortunately without doing a handoff. We clearly had some work to do to get everything working reliably and performant.
>
>
>
> Thus we took ownership of a couple of critical systems around Bazel's testing and release process (Bazel's testing framework, our internal presubmit CI, our external Jenkins, Bazel's infrastructure running on Google Cloud and the Mac Pros in our office) and investigated possible solutions. We went through the following ideas for Bazel's CI:
>
>
> - Reverse engineer the existing Jenkins setup and try to build upon it. => State: Canceled after some work was done. I started with this beginning of this year and made ~50 commits, but it soon became clear, that this is not going to help us much. The problem wasn't so much our configuration, but Jenkins itself. Bad: Jenkins just seemed too slow and buggy and people hated the UX.
> - Create a new Jenkins-powered CI system from scratch. => State: Not even tried. Bad: We believe that Jenkins's bad UX is inherently unfixable and that many problems on our CI actually arose from Jenkins bugs.
> - CircleCI => State: Canceled after initial research. Good: Nice UI, Angular people use it and are happy, seemed to support our pipeline workflows, hosted solution reduces maintenance burden. Bad: Hosted workers with unknown performance (we prefer to run our own), no Windows support, unclear Mac support, Linux only in Docker (but we need to test against a variety of kernels, too).
> - Jetbrains TeamCity => State: Canceled after proof-of-concept. Good: Felt like high-quality software, easy to setup, self-hosted workers. Excellent documentation. Potential cooperation with Jetbrains might have been possible. Performance was good. Bad: Too complex UI, Kotlin DSL (we didn't feel like learning yet another programming language just to configure our CI), somehow the UI and concepts didn't "click" with us nor a few people that we asked to have a look. Job logs of tests looked a bit weird due to TeamCity interleaving Bazel's output and test logs in the same text. Would have required implementing plug-ins in Java to fully support Bazel and the build event stream.
Jakob Buchgraber
Mar 14, 2018, 5:37:20 AM3/14/18
to itt...@wix.com, bazel-discuss
Hi,
yes we would like to have that too :-). We have an open bug for this: https://github.com/bazelbuild/continuous-integration/issues/236. Could you please leave a comment on there that there you would also like to have this feature?
Thanks,
Jakob
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/22e30490-38ab-476c-8017-7ce605dca31b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Jakob Buchgraber
Software Engineer
Google Germany GmbH
Erika-Mann-Straße 33
80636 München
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.
This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.
Austin Schuh
Jun 6, 2018, 7:51:48 PM6/6/18
to Philipp Wollermann, bazel-dev, bazel-discuss
Looking back 3 months later, are you happy with working with Bazel + Buildkite? Is it a path you would recommend to others?
Austin
--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CA%2BAhZogFT9eO5zvsdwpsLzDjS3RFmy%3DU41Zc%2BwwxcPEKULBLVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
László Csomor
Jun 11, 2018, 3:04:23 AM6/11/18
to Austin Schuh, Jakob Buchgraber, Philipp Wollermann, bazel-dev, bazel-discuss
+Jakob Buchgraber
--
László Csomor | Software Engineer | laszlo...@google.com
Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
--
László Csomor | Software Engineer | laszlo...@google.com
Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CABsbf%3DG%2BWKu2QUvW%2B5Cum0K4FbJGZ-zeareqBMSCf0rc2FW3nw%40mail.gmail.com.
Jakob Buchgraber
Jun 25, 2018, 10:50:51 AM6/25/18
to austin...@gmail.com, Philipp Wollermann, bazel-dev, bazel-discuss
Hi Austin,
we couldn't be more pleased with Buildkite. I would definitely recommend
going with it.
Best,
Jakob
Jakob Buchgraber
Software Engineer
Google Germany GmbH
Erika-Mann-Straße 33
80636 München
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.
This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.
On Thu, Jun 7, 2018 at 1:51 AM Austin Schuh <austin...@gmail.com> wrote:
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CABsbf%3DG%2BWKu2QUvW%2B5Cum0K4FbJGZ-zeareqBMSCf0rc2FW3nw%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages