[llvm-dev] Contributing Bazel BUILD files similar to gn

219 views
Skip to first unread message

Geoffrey Martin-Noble via llvm-dev

unread,
Oct 28, 2020, 7:18:48 PM10/28/20
to llvm...@lists.llvm.org, Mehdi Amini, Stella Laurenzo, Tres Popp, Thomas Joerg

Hi all,

tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a side-directory in the monorepo, similar to the gn build.

Some of us have been working on open-source Bazel BUILD files for the LLVM Project. You may have seen us hanging out in the #build-systems discord channel. As you may know, Google uses Bazel internally and has maintained a Bazel BUILD of LLVM for years. Especially with the introduction of MLIR, we've got more and more OSS projects with a Bazel BUILD depending on LLVM (e.g. IREE and TensorFlow). We're also not the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM that they've borrowed from TF. Each of these projects has to jump through some weird hoops to keep their version of the Bazel BUILD files in sync with the code, which requires some fragile combination of scripts and human intervention. Instead, we'd like to move general-purpose Bazel BUILD files into the LLVM Project monorepo. We expect to follow the model of the GN build where these will be maintained by interested contributors rather than expecting the general community to maintain them.

To facilitate and test this we've been developing a standalone repository that just has the Bazel BUILD files. It symlinks together the directory trees on top of a submodule as we would need in the monorepo to to avoid in-tree BUILD files. The configuration is at https://github.com/google/llvm-bazel. We now have those in a good place and think they would be useful upstream.


# Details

## What

Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review) subprojects, potentially expanding to others, as needed. Basically everything currently at https://github.com/google/llvm-bazel.


## Where

In https://github.com/google/llvm-bazel the BUILD files live in a single directory tree matching the structure of the overall llvm-project directory. For users, @llvm-project is a single Bazel repository that includes both LLVM and MLIR subprojects. To maintain this structure, we would probably want to put a `bazel` directory in the monorepo's utils directory, which currently only contains a directory for arcanist. This is different from gn, which is under the LLVM subproject's utils directory. We could similarly put the Bazel BUILD files under llvm/utils/bazel but have them be for the entire llvm project (the subsets that are supported). This seems like an odd structure to me, but I know that the CMake build for LLVM also builds the other subprojects, so maybe this would be preferable.

Alternatively we could split each subproject into a separate Bazel repository and put the Bazel build files under each subproject. I think this fragments the configuration of the BUILD without much benefit.


## Configurations

We currently have configurations for Linux GCC and Clang, MacOS GCC and Clang, and Windows MSVC. Support for other configurations can be added as-desired, but supporting all possible LLVM build configurations is not the goal.


## Support

Support would be similar to the gn build. Contributors could optionally update the Bazel BUILD files as part of their patches, but would be under no obligation to do so.


## Preserving History

I don't *think* the history of llvm-bazel is interesting enough to try to merge it into the monorepo and I was planning to submit this as a single patch, but please let me know if you disagree.


## Benefits to the community

  • Projects that depend on LLVM and use the Bazel build system can avoid duplicating fragile effort. We'll spend more time contributing to LLVM instead :-D

  • Bazel is stricter than CMake in many ways (e.g. it requires that even header dependencies be declared) and can catch layering issues very easily. There's even an optional layering_check feature we could turn on if its use would benefit the community. (though currently the existing problematic layering makes it a burden to maintain on our own). Even without that additional check, as I've been keeping the Bazel build green, I've found and fixed a number of layering issues in the past couple weeks (e.g. https://reviews.llvm.org/rGb49787df9a and https://reviews.llvm.org/rGc17ae2916c).


Here's a patch adding the Bazel build system. It's basically just `cp -r llvm-bazel/llvm-bazel llvm-project/utils/bazel`.

Tom Stellard via llvm-dev

unread,
Oct 29, 2020, 11:23:20 AM10/29/20
to Geoffrey Martin-Noble, llvm...@lists.llvm.org, Mehdi Amini, Stella Laurenzo, Thomas Joerg, Tres Popp
On 10/28/20 7:18 PM, Geoffrey Martin-Noble via llvm-dev wrote:
> Hi all,
>
> tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a
> side-directory in the monorepo, similar to the gn build.
>
> Some of us have been working on open-source Bazel BUILD files for the
> LLVM Project. You may have seen us hanging out in the #build-systems
> discord channel. As you may know, Google uses Bazel internally and has
> maintained a Bazel BUILD of LLVM for years. Especially with the
> introduction of MLIR, we've got more and more OSS projects with a Bazel
> BUILD depending on LLVM (e.g. IREE <https://github.com/google/iree>and
> TensorFlow <https://github.com/tensorflow/tensorflow>). We're also not
> the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM
> that they've borrowed from TF
> <https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD>.
> Each of these projects has to jump through some weird hoops to keep
> their version of the Bazel BUILD files in sync with the code, which
> requires some fragile combination of scripts and human intervention.
> Instead, we'd like to move general-purpose Bazel BUILD files into the
> LLVM Project monorepo. We expect to follow the model of the GN build
> where these will be maintained by interested contributors rather than
> expecting the general community to maintain them.
>
> To facilitate and test this we've been developing a standalone
> repository that just has the Bazel BUILD files. It symlinks together the
> directory trees on top of a submodule as we would need in the monorepo
> to to avoid in-tree BUILD files. The configuration is at
> https://github.com/google/llvm-bazel. We now have those in a good place
> and think they would be useful upstream.
>

Can you explain some of the benefits to using Bazel instead of CMake?

I'm a little concerned about having two 'unsupported' buildsystems
living in tree, and I'm not sure what would stop us from continuing to
add more. I would feel better if we had a set of guidelines to define
the criteria for adding a new buildsytem and also criteria for when we
can remove them.

Would you be able to amend this proposal to include some general
guidelines for adding/removing new buildsystems, so that we can discuss
that too?

Thanks,
Tom


>
> # Details
>
> ## What
>
> Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review

> <https://github.com/google/llvm-bazel/pull/72>) subprojects, potentially

> expanding to others, as needed. Basically everything currently at
> https://github.com/google/llvm-bazel.
>
>
> ## Where
>

> In https://github.com/google/llvm-bazelthe BUILD files live in a single

> directory tree matching the structure of the overall llvm-project
> directory. For users, @llvm-project is a single Bazel repository

> <https://docs.bazel.build/versions/master/build-ref.html#repositories>that

> includes both LLVM and MLIR subprojects. To maintain this structure, we
> would probably want to put a `bazel` directory in the monorepo's utils

> directory <https://github.com/llvm/llvm-project/tree/master/utils>,

> which currently only contains a directory for arcanist. This is
> different from gn, which is under the LLVM subproject's utils directory

> <https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn>. We

> could similarly put the Bazel BUILD files under llvm/utils/bazel but
> have them be for the entire llvm project (the subsets that are
> supported). This seems like an odd structure to me, but I know that the
> CMake build for LLVM also builds the other subprojects

> <https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41>,

> so maybe this would be preferable.
>
> Alternatively we could split each subproject into a separate Bazel
> repository and put the Bazel build files under each subproject. I think
> this fragments the configuration of the BUILD without much benefit.
>
>
> ## Configurations
>
> We currently have configurations for Linux GCC and Clang, MacOS GCC and
> Clang, and Windows MSVC. Support for other configurations can be added
> as-desired, but supporting all possible LLVM build configurations is not
> the goal.
>
>
> ## Support
>
> Support would be similar to the gn build. Contributors could optionally
> update the Bazel BUILD files as part of their patches, but would be
> under no obligation to do so.
>
>
> ## Preserving History
>
> I don't *think* the history of llvm-bazel is interesting enough to try
> to merge it into the monorepo and I was planning to submit this as a
> single patch, but please let me know if you disagree.
>
>
> ## Benefits to the community
>

> *


>
> Projects that depend on LLVM and use the Bazel build system can
> avoid duplicating fragile effort. We'll spend more time contributing
> to LLVM instead :-D
>

> *


>
> Bazel is stricter than CMake in many ways (e.g. it requires that
> even header dependencies be declared) and can catch layering issues
> very easily. There's even an optional layering_check feature we
> could turn on if its use would benefit the community. (though
> currently the existing problematic layering makes it a burden to
> maintain on our own). Even without that additional check, as I've
> been keeping the Bazel build green, I've found and fixed a number of
> layering issues in the past couple weeks (e.g.
> https://reviews.llvm.org/rGb49787df9a

> <https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d>and
> https://reviews.llvm.org/rGc17ae2916c
> <https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a>).
>
>
> Here's a patch <https://reviews.llvm.org/D90352>adding the Bazel build

> system. It's basically just `cp -r llvm-bazel/llvm-bazel
> llvm-project/utils/bazel`.
>

> _______________________________________________
> LLVM Developers mailing list
> llvm...@lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Renato Golin via llvm-dev

unread,
Oct 29, 2020, 12:06:31 PM10/29/20
to tste...@redhat.com, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
On Thu, 29 Oct 2020 at 15:23, Tom Stellard via llvm-dev <llvm...@lists.llvm.org> wrote:
I'm a little concerned about having two 'unsupported' buildsystems
living in tree, and I'm not sure what would stop us from continuing to
add more.  I would feel better if we had a set of guidelines to define
the criteria for adding a new buildsytem and also criteria for when we
can remove them.

I have used Bazel and it doesn't seem to map well to CMake. It seems to be in between CMake and Ninja with a lot of hard-coded dependencies that are cumbersome to keep updating. I'm by no means an expert, and I could very well be wrong, but supporting more than one build system is not trivial (remember the autoconf days?).

For example, when trying to implement the same logic on both will not be trivial. So, whenever we want to add some functionality or improve how we build LLVM with one system, we'll have to do so in multiple build systems that do not easily match each other. If we don't try to match functionality, we'll segregate the community, because people will be able to do X on build system A but not B, and the similar features cluster together and then we have essentially two projects built from the same source code.

Testing this, or worse, trying to fix a buildbot that is built with Bazel (and having to install Java JDK and all its dependencies) on potentially a hardware that you do not have access to, will be a nightmare to debug. The nature of post-commit testing, revert and review of LLVM will not make that simpler. Unless we treat the Bazel build as "not our problem" (which defeats the point of having it?).

To make matters worse, our CMake files are not simple, and do not do all of the things we want them to do in the way we understand completely. There is a lot of kludge that we carry and with that comes in two categories: the things that we hate and would love to fix, and the things that are fixes that we have no idea are there. The former are the reasons why people want to start a new build system, the latter is why they soon realise that was a mistake (insert XKCD joke here).

If the Bazel files can be completely ignored, then it's just more clutter. But if other projects start to use more different build systems and we start packing them all in LLVM, then we'll have a hard time knowing what we build how. I can't really see this scaling.

Two-cents worth.
--renato

Stefan Teleman via llvm-dev

unread,
Oct 29, 2020, 12:55:10 PM10/29/20
to LLVM Dev
On Thu, Oct 29, 2020 at 11:23 AM Tom Stellard via llvm-dev
<llvm...@lists.llvm.org> wrote:

> > tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a
> > side-directory in the monorepo, similar to the gn build.

> Can you explain some of the benefits to using Bazel instead of CMake?

I can, and I will be very brief: None.

--
Stefan Teleman
stefan....@gmail.com

David Blaikie via llvm-dev

unread,
Oct 29, 2020, 3:16:26 PM10/29/20
to Renato Golin, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg

I /believe/ the idea is that, like gn, there are folks maintaining these build systems out of tree anyway - and having them in tree makes it easier to coordinate that effort, with the express intent of not burdening the general community with their upkeep (like gn currently - the idea is that there's no burden on developers to update gn build files (& consequently bazel build files)).

So far the gn build inclusion seems to have gone OK, I think? So maybe the Bazel thing would be similar?

Sounds like the work is being done out of tree anyway, but having it in-tree makes it a bit easier to coordinate interested parties, while not adversely affecting unrelated parties, I think?

Though I'm not sure what the tradeoff/cost of this is compared to having a separate project holding the build files, with LLVM as a git submodule. Not knowing a lot about it, that /sounds/ like it gets most of the benefits/not sure what the costs are?

- Dave

David Blaikie via llvm-dev

unread,
Oct 29, 2020, 3:17:21 PM10/29/20
to Stefan Teleman, LLVM Dev
On Thu, Oct 29, 2020 at 9:55 AM Stefan Teleman via llvm-dev <llvm...@lists.llvm.org> wrote:
On Thu, Oct 29, 2020 at 11:23 AM Tom Stellard via llvm-dev
<llvm...@lists.llvm.org> wrote:

> > tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a
> > side-directory in the monorepo, similar to the gn build.

> Can you explain some of the benefits to using Bazel instead of CMake?

I can, and I will be very brief: None.

This is a fairly unhelpful email - clearly folks using Bazel derive some benefit/have chosen some tradeoff compared to CMake. Doesn't have to be the thing you want, but it's pretty unhelpful to dismiss/diminish the needs of others like this.

- Dave

Chris Tetreault via llvm-dev

unread,
Oct 29, 2020, 3:28:34 PM10/29/20
to David Blaikie, Renato Golin, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg

Could not a public repository for the Bazel build system be created, that has a submodule for the llvm monorepo? Users of the Bazel build system could checkout the Bazel build repo and do the submodule init, and this public repo could be used for collaboration.

 

Thanks,

   Christopher Tetreault

Chris Tetreault via llvm-dev

unread,
Oct 29, 2020, 3:29:20 PM10/29/20
to Renato Golin, tste...@redhat.com, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg

I think Renato has articulated quite well some concerns I have about this but was unable to express. I would very much prefer if we just focus on using CMake effectively.

 

Thanks,

   Christopher Tetreault

 

From: llvm-dev <llvm-dev...@lists.llvm.org> On Behalf Of Renato Golin via llvm-dev
Sent: Thursday, October 29, 2020 9:06 AM
To: tste...@redhat.com
Cc: Mehdi Amini <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble <gc...@google.com>; Thomas Joerg <tjo...@google.com>
Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn

 

On Thu, 29 Oct 2020 at 15:23, Tom Stellard via llvm-dev <llvm...@lists.llvm.org> wrote:

I'm a little concerned about having two 'unsupported' buildsystems
living in tree, and I'm not sure what would stop us from continuing to
add more.  I would feel better if we had a set of guidelines to define
the criteria for adding a new buildsytem and also criteria for when we
can remove them.

 

I have used Bazel and it doesn't seem to map well to CMake. It seems to be in between CMake and Ninja with a lot of hard-coded dependencies that are cumbersome to keep updating. I'm by no means an expert, and I could very well be wrong, but supporting more than one build system is not trivial (remember the autoconf days?).

 

For example, when trying to implement the same logic on both will not be trivial. So, whenever we want to add some functionality or improve how we build LLVM with one system, we'll have to do so in multiple build systems that do not easily match each other. If we don't try to match functionality, we'll segregate the community, because people will be able to do X on build system A but not B, and the similar features cluster together and then we have essentially two projects built from the same source code.

 

Testing this, or worse, trying to fix a buildbot that is built with Bazel (and having to install Java JDK and all its dependencies) on potentially a hardware that you do not have access to, will be a nightmare to debug. The nature of post-commit testing, revert and review of LLVM will not make that simpler. Unless we treat the Bazel build as "not our problem" (which defeats the point of having it?).

 

To make matters worse, our CMake files are not simple, and do not do all of the things we want them to do in the way we understand completely. There is a lot of kludge that we carry and with that comes in two categories: the things that we hate and would love to fix, and the things that are fixes that we have no idea are there. The former are the reasons why people want to start a new build system, the latter is why they soon realise that was a mistake (insert XKCD joke here).

 

If the Bazel files can be completely ignored, then it's just more clutter. But if other projects start to use more different build systems and we start packing them all in LLVM, then we'll have a hard time knowing what we build how. I can't really see this scaling.

 

Two-cents worth.

--renato

Stefan Teleman via llvm-dev

unread,
Oct 29, 2020, 3:41:37 PM10/29/20
to LLVM Dev
> This is a fairly unhelpful email - clearly folks using Bazel derive some benefit/have chosen some tradeoff compared to CMake. Doesn't have to be the thing you want, but it's pretty unhelpful to dismiss/diminish the needs of others like this.

I did not see a rationale for the Bazel proposal, outlining its
benefits over CMake.

Speaking with direct experience with Bazel - Tensorflow - I cannot
think of a single reason why it would/should be considered "better"
over the current CMake.

Everyone has their own favorite build system. That is nice, but it is
not enough of a reason to propose adding it.

I would also like to become informed as to what particular
needs/shortcomings/defects are addressed by Bazel, that are lacking in
/ cannot be addressed by CMake.

Thanks.

Renato Golin via llvm-dev

unread,
Oct 29, 2020, 3:49:33 PM10/29/20
to David Blaikie, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
On Thu, 29 Oct 2020 at 19:16, David Blaikie <dbla...@gmail.com> wrote:
I /believe/ the idea is that, like gn, there are folks maintaining these build systems out of tree anyway - and having them in tree makes it easier to coordinate that effort, with the express intent of not burdening the general community with their upkeep (like gn currently - the idea is that there's no burden on developers to update gn build files (& consequently bazel build files)).

Perhaps the initial assumption about my concerns weren't well articulated. 

I get that those files would be "additional" and other developers won't need to care much about them. 

But what happens when people join the project with experience in Bazel and, instead of building pure LLVM with CMake, they start using Bazel for everything, just because they're used to it?

Bazel is big enough (at least inside Google) that the probability of that happening is not trivial.

What if they create sub-projects that can only build with Bazel? Do we refuse inclusion? But don't we have Bazel files already?

One big example is Android. They used to build LLVM in a very different way, and the inclusion of run-time library files was completely different. So different it was not possible to merge some changes they had (128 bit maths IIRC) because of the amount of work required.

My point is that adding another build system will not necessarily improve the chances of external people contributing to LLVM if they use those build systems. It may very well *reduce* those chances.

Once we get to the point where Bazel support is "complete" enough, and enough other projects that use LLVM use Bazel (I assume many internal Google projects), the problem I describe above is bound to happen sooner or later.

Personally, I'm happy to ignore Bazel and continue using CMake. But I just wanted to make clear that in the past, using a different build system did not increase the chances of contribution, so that's not a given in this case either.

cheers,
--renato

Renato Golin via llvm-dev

unread,
Oct 29, 2020, 3:57:16 PM10/29/20
to Stefan Teleman, LLVM Dev
On Thu, 29 Oct 2020 at 19:41, Stefan Teleman via llvm-dev <llvm...@lists.llvm.org> wrote:
Everyone has their own favorite build system. That is nice, but it is
not enough of a reason to propose adding it.

I would also like to become informed as to what particular
needs/shortcomings/defects are addressed by Bazel, that are lacking in
/ cannot be addressed by CMake.

I don't think they're proposing adding Bazel as a new core build system nor replacing CMake. 

This is just about adding Bazel files to the project so external projects (like Tensorflow) can build LLVM more easily. Also, so that all Bazel-based builds that use LLVM can share the same files without having to reproduce them in every sub-project.

It is a worthy goal in itself, but I think this speaks very loudly to how weird it is to build LLVM. We had to add some horrible hacks on our project because exporting CMake and TD files to our project (in order to use MLIR) was super weird.

So, perhaps there's an underlying goal there to finally fix the LLVM build "once and for all", and export libraries, headers, and meta-files in an orderly fashion, so that wrapping projects don't need to care what build system LLVM is in.

But I'm not a build-system specialist, so I can't even begin to fathom how that would work. I'm not even sure that's possible, so... lots of salt.

In the meantime, having those files wouldn't be the end of the world. But I fear that once we add, they'll stay there forever, and will lead to people ignoring CMake and segregating the project.

cheers,
--renato

Stefan Teleman via llvm-dev

unread,
Oct 29, 2020, 4:03:16 PM10/29/20
to Renato Golin, LLVM Dev
On Thu, Oct 29, 2020 at 3:57 PM Renato Golin <reng...@gmail.com> wrote:

>
> In the meantime, having those files wouldn't be the end of the world. But I fear that once we add, they'll stay there forever, and will lead to people ignoring CMake and segregating the project.

Yes that is my main concern as well.

Build systems for complex projects are ... messy. I believe that what
we have right now - with CMake - works quite well. And I am perfectly
aware of the insane amount of work that has gotten into making LLVM
quite easy to build.

Mehdi AMINI via llvm-dev

unread,
Oct 29, 2020, 4:05:22 PM10/29/20
to Tom Stellard, Mehdi Amini, llvm-dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
(following up on the discussion on Discord)

The question can be taken with multiple angles:

1) The benefit for the LLVM project to switch from CMake to Bazel: it isn't clear even that it would be practical/feasible or serve all the users, this is not the proposal here. Nothing changes for CMake.
2) The benefit for an LLVM developer to use Bazel instead of CMake: I think it is minor, unless you have access to a remote build farm in your environment maybe?
3) The benefit for a project that is willing to use LLVM to use Bazel instead of CMake: this is a tricky topic because there are many variables. The Bazel website is a good starting point I think: https://bazel.build
Some of the practical thing I perceive as interesting with Bazel over CMake personally:
  - Strict checking of the dependencies: this seems strange at first that a missing library dependency won't allow you to include a header which is present on the filesystem, but this is actually very powerful: this is what allow Bazel to have correctness with incremental build and what enables caching at all. This is also quite fundamental to the "distributed build" mode in Bazel: you can farm-out the build to a large distributed cluster with remote cachine.
  - Declarative approach: this enables static analysis of the BUILD configuration/graphs, and transformations as well. For example because of the above, the tooling can figure out unused library dependencies, or adding missing dependencies as well.  
These two aspects are impossible to achieve in a principled way in CMake. Note that it does not imply that I would (or wouldn't) pick Bazel from my next open-source project, or that I would recommend LLVM to adopt it as a primary build system!

I can also miss some reasons why some other projects are using Bazel, but they got >400 attendees at the Bazel conference last year apparently: https://blog.bazel.build/2019/12/20/bazelcon-2019.html
Ultimately I don't judge why other projects are picking Bazel, we're just proposing making their life easier if they start being interested to include some of LLVM in their project.
In particular, putting on my MLIR ecosystem hat, it'd be great if other projects using Bazel out-there who may have a possible use for MLIR and LLVM could have an easier integration path: right now in practice they may use the Bazel configuration that are shipped inside TensorFlow and try to adjust them to BUILD LLVM and send us patches to integrate in TensorFlow. This is difficult because they may want to improve support for the Bazel config on platforms that TensorFlow does not support: it is just not the right place for people interested in building with Bazel to collaborate.


I'm a little concerned about having two 'unsupported' buildsystems
living in tree, and I'm not sure what would stop us from continuing to
add more.

Following up on some concerns from Tom (and others) explained on Discord here:

a) Commit Mailing list traffic for updating these build files. This is a valid point with the GN bot today. It also shows up when I `git log llvm/` in the monorepo annoyingly. 
The proposal would be to have the Bazel and `gn` files in a separate folder at the top level of the monorepo: that way no commits email would be sent to any mailing list, and no update would show up in the `git log llvm/`.
b) CI systems picking up more commits to build when not needed: similarly as above, isolating these in a separate part of the tree will exclude these from bots tracking the llvm/ or clang/ paths.
c) Investing in developing Bazel support means less investment in CMake. It is true that engineers fixing Bazel configs are spending time there instead of in CMake, however the situation is that downstream projects that picked Bazel (for their own reasons, I don't judge) who start using LLVM are spending the time to maintain these Bazel files out-of-tree. We're not making the situation worse by allowing the maintainer of these projects (who are also frequently upstream contributors) to just collaborate on their set of patches in a more coordinated way upstream. Also no public LLVM bots builds with an unsupported build system, any feature is expected to build with CMake on every supported platform I believe. It seems like this worked out well with `gn` in practice?

 
I would feel better if we had a set of guidelines to define
the criteria for adding a new buildsytem and also criteria for when we
can remove them.

Would you be able to amend this proposal to include some general
guidelines for adding/removing new buildsystems, so that we can discuss
that too?


That's an excellent point as well! Could we take inspiration from the experimental (or non-experimental) backends? For example if `gn` does not build anymore, send an email to LLVM-dev@ proposing to remove it and see if any maintainer steps up? Without a community to maintain it we should remove these easily and quickly?

-- 
Mehdi

Sterling Augustine via llvm-dev

unread,
Oct 29, 2020, 4:14:10 PM10/29/20
to Chris Tetreault, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
On Thu, Oct 29, 2020 at 12:29 PM Chris Tetreault via llvm-dev <llvm...@lists.llvm.org> wrote:

I think Renato has articulated quite well some concerns I have about this but was unable to express. I would very much prefer if we just focus on using CMake effectively.

... 

For example, when trying to implement the same logic on both will not be trivial. So, whenever we want to add some functionality or improve how we build LLVM with one system, we'll have to do so in multiple build systems that do not easily match each other.


Google already does all of this work, and has for years. I think it is fair to say that it hasn't been a burden on the community.
 

If we don't try to match functionality, we'll segregate the community, because people will be able to do X on build system A but not B, and the similar features cluster together and then we have essentially two projects built from the same source code.


As long as we keep CMake as the canonical system everything will be fine. It works perfectly well today, except that not everyone gets to see or use the bazel files. They exist right now; they work right now; and it hasn't been a burden on anyone but the people who care about bazel.
  

Testing this, or worse, trying to fix a buildbot that is built with Bazel (and having to install Java JDK and all its dependencies) on potentially a hardware that you do not have access to, will be a nightmare to debug. The nature of post-commit testing, revert and review of LLVM will not make that simpler. Unless we treat the Bazel build as "not our problem" (which defeats the point of having it?).


Google makes it work like this today, with the rest of the project treating it as "not our problem" because they don't even see that they exist. The build bot issues would be real, but I think surmountable, given that Google already cleans up the bazel files, it just doesn't push them. Perhaps an explicit policy that cmake folks don't have to update the bazel files would be helpful.
  

To make matters worse, our CMake files are not simple, and do not do all of the things we want them to do in the way we understand completely. There is a lot of kludge that we carry and with that comes in two categories: the things that we hate and would love to fix, and the things that are fixes that we have no idea are there. The former are the reasons why people want to start a new build system, the latter is why they soon realise that was a mistake (insert XKCD joke here).


It wouldn't be starting a new build system, it would be making a pre-existing, already extremely well functioning one, available to more people.

I can definitely see folks who use cmake not wanting more hassle--that may be a valid reason not to do it. But "it won't work" or "it's hard to keep up" or "it's too complicated" seem well refuted by a multi-year existence proof.

Chris Tetreault via llvm-dev

unread,
Oct 29, 2020, 4:24:53 PM10/29/20
to Sterling Augustine, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg

The problem is that once it’s in community LLVM, it becomes the community’s problem.  The expectation is that individual contributors do not break anything in upstream. Why else would you contribute it to the LLVM monorepo? If the goal is just to enable external-to-google orgs to collaborate on it, why not contribute it as a new repo separate from LLVM? You wouldn’t need to ask anybody’s permission to do this.

 

From: Sterling Augustine <saugu...@google.com>
Sent: Thursday, October 29, 2020 1:14 PM
To: Chris Tetreault <ctet...@quicinc.com>
Cc: Renato Golin <reng...@gmail.com>; tste...@redhat.com; Mehdi Amini <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble <gc...@google.com>; Thomas Joerg <tjo...@google.com>
Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn

 

On Thu, Oct 29, 2020 at 12:29 PM Chris Tetreault via llvm-dev <llvm...@lists.llvm.org> wrote:

Mehdi Amini via llvm-dev

unread,
Oct 29, 2020, 4:46:51 PM10/29/20
to Renato Golin, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
On Thu, Oct 29, 2020 at 12:49 PM Renato Golin <reng...@gmail.com> wrote:
On Thu, 29 Oct 2020 at 19:16, David Blaikie <dbla...@gmail.com> wrote:
I /believe/ the idea is that, like gn, there are folks maintaining these build systems out of tree anyway - and having them in tree makes it easier to coordinate that effort, with the express intent of not burdening the general community with their upkeep (like gn currently - the idea is that there's no burden on developers to update gn build files (& consequently bazel build files)).

Perhaps the initial assumption about my concerns weren't well articulated. 

I get that those files would be "additional" and other developers won't need to care much about them. 

But what happens when people join the project with experience in Bazel and, instead of building pure LLVM with CMake, they start using Bazel for everything, just because they're used to it?

I would propose to have the files in a separate tree from llvm/, mlir/, clang/ ; labelling these clearly as unsupported (either in the path to these files or in the README, or both), and not provide any public documentation on llvm.org that would invite users to work with these. The readme would explain how to use them to include LLVM as a dependency to an existing Bazel project and document the intent as such.


Bazel is big enough (at least inside Google) that the probability of that happening is not trivial.

What if they create sub-projects that can only build with Bazel? Do we refuse inclusion? But don't we have Bazel files already?

This is a fair concern: can we defend against this with a clear policy?
Also: no public bot with Bazel or other build system than CMake should help right?
 

One big example is Android. They used to build LLVM in a very different way, and the inclusion of run-time library files was completely different. So different it was not possible to merge some changes they had (128 bit maths IIRC) because of the amount of work required.

My point is that adding another build system will not necessarily improve the chances of external people contributing to LLVM if they use those build systems. It may very well *reduce* those chances.

My intuition was that by having the file upstream, we would instead encourage such users to track the HEAD of our main branch more closely and so provide them an easier path for upstream work.  The fact that they can get upstream working with their build environment may provide an incentive to upstream along the way, even if they have to do the CMake integration first.
Ultimately while this may facilitate people to go in one direction or another, I suspect they would just reinforce their natural tendency: people interested in working more upstream will have a better path, and people who have less of this tendency may also have an easier path of integration. 

Neil Nelson via llvm-dev

unread,
Oct 29, 2020, 4:54:26 PM10/29/20
to llvm...@lists.llvm.org
Not seeing Bazel at https://packages.ubuntu.com/.

Glad to see that the current required cmake version is being met with
Ubuntu 20.04 and later.

Mehdi AMINI via llvm-dev

unread,
Oct 29, 2020, 5:00:34 PM10/29/20
to Chris Tetreault, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
On Thu, Oct 29, 2020 at 1:24 PM Chris Tetreault via llvm-dev <llvm...@lists.llvm.org> wrote:

The problem is that once it’s in community LLVM, it becomes the community’s problem.  The expectation is that individual contributors do not break anything in upstream.


I would expect that the community by now has concrete experience with `gn` gained over a few years demonstrating that this hasn't been a problem to have this in-tree, without a burden of support on the community.
In particular, I think that a salient point is the guarantee that no public bot would be testing it (I mean here by "no public bot" that no bot would email you when you break it).
 

Why else would you contribute it to the LLVM monorepo? If the goal is just to enable external-to-google orgs to collaborate on it, why not contribute it as a new repo separate from LLVM? You wouldn’t need to ask anybody’s permission to do this.


Yes, we could do this, and you are correct that in many cases a motivation to upstream a component is to make sure it is maintained by the community and works out of the box. 
In this case it is slightly different: we are OK with people to break this. We are already maintaining these files out-of-tree for our own purposes, and this has been the case for years as Sterling mentions. I would even suspect that for Google internal build integration, it is actually easier to have these files internal only rather than unsupported upstream.
So why are we doing it? I mentioned this in another answer: this is mainly to provide a collaboration space for the support of OSS projects using Bazel interested to use LLVM (and some subprojects).
Having them in-tree means that we can publish every day (or more) a git hash that we validate with Bazel on private bots (like `gn`) and every project can use to clone the LLVM monorepo and integrate in their build flow easily. Another repo, submodules, etc. are not making this possible / practical.


 

 

From: Sterling Augustine <saugu...@google.com>
Sent: Thursday, October 29, 2020 1:14 PM
To: Chris Tetreault <ctet...@quicinc.com>
Cc: Renato Golin <reng...@gmail.com>; tste...@redhat.com; Mehdi Amini <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble <gc...@google.com>; Thomas Joerg <tjo...@google.com>
Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn

 

On Thu, Oct 29, 2020 at 12:29 PM Chris Tetreault via llvm-dev <llvm...@lists.llvm.org> wrote:

I think Renato has articulated quite well some concerns I have about this but was unable to express. I would very much prefer if we just focus on using CMake effectively.

... 

For example, when trying to implement the same logic on both will not be trivial. So, whenever we want to add some functionality or improve how we build LLVM with one system, we'll have to do so in multiple build systems that do not easily match each other.

 

Google already does all of this work, and has for years. I think it is fair to say that it hasn't been a burden on the community.

 

If we don't try to match functionality, we'll segregate the community, because people will be able to do X on build system A but not B, and the similar features cluster together and then we have essentially two projects built from the same source code.

 

As long as we keep CMake as the canonical system everything will be fine. It works perfectly well today, except that not everyone gets to see or use the bazel files. They exist right now; they work right now; and it hasn't been a burden on anyone but the people who care about bazel.

  

Testing this, or worse, trying to fix a buildbot that is built with Bazel (and having to install Java JDK and all its dependencies) on potentially a hardware that you do not have access to, will be a nightmare to debug. The nature of post-commit testing, revert and review of LLVM will not make that simpler. Unless we treat the Bazel build as "not our problem" (which defeats the point of having it?).

 

Google makes it work like this today, with the rest of the project treating it as "not our problem" because they don't even see that they exist. The build bot issues would be real, but I think surmountable, given that Google already cleans up the bazel files, it just doesn't push them. Perhaps an explicit policy that cmake folks don't have to update the bazel files would be helpful.

  

To make matters worse, our CMake files are not simple, and do not do all of the things we want them to do in the way we understand completely. There is a lot of kludge that we carry and with that comes in two categories: the things that we hate and would love to fix, and the things that are fixes that we have no idea are there. The former are the reasons why people want to start a new build system, the latter is why they soon realise that was a mistake (insert XKCD joke here).

 

It wouldn't be starting a new build system, it would be making a pre-existing, already extremely well functioning one, available to more people.

 

I can definitely see folks who use cmake not wanting more hassle--that may be a valid reason not to do it. But "it won't work" or "it's hard to keep up" or "it's too complicated" seem well refuted by a multi-year existence proof.

 

_______________________________________________

Dave Lee via llvm-dev

unread,
Oct 29, 2020, 5:01:33 PM10/29/20
to Sterling Augustine, Mehdi Amini, Stella Laurenzo, Tres Popp, LLVM Dev, Geoffrey Martin-Noble, Thomas Joerg
On Thu, Oct 29, 2020 at 1:14 PM Sterling Augustine via llvm-dev <llvm...@lists.llvm.org> wrote:
On Thu, Oct 29, 2020 at 12:29 PM Chris Tetreault via llvm-dev <llvm...@lists.llvm.org> wrote:

I think Renato has articulated quite well some concerns I have about this but was unable to express. I would very much prefer if we just focus on using CMake effectively.

... 

For example, when trying to implement the same logic on both will not be trivial. So, whenever we want to add some functionality or improve how we build LLVM with one system, we'll have to do so in multiple build systems that do not easily match each other.


Google already does all of this work, and has for years. I think it is fair to say that it hasn't been a burden on the community.

As far as I can tell, the only burden would be the existence of bazel files in the repo, but scoped within a directory most will never look at and can ignore. The issue of extra files seems of most concern to anyone who is tracking/optimizing repo size or file and commit count.
 

I can definitely see folks who use cmake not wanting more hassle--that may be a valid reason not to do it.

What would be the hassle to cmake users?

Dave

Renato Golin via llvm-dev

unread,
Oct 29, 2020, 5:16:13 PM10/29/20
to Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
On Thu, 29 Oct 2020 at 20:46, Mehdi Amini <ami...@google.com> wrote:
I would propose to have the files in a separate tree from llvm/, mlir/, clang/ ; labelling these clearly as unsupported (either in the path to these files or in the README, or both), and not provide any public documentation on llvm.org that would invite users to work with these. The readme would explain how to use them to include LLVM as a dependency to an existing Bazel project and document the intent as such.

Sounds good, with the addendum below:

This is a fair concern: can we defend against this with a clear policy?
Also: no public bot with Bazel or other build system than CMake should help right?

Right, as you said later, no emails from broken bots, ie. people that use non-CMake build systems are expected to fix their own builds, even if the breakage came from a third-party change.

Of course, we expect other contributors to help with what they can, but it's not their responsibility. Such is the cost of having a different build system.

My intuition was that by having the file upstream, we would instead encourage such users to track the HEAD of our main branch more closely and so provide them an easier path for upstream work.  The fact that they can get upstream working with their build environment may provide an incentive to upstream along the way, even if they have to do the CMake integration first.

The benefit to keep the files in LLVM is clear to all Bazel users out there. I don't think that's a problem for the rest of LLVM.

My point was that the reason why Arm's patch (128-bit) was "uncontributable" was because their build system was so different, it was impossible to keep both versions on their merged tree. This is a big problem to the Bazel users, not CMake users.

If we keep the policy of "not my problem", I don't see a single problem to CMake users. But that's slightly unfriendly to people that came later to LLVM and "didn't know better" before creating a whole project in Bazel, etc.

I'm strictly not thinking about myself here.

--renato

Sterling Augustine via llvm-dev

unread,
Oct 29, 2020, 5:16:30 PM10/29/20
to Dave Lee, Mehdi Amini, Stella Laurenzo, Tres Popp, LLVM Dev, Geoffrey Martin-Noble, Thomas Joerg
On Thu, Oct 29, 2020 at 2:01 PM Dave Lee <davel...@gmail.com> wrote:
What would be the hassle to cmake users?

Nothing that can't be addressed by policy, such as:

1. No public Bazel buildbots blaming them when the build breaks.
2. Cmake changes are not reverted to due issues with bazel.
2. Requiring that things implemented in bazel also needs to be implemented in cmake. "Things" is somewhat vague here, because there are things that can be done in bazel that can't be done in cmake. But as long as both systems can build everything (bazel's power of analysis is much greater), I think this would be acceptable.

Renato Golin via llvm-dev

unread,
Oct 29, 2020, 5:18:20 PM10/29/20
to Sterling Augustine, Mehdi Amini, Stella Laurenzo, Tres Popp, LLVM Dev, Geoffrey Martin-Noble, Thomas Joerg
On Thu, 29 Oct 2020 at 20:14, Sterling Augustine <saugu...@google.com> wrote:

To make matters worse, our CMake files are not simple, and do not do all of the things we want them to do in the way we understand completely. There is a lot of kludge that we carry and with that comes in two categories: the things that we hate and would love to fix, and the things that are fixes that we have no idea are there. The former are the reasons why people want to start a new build system, the latter is why they soon realise that was a mistake (insert XKCD joke here).


It wouldn't be starting a new build system, it would be making a pre-existing, already extremely well functioning one, available to more people.

I can definitely see folks who use cmake not wanting more hassle--that may be a valid reason not to do it. But "it won't work" or "it's hard to keep up" or "it's too complicated" seem well refuted by a multi-year existence proof.

That is definitely not what I meant here.

Chris Tetreault via llvm-dev

unread,
Oct 29, 2020, 5:35:47 PM10/29/20
to Mehdi AMINI, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg

Honestly, I’m hearing that some people would like the Bazel build system to be in community master, and the argument basically boils down to “It’ll be fine. It’ll just sit there and mind its own business and you don’t have to care about it.”

 

> So why are we doing it? I mentioned this in another answer: this is mainly to provide a collaboration space for the support of OSS projects using Bazel interested to use LLVM (and some subprojects). …

 

Which could be handled by having it in an external public repo.

 

> Having them in-tree means that we can publish every day (or more) a git hash that we validate with Bazel on private bots (like `gn`) and every project can use to clone the LLVM monorepo and integrate in their build flow easily.

 

You could still publish this info: “Today, the head of llvm-bazel is confirmed to work with LLVM monorepo sha [foo]”. I don’t think two git clones is significantly harder than one. I submit that in a way this is simpler because you can always advertise the head of the bazel repo. If the Bazel build system were in the community repo, then you might have to tell users to use an older version of the bazel build if a fix went into the monorepo in the afternoon, but the next morning’s nightly finds that the most recent sha that passes the tests is prior to that fix.

 

I guess my concern is that I’m not really hearing a compelling (to my ear) argument for this inclusion. I guess it would make the lives of google employees easier? Then what’s to stop every large org from committing their internal stuff to master?

 

From: Mehdi AMINI <joke...@gmail.com>
Sent: Thursday, October 29, 2020 2:00 PM
To: Chris Tetreault <ctet...@quicinc.com>
Cc: Sterling Augustine <saugu...@google.com>; Mehdi Amini <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble <gc...@google.com>; Thomas Joerg <tjo...@google.com>
Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn

 

On Thu, Oct 29, 2020 at 1:24 PM Chris Tetreault via llvm-dev <llvm...@lists.llvm.org> wrote:

The problem is that once it’s in community LLVM, it becomes the community’s problem.  The expectation is that individual contributors do not break anything in upstream.

 

I would expect that the community by now has concrete experience with `gn` gained over a few years demonstrating that this hasn't been a problem to have this in-tree, without a burden of support on the community.

In particular, I think that a salient point is the guarantee that no public bot would be testing it (I mean here by "no public bot" that no bot would email you when you break it).

 

Why else would you contribute it to the LLVM monorepo? If the goal is just to enable external-to-google orgs to collaborate on it, why not contribute it as a new repo separate from LLVM? You wouldn’t need to ask anybody’s permission to do this.

 

Yes, we could do this, and you are correct that in many cases a motivation to upstream a component is to make sure it is maintained by the community and works out of the box. 

In this case it is slightly different: we are OK with people to break this. We are already maintaining these files out-of-tree for our own purposes, and this has been the case for years as Sterling mentions. I would even suspect that for Google internal build integration, it is actually easier to have these files internal only rather than unsupported upstream.

So why are we doing it? I mentioned this in another answer: this is mainly to provide a collaboration space for the support of OSS projects using Bazel interested to use LLVM (and some subprojects).

Having them in-tree means that we can publish every day (or more) a git hash that we validate with Bazel on private bots (like `gn`) and every project can use to clone the LLVM monorepo and integrate in their build flow easily. Another repo, submodules, etc. are not making this possible / practical.

 

 

 

 

From: Sterling Augustine <saugu...@google.com>
Sent: Thursday, October 29, 2020 1:14 PM
To: Chris Tetreault <ctet...@quicinc.com>
Cc: Renato Golin <reng...@gmail.com>; tste...@redhat.com; Mehdi Amini <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble <gc...@google.com>; Thomas Joerg <tjo...@google.com>
Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn

 

On Thu, Oct 29, 2020 at 12:29 PM Chris Tetreault via llvm-dev <llvm...@lists.llvm.org> wrote:

I think Renato has articulated quite well some concerns I have about this but was unable to express. I would very much prefer if we just focus on using CMake effectively.

... 

For example, when trying to implement the same logic on both will not be trivial. So, whenever we want to add some functionality or improve how we build LLVM with one system, we'll have to do so in multiple build systems that do not easily match each other.

 

Google already does all of this work, and has for years. I think it is fair to say that it hasn't been a burden on the community.

 

If we don't try to match functionality, we'll segregate the community, because people will be able to do X on build system A but not B, and the similar features cluster together and then we have essentially two projects built from the same source code.

 

As long as we keep CMake as the canonical system everything will be fine. It works perfectly well today, except that not everyone gets to see or use the bazel files. They exist right now; they work right now; and it hasn't been a burden on anyone but the people who care about bazel.

  

Testing this, or worse, trying to fix a buildbot that is built with Bazel (and having to install Java JDK and all its dependencies) on potentially a hardware that you do not have access to, will be a nightmare to debug. The nature of post-commit testing, revert and review of LLVM will not make that simpler. Unless we treat the Bazel build as "not our problem" (which defeats the point of having it?).

 

Google makes it work like this today, with the rest of the project treating it as "not our problem" because they don't even see that they exist. The build bot issues would be real, but I think surmountable, given that Google already cleans up the bazel files, it just doesn't push them. Perhaps an explicit policy that cmake folks don't have to update the bazel files would be helpful.

  

To make matters worse, our CMake files are not simple, and do not do all of the things we want them to do in the way we understand completely. There is a lot of kludge that we carry and with that comes in two categories: the things that we hate and would love to fix, and the things that are fixes that we have no idea are there. The former are the reasons why people want to start a new build system, the latter is why they soon realise that was a mistake (insert XKCD joke here).

 

It wouldn't be starting a new build system, it would be making a pre-existing, already extremely well functioning one, available to more people.

 

I can definitely see folks who use cmake not wanting more hassle--that may be a valid reason not to do it. But "it won't work" or "it's hard to keep up" or "it's too complicated" seem well refuted by a multi-year existence proof.

 

_______________________________________________

Keith Smiley via llvm-dev

unread,
Oct 29, 2020, 6:17:44 PM10/29/20
to Geoffrey Martin-Noble, llvm...@lists.llvm.org, Mehdi Amini, Stella Laurenzo, Thomas Joerg, Tres Popp
I want to jump in with some general support for this addition as a non-googler + frequent bazel user. I find moving between projects that use bazel much more palatable than moving between bazel + cmake projects.

I also think there's a huge benefit in having strict dependencies. In my experience this is especially true for tests. This way you know you've always correctly rebuilt the necessary inputs to a single test, vs using `lit` directly and having to know / remember which set of binaries are required to be rebuilt based on your current changes.
--
Keith Smiley


Mehdi AMINI via llvm-dev

unread,
Oct 29, 2020, 6:48:35 PM10/29/20
to Chris Tetreault, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
On Thu, Oct 29, 2020 at 2:35 PM Chris Tetreault <ctet...@quicinc.com> wrote:

Honestly, I’m hearing that some people would like the Bazel build system to be in community master, and the argument basically boils down to “It’ll be fine. It’ll just sit there and mind its own business and you don’t have to care about it.”


Not really: this argument is only the answer to why it does not bear any weight on non-Bazel users, just like `gn` does already today.

I think I explained the motivation to do it, but I can restate it: many LLVM contributors need to collaborate on this piece of infrastructure that is very specific to LLVM and enabling some users of LLVM: the natural place of collaboration is the monorepo. 
 

 

> So why are we doing it? I mentioned this in another answer: this is mainly to provide a collaboration space for the support of OSS projects using Bazel interested to use LLVM (and some subprojects). …

 

Which could be handled by having it in an external public repo.


Sure, just like almost every new code could be handled in an external repo. However when many LLVM contributors are interested to collaborate on something highly coupled to LLVM it seems like the natural place to do it.
Also I don't know for Qualcomm, but most companies will want you to sign a CLA if they provide this "external repo" where we can collaborate, and other parties won't be able to collaborate. The LLVM project is in general seen as quite "neutral" for collaborating.
 

 

> Having them in-tree means that we can publish every day (or more) a git hash that we validate with Bazel on private bots (like `gn`) and every project can use to clone the LLVM monorepo and integrate in their build flow easily.

 

You could still publish this info: “Today, the head of llvm-bazel is confirmed to work with LLVM monorepo sha [foo]”. I don’t think two git clones is significantly harder than one.


For a developer at their desk, you could say it is just an inconvenience that can be worked around (scripting, etc.). 
For the project on the other hand, Bazel has native support to clone a repo and build it itself as dependency.  For example TensorFlow has many dependencies, and it just points to a commit in the source repo: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/workspace.bzl#L689-L697
You can see how it is convenient to update the SHA1 there and have it just work for any Bazel user.

 

I submit that in a way this is simpler because you can always advertise the head of the bazel repo. If the Bazel build system were in the community repo, then you might have to tell users to use an older version of the bazel build if a fix went into the monorepo in the afternoon, but the next morning’s nightly finds that the most recent sha that passes the tests is prior to that fix.


This is not different from "a commit broke the ARM bootstrap and a user who checked out the repo at the time will be broken". From this point of view this configuration is no different than any other, except that we don't revert or notify the author of a breaking change, a set of volunteers monitor a silent bot and fix-forward as needed, like `gn`.
It is just much easier to have a bot publishing the "known good" revision of the monorepo.
 

I guess my concern is that I’m not really hearing a compelling (to my ear) argument for this inclusion.


Sure, but if other contributors have a strong interest, and you don't really have a strong objection here that we need to address, we should be able to get past that?
 

I guess it would make the lives of google employees easier? 


I explained before that Google internal integration flow is likely better without this at the moment, TensorFlow itself is also in a reasonably good spot at the moment. But Google is also not a monolithic place, some people are working on small independent projects that they are open-sourcing, and would like to be able to use LLVM. 

 Then what’s to stop every large org from committing their internal stuff to master?

 

If their "internal stuff" is highly-coupled to LLVM, has zero-cost maintenance on the community, and is something that multiple other parties can benefit and established members of the community want to maintain and collaborate on, why not?


I mentioned it before, but Bazel is not something internal or specific to Google: it isn't (actually there are many incompatibilities between Bazel and the internal system), 400 people attended the Bazel conference last year. I attended this conference 3 years ago when I was at Tesla trying to deploy Bazel internally. Many other companies are using Bazel, open-source projects as well. Feel free to watch the talks online about SpaceX or Two Sigma and Uber for example


I'm not trying to convince anyone to use Bazel, it has drawbacks, but the point here is to recognize that this is about OpenSource communities that Bazel is serving: these are users, some of us in the LLVM community are trying to provide these users with a reasonably good integration story, and we're ready to pay the cost for everyone.

Chris Tetreault via llvm-dev

unread,
Oct 29, 2020, 7:05:07 PM10/29/20
to Mehdi AMINI, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg

A at the end of the day, I’m just one person stating his opinion. You’ve submitted a proposal to make a change to the community, and I’ve stated my concerns as a member of the community. If broad support for this change is expressed by members of the community, then fine. As it stands, I’m seeing lots of googlers voicing support, and not much support from others. I’m not the arbiter of what gets done or not done. All I can do is state my opinions, which I have done.

 

If this contribution is accepted, I would request that the promise that the CMake build system will remain the canonical build system, and that no build bot will ever email me that I broke it, be documented.

 

For the record, I don’t find the argument that “gn already does this, so why not Bazel” to be compelling. If I were around when the gn build was added, I would have complained about it too.

Zachary Turner via llvm-dev

unread,
Oct 29, 2020, 7:11:52 PM10/29/20
to Renato Golin, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
On Thu, Oct 29, 2020 at 12:49 PM Renato Golin via llvm-dev <llvm...@lists.llvm.org> wrote:
On Thu, 29 Oct 2020 at 19:16, David Blaikie <dbla...@gmail.com> wrote:
I /believe/ the idea is that, like gn, there are folks maintaining these build systems out of tree anyway - and having them in tree makes it easier to coordinate that effort, with the express intent of not burdening the general community with their upkeep (like gn currently - the idea is that there's no burden on developers to update gn build files (& consequently bazel build files)).

Perhaps the initial assumption about my concerns weren't well articulated. 

I get that those files would be "additional" and other developers won't need to care much about them. 

But what happens when people join the project with experience in Bazel and, instead of building pure LLVM with CMake, they start using Bazel for everything, just because they're used to it?

Didn't the community already go through this exact discussion when gn was added?  Let me ask a different question.  If gn support was permitted, on what grounds should we refuse a different parallel build system?  Either we should allow people to contribute build systems upstream that they wish to maintain, or we should keep every buidl system other than CMake out of the tree. 

David Blaikie via llvm-dev

unread,
Oct 29, 2020, 7:16:05 PM10/29/20
to Stefan Teleman, LLVM Dev
On Thu, Oct 29, 2020 at 12:41 PM Stefan Teleman <stefan....@gmail.com> wrote:
> This is a fairly unhelpful email - clearly folks using Bazel derive some benefit/have chosen some tradeoff compared to CMake. Doesn't have to be the thing you want, but it's pretty unhelpful to dismiss/diminish the needs of others like this.

I did not see a rationale for the Bazel proposal, outlining its
benefits over CMake.

Speaking with direct experience with Bazel - Tensorflow - I cannot
think of a single reason why it would/should be considered "better"
over the current CMake.

Everyone has their own favorite build system. That is nice, but it is
not enough of a reason to propose adding it.

I would also like to become informed as to what particular
needs/shortcomings/defects are addressed by Bazel, that are lacking in
/ cannot be addressed by CMake.

I expect most of it is probably a statement free of value judgments: Some other projects chose to use it/some folks have to use it for other reasons, clearly there's enough use that it's motivated folks to have/maintain Bazel builds for LLVM for years. Rather than judging their choices as bad/lesser/wrong - might be useful to accept that some folks had their reasons and they're trying to make the most of the situation. I don't think anyone's making an argument that LLVM should switch to Bazel/that that would be better than the CMake we're using, and I think it's helpful to return the favor and not suggest that other projects would be better off switching to CMake over Bazel - they no doubt have their reasons.

- Dave
 

Stefan Teleman via llvm-dev

unread,
Oct 29, 2020, 7:49:29 PM10/29/20
to David Blaikie, LLVM Dev
On Thu, Oct 29, 2020 at 7:16 PM David Blaikie <dbla...@gmail.com> wrote:

> I expect most of it is probably a statement free of value judgments: Some other projects chose to use it/some folks have to use it for other reasons, clearly there's enough use that it's motivated folks to have/maintain Bazel builds for LLVM for years. Rather than judging their choices as bad/lesser/wrong - might be useful to accept that some folks had their reasons and they're trying to make the most of the situation. I don't think anyone's making an argument that LLVM should switch to Bazel/that that would be better than the CMake we're using, and I think it's helpful to return the favor and not suggest that other projects would be better off switching to CMake over Bazel - they no doubt have their reasons.

Please do not manufacture statements that I did not make. I never
suggested, or stated, anywhere, that some other imaginary project
using Bazel should switch to CMake.

I did state that I do not find Bazel to be a better alternative to
CMake. My statement is based on direct experience with both.

If the intent behind Bazel is not to present it as a better
alternative to CMake, then what is the intent? Instead of maintaining
this impenetrable mystery as to why a Bazel build system should be
included in LLVM, please take the time to advocate for Bazel with
technical facts, than "someone at Google really likes it".

Just because someone likes and maintains an alternative build system
for LLVM, somewhere, that does not automatically mean, or imply that
it should be upstreamed.

For all I know, someone might be building their fork of LLVM with
autoconf. I am sure they have their own very good reasons for doing
so. Should we, therefore, bring back autoconf?

Thanks.

--
Stefan Teleman
stefan....@gmail.com

Chris Tetreault via llvm-dev

unread,
Oct 29, 2020, 7:50:12 PM10/29/20
to Zachary Turner, Renato Golin, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg

I think this argument is the slippery slope in action. Just because we allowed the gn build system to be added previously, does not mean that we should allow a new build system now. And forbidding this build system now does not mean that we must kick gn out of the repo.

 

We should accept or reject Bazel on its merits alone, and not based on historical precedent.

 

From: llvm-dev <llvm-dev...@lists.llvm.org> On Behalf Of Zachary Turner via llvm-dev
Sent: Thursday, October 29, 2020 4:11 PM
To: Renato Golin <reng...@gmail.com>

Cc: Mehdi Amini <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble <gc...@google.com>; Thomas Joerg <tjo...@google.com>
Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn

 

 

 

On Thu, Oct 29, 2020 at 12:49 PM Renato Golin via llvm-dev <llvm...@lists.llvm.org> wrote:

Krzysztof Parzyszek via llvm-dev

unread,
Oct 29, 2020, 7:57:01 PM10/29/20
to llvm...@lists.llvm.org

On the grounds that it was a bad idea after all.

 

Any commits going into the LLVM repository should not break any part of it, at least not without a consideration for a fix.  There is an exception to it---experimental targets.  They can be broken, but they are there with the explicit intent of becoming officially supported.

 

Same thing applies to the cmake files.  If they get broken, they need to be fixed, but the same doesn’t apply to the extraneous build systems.  They can be broken and never fixed.  There is no commitment from the community as a whole to keep them working.  IMO, this isn’t right, and files like that should not be a part of the official repository.

 

Whether GN or Bazel have superior features is irrelevant.  Unless their configuration files are a part of a longer-term transition process, they don’t belong in the repo.

 

 

--

Krzysztof Parzyszek  kpar...@quicinc.com   AI tools development

 

From: llvm-dev <llvm-dev...@lists.llvm.org> On Behalf Of Zachary Turner via llvm-dev
Sent: Thursday, October 29, 2020 6:11 PM
To: Renato Golin <reng...@gmail.com>
Cc: Mehdi Amini <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble <gc...@google.com>; Thomas Joerg <tjo...@google.com>
Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn

 

 

 

On Thu, Oct 29, 2020 at 12:49 PM Renato Golin via llvm-dev <llvm...@lists.llvm.org> wrote:

Geoffrey Martin-Noble via llvm-dev

unread,
Oct 29, 2020, 7:58:43 PM10/29/20
to Zachary Turner, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Thomas Joerg
This seems to have fragmented into a few separate threads, so apologies if I'm missing someone's response. I don't think I'm going to be able to effectively reply inline.

The intention here is not to propose Bazel be another community-maintained build system or that it replace CMake. In my initial message I deliberately didn't focus on the specific advantages to Bazel because I'm not really trying to convince anyone to use it. Personally I prefer it to CMake, but it's got some annoying parts I don't like as well (my project uses both). The goal here is that if you don't care about Bazel, you should not be impacted. But some projects (and people) *do* use Bazel and depend on LLVM and right now that means copying around different versions of these files. We've been working on at least consolidating these in https://github.com/google/llvm-bazel. I think Mehdi already summarized the reasons we think it would make more sense for these to be in-tree than in a separate repo: it's a more natural collaboration point.

Tom raised some specific concerns about the grounds for adding or removing a build system and when we know we've got bit rot, pointing out that it can be harder to tell with a build system if it's not one you actually use! In this case, we've got a build bot (lowercase, I'm using BuildKite) that builds against head every 15 minutes or so (there are currently some delays caused by pulling in new changes from the monorepo). Does a functioning bot with a clearly visible (though not noisy!) status indicator seem like a reasonable requirement? If this bot remains broken for a long time and/or no one has been updating the build files, then that would be an indication of bit rot. Someone would send a message to the list proposing the build files be deleted and doing so should be relatively easy.

I think some of the other general concerns about people using Bazel instead of CMake and assuming support or breaking the CMake build should be solvable by Bazel being in a side-directory (as proposed) with a clear readme that explains the level of support. I'll draft such a readme to include with the patch (I was waiting to see some responses to the RFC before doing so).

A side point regarding extra commit traffic. I proposed putting these in the top-level utils/ directory. Would it also make sense to move gn there to similarly remove it from commit mailing list traffic?

I believe that Tom actually highlighted this particular slippery slope as a concern. Hopefully having some reasonable maintenance standards as described above can help us avoid it. But I do think we can use gn as an experiment into how that went. It's been a couple years. Has anyone experienced issues with their presence in the monorepo?

David Blaikie via llvm-dev

unread,
Oct 29, 2020, 9:03:15 PM10/29/20
to Chris Tetreault, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg


On Thu, Oct 29, 2020 at 4:50 PM Chris Tetreault via llvm-dev <llvm...@lists.llvm.org> wrote:

(side note: Chris: There's /something/ about your emails that isn't threading in gmail at least - not sure if it's something you're aware of/something you can do anything about, but figured I'd mention (hmm, looks like they thread correctly on the llvm-dev archive, so the headers are probably correct - the subject lines look the same, so I really don't know what gmail's doing with them))

I think this argument is the slippery slope in action. Just because we allowed the gn build system to be added previously, does not mean that we should allow a new build system now. And forbidding this build system now does not mean that we must kick gn out of the repo.

 

We should accept or reject Bazel on its merits alone, and not based on historical precedent.


I don't entirely agree - I think it's worth asking whether there's significant differences between this and previous choices. Precedent has bearing to me - "slippery slope" is a fallacy that if we allow A we must allow B, specifically when B does not follow from A. That we allow gn doesn't necessarily mean we have to allow all build systems, or that we have to allow Bazel - they are different, but are they different in significant ways that matter I think is the question.

I think relevant questions to ask to avoid rehashing the same decisions would be:

What problems do you/we find with the gn integration, that we could learn from to avoid making similar mistakes?
Have there been problems/costs to the community with the gn integration that we'd like to avoid incurring more of?
Otherwise, are there things that make Bazel integration different from gn that might make it worse/different in terms of costs to the community?
 

 

From: llvm-dev <llvm-dev...@lists.llvm.org> On Behalf Of Zachary Turner via llvm-dev
Sent: Thursday, October 29, 2020 4:11 PM
To: Renato Golin <reng...@gmail.com>
Cc: Mehdi Amini <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble <gc...@google.com>; Thomas Joerg <tjo...@google.com>
Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn

 

 

 

On Thu, Oct 29, 2020 at 12:49 PM Renato Golin via llvm-dev <llvm...@lists.llvm.org> wrote:

On Thu, 29 Oct 2020 at 19:16, David Blaikie <dbla...@gmail.com> wrote:

I /believe/ the idea is that, like gn, there are folks maintaining these build systems out of tree anyway - and having them in tree makes it easier to coordinate that effort, with the express intent of not burdening the general community with their upkeep (like gn currently - the idea is that there's no burden on developers to update gn build files (& consequently bazel build files)).

 

Perhaps the initial assumption about my concerns weren't well articulated. 

 

I get that those files would be "additional" and other developers won't need to care much about them. 

 

But what happens when people join the project with experience in Bazel and, instead of building pure LLVM with CMake, they start using Bazel for everything, just because they're used to it?

 

Didn't the community already go through this exact discussion when gn was added?  Let me ask a different question.  If gn support was permitted, on what grounds should we refuse a different parallel build system?  Either we should allow people to contribute build systems upstream that they wish to maintain, or we should keep every buidl system other than CMake out of the tree. 

_______________________________________________

Shoaib Meenai via llvm-dev

unread,
Oct 29, 2020, 9:05:08 PM10/29/20
to tste...@redhat.com, Geoffrey Martin-Noble, llvm...@lists.llvm.org, Mehdi Amini, Stella Laurenzo, Thomas Joerg, Tres Popp
The main benefit I see is the ease of integrating into an existing Bazel build system.

At Facebook, we use Buck (which is inspired by Blaze, as is Bazel). Our main development repository uses Buck (and you get a variety of benefits from such build systems when you have the required infrastructure integration: remote caching, distributed builds, hermeticity, etc.), and the Buck build sets up particular flags, uses a specific sysroot, etc. We have people who want to develop LLVM-based tooling (that uses the LLVM and Clang libraries) in this repository, which means the LLVM and Clang libraries we build with CMake also need to be built with the same flags, sysroot, etc., which entails a bunch of duplication (and keeping up with any changes to the Buck build system). It's much more convenient to be able to build the LLVM libraries directly with the build system you're using for the rest of your build, so that they automatically get the right build settings.

We build our libraries internally with CMake today, but we've considered moving them to Buck for this reason. Having Bazel files in-tree would be mildly more convenient for us (Buck and Bazel are similar enough that we think we could machine-translate the Bazel files for our use), although we're also fine grabbing them from some other public repository.

On 10/29/20, 8:23 AM, "llvm-dev on behalf of Tom Stellard via llvm-dev" <llvm-dev...@lists.llvm.org on behalf of llvm...@lists.llvm.org> wrote:

On 10/28/20 7:18 PM, Geoffrey Martin-Noble via llvm-dev wrote:
> Hi all,
>
> tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a
> side-directory in the monorepo, similar to the gn build.
>
> Some of us have been working on open-source Bazel BUILD files for the
> LLVM Project. You may have seen us hanging out in the #build-systems
> discord channel. As you may know, Google uses Bazel internally and has
> maintained a Bazel BUILD of LLVM for years. Especially with the
> introduction of MLIR, we've got more and more OSS projects with a Bazel
> BUILD depending on LLVM (e.g. IREE <https://github.com/google/iree>and
> TensorFlow <https://github.com/tensorflow/tensorflow>). We're also not
> the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM
> that they've borrowed from TF
> <https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD>.
> Each of these projects has to jump through some weird hoops to keep
> their version of the Bazel BUILD files in sync with the code, which
> requires some fragile combination of scripts and human intervention.
> Instead, we'd like to move general-purpose Bazel BUILD files into the
> LLVM Project monorepo. We expect to follow the model of the GN build
> where these will be maintained by interested contributors rather than
> expecting the general community to maintain them.
>
> To facilitate and test this we've been developing a standalone
> repository that just has the Bazel BUILD files. It symlinks together the
> directory trees on top of a submodule as we would need in the monorepo
> to to avoid in-tree BUILD files. The configuration is at
> https://github.com/google/llvm-bazel. We now have those in a good place
> and think they would be useful upstream.
>

Can you explain some of the benefits to using Bazel instead of CMake?

I'm a little concerned about having two 'unsupported' buildsystems
living in tree, and I'm not sure what would stop us from continuing to
add more. I would feel better if we had a set of guidelines to define
the criteria for adding a new buildsytem and also criteria for when we
can remove them.

Would you be able to amend this proposal to include some general
guidelines for adding/removing new buildsystems, so that we can discuss
that too?

Thanks,
Tom


>
> # Details
>
> ## What
>
> Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review
> <https://github.com/google/llvm-bazel/pull/72>) subprojects, potentially
> expanding to others, as needed. Basically everything currently at
> https://github.com/google/llvm-bazel.
>
>
> ## Where
>
> In https://github.com/google/llvm-bazelthe BUILD files live in a single
> directory tree matching the structure of the overall llvm-project
> directory. For users, @llvm-project is a single Bazel repository
> <https://docs.bazel.build/versions/master/build-ref.html#repositories >that
> includes both LLVM and MLIR subprojects. To maintain this structure, we
> would probably want to put a `bazel` directory in the monorepo's utils
> directory <https://github.com/llvm/llvm-project/tree/master/utils>,
> which currently only contains a directory for arcanist. This is
> different from gn, which is under the LLVM subproject's utils directory
> <https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn>. We
> could similarly put the Bazel BUILD files under llvm/utils/bazel but
> have them be for the entire llvm project (the subsets that are
> supported). This seems like an odd structure to me, but I know that the
> CMake build for LLVM also builds the other subprojects
> <https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41>,
> so maybe this would be preferable.
>
> Alternatively we could split each subproject into a separate Bazel
> repository and put the Bazel build files under each subproject. I think
> this fragments the configuration of the BUILD without much benefit.
>
>
> ## Configurations
>
> We currently have configurations for Linux GCC and Clang, MacOS GCC and
> Clang, and Windows MSVC. Support for other configurations can be added
> as-desired, but supporting all possible LLVM build configurations is not
> the goal.
>
>
> ## Support
>
> Support would be similar to the gn build. Contributors could optionally
> update the Bazel BUILD files as part of their patches, but would be
> under no obligation to do so.
>
>
> ## Preserving History
>
> I don't *think* the history of llvm-bazel is interesting enough to try
> to merge it into the monorepo and I was planning to submit this as a
> single patch, but please let me know if you disagree.
>
>
> ## Benefits to the community
>
> *
>
> Projects that depend on LLVM and use the Bazel build system can
> avoid duplicating fragile effort. We'll spend more time contributing
> to LLVM instead :-D
>
> *
>
> Bazel is stricter than CMake in many ways (e.g. it requires that
> even header dependencies be declared) and can catch layering issues
> very easily. There's even an optional layering_check feature we
> could turn on if its use would benefit the community. (though
> currently the existing problematic layering makes it a burden to
> maintain on our own). Even without that additional check, as I've
> been keeping the Bazel build green, I've found and fixed a number of
> layering issues in the past couple weeks (e.g.
> https://reviews.llvm.org/rGb49787df9a
> Here's a patch <https://reviews.llvm.org/D90352 >adding the Bazel build
> system. It's basically just `cp -r llvm-bazel/llvm-bazel
> llvm-project/utils/bazel`.
>

David Blaikie via llvm-dev

unread,
Oct 29, 2020, 9:12:15 PM10/29/20
to Stefan Teleman, LLVM Dev
On Thu, Oct 29, 2020 at 4:49 PM Stefan Teleman <stefan....@gmail.com> wrote:
On Thu, Oct 29, 2020 at 7:16 PM David Blaikie <dbla...@gmail.com> wrote:

> I expect most of it is probably a statement free of value judgments: Some other projects chose to use it/some folks have to use it for other reasons, clearly there's enough use that it's motivated folks to have/maintain Bazel builds for LLVM for years. Rather than judging their choices as bad/lesser/wrong - might be useful to accept that some folks had their reasons and they're trying to make the most of the situation. I don't think anyone's making an argument that LLVM should switch to Bazel/that that would be better than the CMake we're using, and I think it's helpful to return the favor and not suggest that other projects would be better off switching to CMake over Bazel - they no doubt have their reasons.

Please do not manufacture statements that I did not make. I never
suggested, or stated, anywhere, that some other imaginary project
using Bazel should switch to CMake.

Sorry, that seems to be the question though - other projects that have llvm as a dependency use Bazel. No one suggested that Bazel was better than CMake or that Bazel should be used instead of CMake.
 
I did state that I do not find Bazel to be a better alternative to
CMake. My statement is based on direct experience with both. 

If the intent behind Bazel is not to present it as a better
alternative to CMake, then what is the intent?

The original proposal seemed to outline the intent and motivation - that other projects with LLVM as a dependency use Bazel and would benefit from having Bazel build files for LLVM in a central place to work on them - not to replace the CMake build system. Same as the 'gn' integration - no one's suggesting it's better, just that it's what some other projects that use as LLVM as a dependency do use Bazel.

"As you may know, Google uses Bazel internally and has maintained a Bazel BUILD of LLVM for years. Especially with the introduction of MLIR, we've got more and more OSS projects with a Bazel BUILD depending on LLVM (e.g. IREE and TensorFlow). We're also not the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM that they've borrowed from TF. Each of these projects has to jump through some weird hoops to keep their version of the Bazel BUILD files in sync with the code, which requires some fragile combination of scripts and human intervention. Instead, we'd like to move general-purpose Bazel BUILD files into the LLVM Project monorepo. We expect to follow the model of the GN build where these will be maintained by interested contributors rather than expecting the general community to maintain them."
 
Instead of maintaining
this impenetrable mystery as to why a Bazel build system should be
included in LLVM, please take the time to advocate for Bazel with
technical facts, than "someone at Google really likes it".

That's the technical facts though: A variety of other projects with LLVM as a dependency use Bazel, for whatever their reasons, and are currently maintaining Bazel build files out of tree and it would be easier for them to coordinate in-tree instead.
 
Just because someone likes and maintains an alternative build system
for LLVM, somewhere, that does not automatically mean, or imply that
it should be upstreamed.

For all I know, someone might be building their fork of LLVM with
autoconf. I am sure they have their own very good reasons for doing
so. Should we, therefore, bring back autoconf?

If there were a diversity of involved parties and they were proposing to add it in the same way as the gn build system (importantly: not the same way autoconf was maintained previously, where it was on equal footing with the CMake build - with emailing buildbots, etc) - yeah, that seems plausible to me.

- Dave 

David Blaikie via llvm-dev

unread,
Oct 29, 2020, 9:14:37 PM10/29/20
to Krzysztof Parzyszek, llvm...@lists.llvm.org
On Thu, Oct 29, 2020 at 4:57 PM Krzysztof Parzyszek via llvm-dev <llvm...@lists.llvm.org> wrote:

On the grounds that it was a bad idea after all.


Could you clarify this a bit further? Are there particular outcomes of that decision now that we've got some practical experience with it for a while now that was unanticipated in the original proposal/acceptance? What sort of ongoing costs/pain do you find the gn build system is causing the LLVM project?
 
_______________________________________________

Stefan Teleman via llvm-dev

unread,
Oct 29, 2020, 9:28:33 PM10/29/20
to David Blaikie, LLVM Dev
On Thu, Oct 29, 2020 at 9:12 PM David Blaikie <dbla...@gmail.com> wrote:


>> Instead of maintaining
>> this impenetrable mystery as to why a Bazel build system should be
>> included in LLVM, please take the time to advocate for Bazel with
>> technical facts, than "someone at Google really likes it".
>
>
> That's the technical facts though: A variety of other projects with LLVM as a dependency use Bazel, for whatever their reasons, and are currently maintaining Bazel build files out of tree and it would be easier for them to coordinate in-tree instead.

I fail to see how any of these are technical facts. Whatever "variety
of other projects with LLVM as a dependency" choose to use for their
build system is their business.

Let's be a bit more precise here: this "variety of other projects with
LLVM as a dependency" aren't just random projects off the Internet.
These are all Google projects. Correct?

So, in final analysis, this has nothing to do with Bazel's technical
merits. It has everything to do with "It's convenient for Google".
Regardless of whether the larger LLVM community agrees with the idea,
or not. Which, so far, it does not seem to me that it has.

Thanks for clarifying.

Johannes Doerfert via llvm-dev

unread,
Oct 29, 2020, 9:44:56 PM10/29/20
to Geoffrey Martin-Noble, llvm...@lists.llvm.org, Mehdi Amini, Stella Laurenzo, Thomas Joerg, Tres Popp
(see below)


On 10/28/20 6:18 PM, Geoffrey Martin-Noble via llvm-dev wrote:
> Hi all,
>
> tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a
> side-directory in the monorepo, similar to the gn build.
>
> Some of us have been working on open-source Bazel BUILD files for the
LLVM
> Project. You may have seen us hanging out in the #build-systems discord

> channel. As you may know, Google uses Bazel internally and has

maintained a
> Bazel BUILD of LLVM for years. Especially with the introduction of MLIR,
> we've got more and more OSS projects with a Bazel BUILD depending on LLVM

> (e.g. IREE <https://github.com/google/iree> and TensorFlow

> <https://github.com/tensorflow/tensorflow>). We're also not the only ones


> using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM that they've
borrowed
> from TF

> <https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD>.


> Each of these projects has to jump through some weird hoops to keep their
> version of the Bazel BUILD files in sync with the code, which
requires some
> fragile combination of scripts and human intervention. Instead, we'd like
> to move general-purpose Bazel BUILD files into the LLVM Project monorepo.
> We expect to follow the model of the GN build where these will be
> maintained by interested contributors rather than expecting the general
> community to maintain them.
>

> To facilitate and test this we've been developing a standalone repository
> that just has the Bazel BUILD files. It symlinks together the directory
> trees on top of a submodule as we would need in the monorepo to to avoid
> in-tree BUILD files. The configuration is at
> https://github.com/google/llvm-bazel. We now have those in a good
place and
> think they would be useful upstream.
>

> # Details
>
> ## What
>
> Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review
> <https://github.com/google/llvm-bazel/pull/72>) subprojects, potentially
> expanding to others, as needed. Basically everything currently at
> https://github.com/google/llvm-bazel.
>
> ## Where
>

> In https://github.com/google/llvm-bazel the BUILD files live in a single

>    -


>
>    Projects that depend on LLVM and use the Bazel build system can avoid
>    duplicating fragile effort. We'll spend more time contributing to LLVM
>    instead :-D

>    -


>
>    Bazel is stricter than CMake in many ways (e.g. it requires that even
>    header dependencies be declared) and can catch layering issues
very easily.
>    There's even an optional layering_check feature we could turn on
if its use
>    would benefit the community. (though currently the existing
problematic
>    layering makes it a burden to maintain on our own). Even without that
>    additional check, as I've been keeping the Bazel build green, I've
found
>    and fixed a number of layering issues in the past couple weeks (e.g.
>    https://reviews.llvm.org/rGb49787df9a
> <https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d>
>    and https://reviews.llvm.org/rGc17ae2916c
> <https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a>).
>
>
> Here's a patch <https://reviews.llvm.org/D90352> adding the Bazel build
> system. It's basically just `cp -r llvm-bazel/llvm-bazel
> llvm-project/utils/bazel`.

Doesn't the last paragraph mean all benefits derived from this can be
described either as:
  (1) users do not need to clone the llvm-bazel git repo but get the
      files in llvm-project, or
  (2) "interested contributors" could send patches to llvm-project
      instead of llvm-bazel to update the bazel build.

TBH, I have no interest in using bazel nor anything against it being
merged per se. I just find it curious that we merge another build system
"at no cost" for the community (I think I picked that up in the thread
but I might have imagined the phrasing). I mean, there is always "a
cost"* so it boils down to determine if the benefit is worth it.

~ Johannes


* i.a., people will assume we (=the LLVM community) maintain(s) a bazel
  build, which can certainly be a benefit but also a cost", e.g., when
  the build is not properly maintained, support is scarce, etc. and
  emails come in complaining about it (not thinking of prior examples
  here.)

Johannes Doerfert via llvm-dev

unread,
Oct 29, 2020, 10:30:19 PM10/29/20
to Mehdi AMINI, Chris Tetreault, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
I replied only selectively.

Wouldn't your argument hold for anything that "just lives" in the mono
repo but doesn't impact people? I mean, where is the line for stuff that
some contributors have "strong interest" in and others can't really
"hear a compelling argument for inclusion"? People raise concerns here
and from where I am sitting they are brushed over easily and more
aggressively as the thread progresses (up to the email I respond to).


>
>
>> I guess it would make the lives of google employees easier?
>>
> I explained before that Google internal integration flow is likely better
> without this at the moment, TensorFlow itself is also in a reasonably good
> spot at the moment. But Google is also not a monolithic place, some people
> are working on small independent projects that they are open-sourcing, and
> would like to be able to use LLVM.
>
>> Then what’s to stop every large org from committing their internal stuff
> to master?
>
>
>
> If their "internal stuff" is highly-coupled to LLVM, has zero-cost
> maintenance on the community, and is something that multiple other parties
> can benefit and established members of the community want to maintain and
> collaborate on, why not?

Let's be honest, nothing has "zero-cost". It seems unhelpful to pretend
it does. (FWIW, I explained a simple scenario that would make the bazel
inclusion "costly" in my previous mail.)


>
> I mentioned it before, but Bazel is not something internal or specific to
> Google: it isn't (actually there are many incompatibilities between Bazel
> and the internal system), 400 people attended the Bazel conference last
> year. I attended this conference 3 years ago when I was at Tesla trying to
> deploy Bazel internally. Many other companies are using Bazel, open-source
> projects as well. Feel free to watch the talks online about SpaceX

> <https://www.youtube.com/watch?v=t_3bckhV_YI> or Two Sigma and Uber
> <https://www.youtube.com/watch?v=_bPyEbAyC0s> for example

Let's not conflate "using bazel" and "benefit for LLVM", the former
is not up for debate here. (I mean, a lot of people use autoconf but
we got rid of it anyway).

That said, I think the original question is highly relevant. As I also
mentioned somewhere above, where do we draw the line is the key to this
RFC at the end of the day. A lot of the arguments I hear pro integration
apply to various other things that currently live out-of-tree, some of
which were proposed and not integrated. I think we should not dismiss
this easily, no matter on which side of the argument you are this time.

~ Johannes

>
>
> I'm not trying to convince anyone to use Bazel, it has drawbacks, but the
> point here is to recognize that this is about OpenSource communities that
> Bazel is serving: these are users, some of us in the LLVM community are
> trying to provide these users with a reasonably good integration story, and
> we're ready to pay the cost for everyone.
>
>
>
>>

>> *From:* Mehdi AMINI <joke...@gmail.com>
>> *Sent:* Thursday, October 29, 2020 2:00 PM
>> *To:* Chris Tetreault <ctet...@quicinc.com>
>> *Cc:* Sterling Augustine <saugu...@google.com>; Mehdi Amini <


>> ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <
>> laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble
>> <gc...@google.com>; Thomas Joerg <tjo...@google.com>

>> *Subject:* [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to

>> *From:* Sterling Augustine <saugu...@google.com>
>> *Sent:* Thursday, October 29, 2020 1:14 PM
>> *To:* Chris Tetreault <ctet...@quicinc.com>
>> *Cc:* Renato Golin <reng...@gmail.com>; tste...@redhat.com; Mehdi Amini


>> <ami...@google.com>; LLVM Dev <llvm...@lists.llvm.org>; Stella Laurenzo <
>> laur...@google.com>; Tres Popp <tp...@google.com>; Geoffrey Martin-Noble
>> <gc...@google.com>; Thomas Joerg <tjo...@google.com>

>> *Subject:* [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to

Eric Christopher via llvm-dev

unread,
Oct 29, 2020, 10:40:17 PM10/29/20
to Johannes Doerfert, Mehdi Amini, llvm-dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
Absolutely. This could happen. The main reason behind this is to make integating among a number of llvm based projects that use bazel (TF and TF-based projects primarily, though it sounds like FB's internal process would be helped as their system is similar to bazel).
 
TBH, I have no interest in using bazel nor anything against it being
merged per se. I just find it curious that we merge another build system
"at no cost" for the community (I think I picked that up in the thread
but I might have imagined the phrasing). I mean, there is always "a
cost"* so it boils down to determine if the benefit is worth it.


As far as I can think the cost is...
 

* i.a., people will assume we (=the LLVM community) maintain(s) a bazel
   build, which can certainly be a benefit but also a cost", e.g., when
   the build is not properly maintained, support is scarce, etc. and
   emails come in complaining about it (not thinking of prior examples
   here.)


... this. If the system becomes a source of problems or user complaints then I think it's absolutely reasonable to remove it.

-eric

Eric Christopher via llvm-dev

unread,
Oct 29, 2020, 10:43:40 PM10/29/20
to Johannes Doerfert, Mehdi Amini, Stella Laurenzo, Tres Popp, LLVM Dev, Geoffrey Martin-Noble, Thomas Joerg


On Thu, Oct 29, 2020 at 10:30 PM Johannes Doerfert via llvm-dev <llvm...@lists.llvm.org> wrote:
I replied only selectively.


:)
 

Let's not conflate "using bazel" and "benefit for LLVM", the former
is not up for debate here. (I mean, a lot of people use autoconf but
we got rid of it anyway).


To be fair the build system maintainer at the time (me) didn't want to maintain autoconf anymore especially when we had cmake and no one wanted to step forward to maintain it. :)

-eric

Mehdi AMINI via llvm-dev

unread,
Oct 29, 2020, 10:49:34 PM10/29/20
to Johannes Doerfert, Mehdi Amini, Stella Laurenzo, Tres Popp, LLVM Dev, Geoffrey Martin-Noble, Thomas Joerg
Sorry, I invite you to reread the thread again and revisit your impression: Tom and Renato expressed clear concerns, and I believe I really tried to listen and address these with concrete proposals to mitigate: http://lists.llvm.org/pipermail/llvm-dev/2020-October/146182.html
However there is not much I can do to address folks who object because "they don't see the interest" in it, this isn't a productive way of moving forward with such proposal IMO.
 
 
>> I guess it would make the lives of google employees easier?
>>
> I explained before that Google internal integration flow is likely better
> without this at the moment, TensorFlow itself is also in a reasonably good
> spot at the moment. But Google is also not a monolithic place, some people
> are working on small independent projects that they are open-sourcing, and
> would like to be able to use LLVM.
>
>>   Then what’s to stop every large org from committing their internal stuff
> to master?
>
>
>
> If their "internal stuff" is highly-coupled to LLVM, has zero-cost
> maintenance on the community, and is something that multiple other parties
> can benefit and established members of the community want to maintain and
> collaborate on, why not?

Let's be honest, nothing has "zero-cost".

I hope you're not implying I'd be dishonest here right?
 
It seems unhelpful to pretend it does. (FWIW, I explained a simple scenario that would make the bazel
inclusion "costly" in my previous mail.)

"zero-cost" is well defined: it is "as a community member: feel free to ignore, no one will bother you about it", and a subset of the community signed up for the maintenance. 
I think it is also helpful to be concrete here: we have existing data and history with `gn`, it isn't hypothetical.

To be sure I address your previous email, that was about user expectations right? i.e. was it this part:

> people will assume we (=the LLVM community) maintain(s) a bazel build, which can certainly be a benefit but also a cost", e.g., when the build is not properly maintained, support is scarce, etc. and emails come in complaining about it (not thinking of prior examples here.)

Isn't this similar to the concerns from Renato here: http://lists.llvm.org/pipermail/llvm-dev/2020-October/146179.html ?
I acknowledge this as very valid concerns and offered some possibility to mitigate: http://lists.llvm.org/pipermail/llvm-dev/2020-October/146188.html

 


>
> I mentioned it before, but Bazel is not something internal or specific to
> Google: it isn't (actually there are many incompatibilities between Bazel
> and the internal system), 400 people attended the Bazel conference last
> year. I attended this conference 3 years ago when I was at Tesla trying to
> deploy Bazel internally. Many other companies are using Bazel, open-source
> projects as well. Feel free to watch the talks online about SpaceX
> <https://www.youtube.com/watch?v=t_3bckhV_YI> or Two Sigma and Uber
> <https://www.youtube.com/watch?v=_bPyEbAyC0s> for example

Let's not conflate "using bazel" and "benefit for LLVM", the former
is not up for debate here. (I mean, a lot of people use autoconf but
we got rid of it anyway).

I doubt we wouldn't have got rid of Autoconf if a chunk of the community offered to maintain it at "no cost" (again see definition).

 

That said, I think the original question is highly relevant. As I also
mentioned somewhere above, where do we draw the line is the key to this
RFC at the end of the day. A lot of the arguments I hear pro integration
apply to various other things that currently live out-of-tree, some of
which were proposed and not integrated.

Can you provide more concrete reference to these things that could have been integrated in similar "zero cost" fashion?
I'm all for consistency, and the only point of comparison here is `gn`.

Johannes Doerfert via llvm-dev

unread,
Oct 29, 2020, 11:10:19 PM10/29/20
to Mehdi AMINI, Mehdi Amini, Stella Laurenzo, Tres Popp, LLVM Dev, Geoffrey Martin-Noble, Thomas Joerg

Long story short, I did not try to imply you were dishonest.

I'm saying that the sentence "has zero-cost maintenance on the community"
cannot be true in a general sense but only in a narrow one. I believe that
everything has cost. I added, "let's be honest", because the cost is not
obvious and one can easily overlook it. However, I assumed we all know
there has to be one as it would otherwise conflict with some universal
law or something. The way I see it you acknowledge the existence in a few
other places.

It broke, ppl complained, and nobody wanted to fix it. That is the
kind of technical debt (aka. cost) you can accumulate.


>
>> That said, I think the original question is highly relevant. As I also
>> mentioned somewhere above, where do we draw the line is the key to this
>> RFC at the end of the day. A lot of the arguments I hear pro integration
>> apply to various other things that currently live out-of-tree, some of
>> which were proposed and not integrated.
>
> Can you provide more concrete reference to these things that could have
> been integrated in similar "zero cost" fashion?
> I'm all for consistency, and the only point of comparison here is `gn`.

Let's say RV, in a subfolder not build by default. Or any other
project that was proposed for inclusion without being build by
default. (I remember also the discussion if we can/should add
isl to llvm, pre-mono repo.)

Eric Astor via llvm-dev

unread,
Oct 29, 2020, 11:19:12 PM10/29/20
to Eric Christopher, Mehdi Amini, llvm-dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
I *am* a Googler, though not directly involved with the teams that maintain the internal LLVM build. I happen to be a big fan of Bazel - and mostly build LLVM with the internal Bazel build, rather than the external CMake, because the better caching and remote-build-farm support is such an enormous help. (Also, I find the CMake build & build options kind of impenetrable.) However, I'm writing this particular email on my personal account, with personal resources, well past the close of business; my Google hat is firmly on the shelf, and I'm speaking as just an individual contributor.

When I first started contributing to LLVM, I was confused by the GN build's existence. I didn't understand who was supposed to maintain it, whether I should use it, what the benefits were... you name it.

I agree with some of the first comments on this thread. I'd suggest that we set aside the question of contributing Bazel BUILD files into the LLVM repository for now, and start by proposing a general policy around alternate/unsupported build systems in relation to the main repository. (GN can have an exception if needed.) The fact that the GN build is basically working, and doesn't confuse too many people, is a data point - but going from 1 alternate build system to 2 seems like a good point to pause and set an actual set of constraints and goals. Eventually, someone may want a third, and we should know what the guidelines are so we don't hash out the decision from scratch again!

I don't think I could draft the RFC in question - I don't have enough experience with the community yet to judge what's really needed - but I'd be glad to help out with it. The idea should be to minimize the cost (to nearly zero) for both experienced LLVM contributors and new LLVM contributors. A few requirements I'd suggest, mostly put together from this thread:
  • CMake should be able to build (and test!) everything the alternate build system can, at all times.
  • There must be a clear group who want to maintain the alternate build system.
  • The alternate build system's files should be isolated in a separate directory, with a README explaining that this is an alternate build system for LLVM, maintained by its own smaller community - and is not supported by the community at large.
  • The alternate build system must have independent buildbots, which do not email the larger community; people can opt into being emailed about this. (And should, if they're contributing to it!)
  • If the buildbots are red for an extended time, we should put out a call for maintainers to fix the issues; if not answered in a reasonable time, we shouldn't be afraid to delete the alternate build system.
I do also see the argument for the git submodule approach. It looks like a .gitmodules file would theoretically let a repository of Bazel BUILD files specify exactly which LLVM commit it currently tracks - and you could fetch the corresponding updates in both with a single command. I think that addresses the main point I noticed brought up on this side of the argument. Any RFC here probably needs to present pros & cons of both approaches. We'll need to hash those out in general discussion before people start looking for consensus, so people understand what they're deciding on.

Best,
- Eric

Mehdi AMINI via llvm-dev

unread,
Oct 29, 2020, 11:39:03 PM10/29/20
to Johannes Doerfert, Mehdi Amini, Stella Laurenzo, Tres Popp, LLVM Dev, Geoffrey Martin-Noble, Thomas Joerg
Yes, I know you :)
(actually I thought I included a wink smiley above, but apparently not, sorry about that)
I don't know what RV is? 
 
Or any other
project that was proposed for inclusion without being build by
default. (I remember also the discussion if we can/should add
isl to llvm, pre-mono repo.)

I am not sure I agree that we can compare new "projects" (or something like ISL) with "utilities for LLVM users".
I would expect a more comparable situation to me to be: 
- the gdb scripts in llvm/utils/gdb-scripts/prettyprinters.py
- IDE visualizer in llvm/utils/LLVMVisualizers
- The Visual Studio Code syntax highlighting for LLVM IR and TableGen in llvm/utils/vscode ; and similar for kate, jedit, vim, textmate, ...
- the gn files in llvm/utils/gn

The general theme here is that these are not "new projects" in themselves: they are highly coupled to LLVM itself and only allow a specific subset of users to plug their tool/workflow into LLVM at a given revision.
Also all of these are "zero cost" in that they may be "broken" and maintained with best effort (I don't think we revert someone breaking any of the visualizer or syntax highlighter?). And none of these are really core to LLVM, and each could be in a separate repo where the interested parties could maintain it.

Best,

-- 
Mehdi

Eric Christopher via llvm-dev

unread,
Oct 29, 2020, 11:48:28 PM10/29/20
to Mehdi AMINI, Mehdi Amini, LLVM Dev, Stella Laurenzo, Tres Popp, Geoffrey Martin-Noble, Thomas Joerg
FWIW this is the compelling argument to me and why I've also relaxed significantly over the years on this sort of thing. It's cheap to remove code when it's not useful anymore and we can do our best with documentation to handle a lot of the new user problems (as Eric Astor mentioned). Probably even a README in the various directories making sure that people know that cmake is the supported system "just in case they didn't read the documentation" :)

-eric

Johannes Doerfert via llvm-dev

unread,
Oct 29, 2020, 11:59:07 PM10/29/20