[llvm-dev] Upcoming change with how libc++, libc++abi and libunwind are being built
231 views
Skip to first unread message
Louis Dionne via llvm-dev
Oct 11, 2021, 12:18:04 PM10/11/21
to llvm-dev, cfe...@lists.llvm.org, Libc++ Dev
Hi folks,
This message is both a heads up and a request for comments on an upcoming change we want to make to the way we build libc++/libc++abi/libunwind.
For a long time now, those three projects have supported various ways to be built:
1. The Project build, where one would use <root>/llvm as the root of the CMake invocation and pass LLVM_ENABLE_PROJECTS.
2. The Standalone build, where one would perform one CMake invocation per project being built, and fill in various CMake variables to tell each project where to find bits of the other projects.
3. The "Runtimes" or "Bootstrapping" build, where one would use <root>/llvm as the root of the CMake invocation and pass LLVM_ENABLE_RUNTIMES. This would result in Clang being built, and then the runtimes being built with that fresh Clang.
4. The "Unified Standalone" build, where one would use <root>/libcxx/utils/ci/runtimes as the root of the CMake invocation and pass LLVM_ENABLE_PROJECTS. This was introduced to work around the fact that the Project build doesn't work on several embedded platforms, and I think it is only used at Apple.
Through the years, this has gotten rather confusing and difficult to support. We currently support all of those on a best-effort basis and we do have build bots checking each of them, however the complexity is often getting in the way of providing proper support and improving the build system of those projects. We've been trying to migrate to a simpler way to build the runtimes that works for all use cases (notably embedded platforms) for a long time, and we recently got to a point where this is possible (thanks mainly to @phosek and @mstorsjo). To summarize what we're aiming for, we'll now have two ways of building libc++/lib++abi/libunwind:
1. The "Default" build, where the CMake invocation is rooted at <root>/runtimes and one uses LLVM_ENABLE_RUNTIMES.
2. The "Bootstrapping" build, where the CMake invocation is rooted at <root>/llvm and one uses LLVM_ENABLE_RUNTIMES (this is the current "Bootstrapping" build unchanged).
If you have been using the "Project", "Standalone" or "Unified Standalone" builds, you should be moving to the new "Default" build. This should be really easy to achieve, basically you can just change LLVM_ENABLE_PROJECTS to LLVM_ENABLE_RUNTIMES, perform a single invocation of CMake instead of multiple invocations, and pass all the LIBCXX_FOO, LIBCXXABI_FOO and LIBUNWIND_FOO options to that single CMake invocation. See the release note in the PR linked below for more details.
We are aiming to deprecate all the old ways to build as soon as possible, and to keep supporting them until after the LLVM 14 release, at which point we would like to remove support for those and get started with making improvements. The review at https://reviews.llvm.org/D111356 enshrines this deprecation in the documentation. For technical discussion on this proposed change, please comment on that review to keep everything in one place.
Cheers,
Louis
Questions you might be asking yourself:
Q: Why doesn't the Project build work for embedded platforms?
A: A lot of CMake code in <root>/llvm expects that we're building for a target with the usual capabilities for a build host, but that isn't the case for several embedded targets. While it doesn't make sense to build LLVM or Clang for those embedded targets (which would never be used as build hosts), it makes sense to build the runtimes for those targets if you are cross-compiling.
Q: Currently, we use the Standalone build and are not required to have all of the Monorepo available. Does this change our ability to do that?
A: Not in the mid term. Building any of the runtimes already requires all the runtimes to be co-located. The only thing you could get away with previously was not having the part in <root>/llvm. With the "Default" build, this will be possible as soon as we move a few CMake helpers from <root>/llvm/cmake to <root>/cmake. Then, you'll only have to have <root>/runtimes, <root>/cmake and the projects you are building.
Q: What's the difference between the "Runtimes build" and the "Bootstrapping build"?
A: None. We're trying to move away from the term "Runtimes build" because it's waaaay too confusing, and doesn't describe the fact that we're really building the runtimes with a just-built Clang.
Q: What's up with naming the new default build the "Default build"?
A: The name sucks, but we had to pick one and the "Runtimes build" was already taken for the build that does a bootstrap. Suggestions are welcome -- basically what this performs is a normal build of the runtimes using your system compiler, like you used to do with LLVM_ENABLE_PROJECTS (but it's better).
Q: What are the benefits of this migration beyond simplifying the code for maintainers?
A: Reducing the number of ways we can build the runtimes means that all the workflows for building the runtimes will become simpler for users (especially vendors). Furthermore, the new "Default" build is as simple as the old Project build, but it is faster to configure and works on embedded platforms. In the near future, it should also be possible to perform the equivalent of an old "Project build" without having all of the monorepo available.
This message is both a heads up and a request for comments on an upcoming change we want to make to the way we build libc++/libc++abi/libunwind.
For a long time now, those three projects have supported various ways to be built:
1. The Project build, where one would use <root>/llvm as the root of the CMake invocation and pass LLVM_ENABLE_PROJECTS.
2. The Standalone build, where one would perform one CMake invocation per project being built, and fill in various CMake variables to tell each project where to find bits of the other projects.
3. The "Runtimes" or "Bootstrapping" build, where one would use <root>/llvm as the root of the CMake invocation and pass LLVM_ENABLE_RUNTIMES. This would result in Clang being built, and then the runtimes being built with that fresh Clang.
4. The "Unified Standalone" build, where one would use <root>/libcxx/utils/ci/runtimes as the root of the CMake invocation and pass LLVM_ENABLE_PROJECTS. This was introduced to work around the fact that the Project build doesn't work on several embedded platforms, and I think it is only used at Apple.
Through the years, this has gotten rather confusing and difficult to support. We currently support all of those on a best-effort basis and we do have build bots checking each of them, however the complexity is often getting in the way of providing proper support and improving the build system of those projects. We've been trying to migrate to a simpler way to build the runtimes that works for all use cases (notably embedded platforms) for a long time, and we recently got to a point where this is possible (thanks mainly to @phosek and @mstorsjo). To summarize what we're aiming for, we'll now have two ways of building libc++/lib++abi/libunwind:
1. The "Default" build, where the CMake invocation is rooted at <root>/runtimes and one uses LLVM_ENABLE_RUNTIMES.
2. The "Bootstrapping" build, where the CMake invocation is rooted at <root>/llvm and one uses LLVM_ENABLE_RUNTIMES (this is the current "Bootstrapping" build unchanged).
If you have been using the "Project", "Standalone" or "Unified Standalone" builds, you should be moving to the new "Default" build. This should be really easy to achieve, basically you can just change LLVM_ENABLE_PROJECTS to LLVM_ENABLE_RUNTIMES, perform a single invocation of CMake instead of multiple invocations, and pass all the LIBCXX_FOO, LIBCXXABI_FOO and LIBUNWIND_FOO options to that single CMake invocation. See the release note in the PR linked below for more details.
We are aiming to deprecate all the old ways to build as soon as possible, and to keep supporting them until after the LLVM 14 release, at which point we would like to remove support for those and get started with making improvements. The review at https://reviews.llvm.org/D111356 enshrines this deprecation in the documentation. For technical discussion on this proposed change, please comment on that review to keep everything in one place.
Cheers,
Louis
Questions you might be asking yourself:
Q: Why doesn't the Project build work for embedded platforms?
A: A lot of CMake code in <root>/llvm expects that we're building for a target with the usual capabilities for a build host, but that isn't the case for several embedded targets. While it doesn't make sense to build LLVM or Clang for those embedded targets (which would never be used as build hosts), it makes sense to build the runtimes for those targets if you are cross-compiling.
Q: Currently, we use the Standalone build and are not required to have all of the Monorepo available. Does this change our ability to do that?
A: Not in the mid term. Building any of the runtimes already requires all the runtimes to be co-located. The only thing you could get away with previously was not having the part in <root>/llvm. With the "Default" build, this will be possible as soon as we move a few CMake helpers from <root>/llvm/cmake to <root>/cmake. Then, you'll only have to have <root>/runtimes, <root>/cmake and the projects you are building.
Q: What's the difference between the "Runtimes build" and the "Bootstrapping build"?
A: None. We're trying to move away from the term "Runtimes build" because it's waaaay too confusing, and doesn't describe the fact that we're really building the runtimes with a just-built Clang.
Q: What's up with naming the new default build the "Default build"?
A: The name sucks, but we had to pick one and the "Runtimes build" was already taken for the build that does a bootstrap. Suggestions are welcome -- basically what this performs is a normal build of the runtimes using your system compiler, like you used to do with LLVM_ENABLE_PROJECTS (but it's better).
Q: What are the benefits of this migration beyond simplifying the code for maintainers?
A: Reducing the number of ways we can build the runtimes means that all the workflows for building the runtimes will become simpler for users (especially vendors). Furthermore, the new "Default" build is as simple as the old Project build, but it is faster to configure and works on embedded platforms. In the near future, it should also be possible to perform the equivalent of an old "Project build" without having all of the monorepo available.
Q: Does this affect other "runtimes" projects like compiler-rt?
A: Not for the time being. We will probably attempt to move those over in the future as well, however for the time being I can only confirm that this works fine for libc++, libc++abi and libunwind, so that's where my deprecation efforts are focused.
Michael Kruse via llvm-dev
Oct 18, 2021, 12:24:10 PM10/18/21
to Louis Dionne, llvm-dev, clang developer list, Libc++ Dev
What should the buildbots do after this change? Should they all use
the bootstrap build? I am worried that this would reduces the test
coverage since it would only build the runtimes with the top-of-tree
clang. As by [1], at least libc++ supports also compilation with gcc
and AppleClang.
the bootstrap build? I am worried that this would reduces the test
coverage since it would only build the runtimes with the top-of-tree
clang. As by [1], at least libc++ supports also compilation with gcc
and AppleClang.
Name suggestion for the "default" build: "single target" build. Since
only the bootstrap build supports multiple targets with
LLVM_RUNTIME_TARGETS.
Am Mo., 11. Okt. 2021 um 11:18 Uhr schrieb Louis Dionne via llvm-dev
<llvm...@lists.llvm.org>:
> _______________________________________________
> LLVM Developers mailing list
> llvm...@lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Louis Dionne via llvm-dev
Oct 18, 2021, 2:06:47 PM10/18/21
to Michael Kruse, llvm-dev, clang developer list, Libc++ Dev
On Mon, Oct 18, 2021 at 12:23 PM Michael Kruse <llv...@meinersbur.de> wrote:
What should the buildbots do after this change? Should they all use
the bootstrap build? I am worried that this would reduces the test
coverage since it would only build the runtimes with the top-of-tree
clang. As by [1], at least libc++ supports also compilation with gcc
and AppleClang.
The build bots that wish to build libc++ with a supported non-clang-trunk compiler
can switch to the "default build", which is basically the equivalent of the Project
build we had before. The procedure for doing that is explained in the release notes
Please notify me or someone in the libcxx Discord channel if you encounter issues
with making such a change. It should be straightforward, but I'd like to make sure
the transition is frictionless for everyone, so let me know if you encounter issues
and we'll try to solve them.
Name suggestion for the "default" build: "single target" build. Since
only the bootstrap build supports multiple targets with
LLVM_RUNTIME_TARGETS.
Interesting suggestion. I'll definitely keep that in mind. I'll open up a review to officialize
the name of the new "default" build and pitch that suggestion there, thanks!
Cheers,
Louis
Michael Kruse via llvm-dev
Oct 18, 2021, 2:50:14 PM10/18/21
to Louis Dionne, llvm-dev, clang developer list, Libc++ Dev
Am Mo., 18. Okt. 2021 um 13:06 Uhr schrieb Louis Dionne <ldio...@gmail.com>:
> On Mon, Oct 18, 2021 at 12:23 PM Michael Kruse <llv...@meinersbur.de> wrote:
> The build bots that wish to build libc++ with a supported non-clang-trunk compiler
> can switch to the "default build", which is basically the equivalent of the Project
> build we had before.
> On Mon, Oct 18, 2021 at 12:23 PM Michael Kruse <llv...@meinersbur.de> wrote:
> The build bots that wish to build libc++ with a supported non-clang-trunk compiler
> can switch to the "default build", which is basically the equivalent of the Project
> build we had before.
Which buildbots will test the "default build" for gcc/AppleClang/Clang
11/Clang 12? Individual buildbot maintainers may be interested in
keeping their buildbot green, not necessarily to ensure coverage.
Michael
Louis Dionne via llvm-dev
Oct 18, 2021, 4:19:28 PM10/18/21
to Michael Kruse, llvm-dev, clang developer list, Libc++ Dev
On Mon, Oct 18, 2021 at 2:49 PM Michael Kruse <llv...@meinersbur.de> wrote:
Am Mo., 18. Okt. 2021 um 13:06 Uhr schrieb Louis Dionne <ldio...@gmail.com>:
> On Mon, Oct 18, 2021 at 12:23 PM Michael Kruse <llv...@meinersbur.de> wrote:
> The build bots that wish to build libc++ with a supported non-clang-trunk compiler
> can switch to the "default build", which is basically the equivalent of the Project
> build we had before.
Which buildbots will test the "default build" for gcc/AppleClang/Clang
11/Clang 12? Individual buildbot maintainers may be interested in
keeping their buildbot green, not necessarily to ensure coverage.
Nowadays, we handle all libc++'s own build bots through BuildKite at https://buildkite.com/llvm-project/libcxx-ci.
That pipeline contains all the configurations that we support officially. If someone has a build bot and
they think they are adding coverage, we'd like to know it so we can look into migrating it to BuildKite
and owning it ourselves.
Otherwise, they should just make whatever change is needed to keep their bot green. In most cases I
would expect the easiest thing to do here would be to use the Bootstrapping build so as to always build
against the latest Clang, however they can also use the "Default build" with a suitably-recent system
compiler if they want to avoid building all of Clang on their bot.
Cheers,
Louis
Michael
Michael Kruse via llvm-dev
Oct 19, 2021, 12:45:34 AM10/19/21
to Louis Dionne, llvm-dev, Libc++ Dev
Am Mo., 18. Okt. 2021 um 15:19 Uhr schrieb Louis Dionne <ldio...@gmail.com>:
> Nowadays, we handle all libc++'s own build bots through BuildKite at https://buildkite.com/llvm-project/libcxx-ci.
> Nowadays, we handle all libc++'s own build bots through BuildKite at https://buildkite.com/llvm-project/libcxx-ci.
I didn't know. Good that there is systematic testing, but I am a bit
concerned about the fracturing of the CI infrastructure (Buildbot,
Green dragon, GitHub actions, Buildkite, ...)
Brian Cain via llvm-dev
Oct 19, 2021, 9:17:10 AM10/19/21
to Louis Dionne, Michael Kruse, llvm-dev, clang developer list
Louis,
Do libcxx/abi, clang code reviews on phabricator trigger these buildkite builds similar to some of the other monorepo content?
-Brian
From: libcxx-dev <libcxx-de...@lists.llvm.org>
On Behalf Of Louis Dionne via libcxx-dev
Sent: Monday, October 18, 2021 3:19 PM
To: Michael Kruse <llv...@meinersbur.de>
Cc: llvm-dev <llvm...@lists.llvm.org>; clang developer list <cfe...@lists.llvm.org>; Libc++ Dev <libcx...@lists.llvm.org>
Subject: Re: [libcxx-dev] [llvm-dev] Upcoming change with how libc++, libc++abi and libunwind are being built
WARNING: This email originated from outside of Qualcomm. Please be wary of any links or attachments, and do not enable macros.
Louis Dionne via llvm-dev
Oct 19, 2021, 12:25:42 PM10/19/21
to Brian Cain, llvm-dev, clang developer list
On Tue, Oct 19, 2021 at 9:17 AM Brian Cain <bc...@quicinc.com> wrote:
Louis,
Do libcxx/abi, clang code reviews on phabricator trigger these buildkite builds similar to some of the other monorepo content?
Yes. There is both a rolling build (every 4 hours I think) building `main`, and also the full CI pipeline runs every time you upload a review on Phabricator that touches libc++, libc++abi or libunwind.
It's been working like that for roughly one year now and it's the best thing ever, we ship way fewer bugs and our development velocity has increased a lot.
Louis
Louis Dionne via llvm-dev
Oct 19, 2021, 12:37:32 PM10/19/21
to Michael Kruse, llvm-dev, Libc++ Dev
On Tue, Oct 19, 2021 at 12:45 AM Michael Kruse <llv...@meinersbur.de> wrote:
Am Mo., 18. Okt. 2021 um 15:19 Uhr schrieb Louis Dionne <ldio...@gmail.com>:
> Nowadays, we handle all libc++'s own build bots through BuildKite at https://buildkite.com/llvm-project/libcxx-ci.
I didn't know. Good that there is systematic testing, but I am a bit
concerned about the fracturing of the CI infrastructure (Buildbot,
Green dragon, GitHub actions, Buildkite, ...)
For libc++/libc++abi/libunwind, it's actually quite simple. There's exactly one place to look, and it's BuildKite. And it runs systematically on all changes.
- Buildbot: I removed all the libc++/libc++abi/libunwind buildbots over the past year.
- Green Dragon: Ditto, I removed everything and we have Mac builders in BuildKite.
- GitHubActions: Nobody's using that yet AFAICT.
I will actually have a talk at the upcoming Dev Meeting where I will advocate for all projects moving to the same model, since we've had so many incredible benefits from doing it in libc++ & friends.
Cheers,
Louis
Michael
Mehdi AMINI via llvm-dev
Oct 19, 2021, 1:39:57 PM10/19/21
to Louis Dionne, llvm-dev, Libc++ Dev
On Tue, Oct 19, 2021 at 9:37 AM Louis Dionne via llvm-dev <llvm...@lists.llvm.org> wrote:
On Tue, Oct 19, 2021 at 12:45 AM Michael Kruse <llv...@meinersbur.de> wrote:Am Mo., 18. Okt. 2021 um 15:19 Uhr schrieb Louis Dionne <ldio...@gmail.com>:
> Nowadays, we handle all libc++'s own build bots through BuildKite at https://buildkite.com/llvm-project/libcxx-ci.
I didn't know. Good that there is systematic testing, but I am a bit
concerned about the fracturing of the CI infrastructure (Buildbot,
Green dragon, GitHub actions, Buildkite, ...)For libc++/libc++abi/libunwind, it's actually quite simple. There's exactly one place to look, and it's BuildKite. And it runs systematically on all changes.
Did you find a solution for having blamelist and blame emails on BuildKite?
Improving the Buildkite UI to have a good log scrubbing to the level of what we have with buildbot is doable, but I didn't really see how to fix the above.
(I'm doing the opposite move right now: leaving BuildKite for MLIR CI to adopt buildbot).
- Buildbot: I removed all the libc++/libc++abi/libunwind buildbots over the past year.- Green Dragon: Ditto, I removed everything and we have Mac builders in BuildKite.- GitHubActions: Nobody's using that yet AFAICT.I will actually have a talk at the upcoming Dev Meeting where I will advocate for all projects moving to the same model, since we've had so many incredible benefits from doing it in libc++ & friends.Cheers,Louis
Michael
_______________________________________________
Michael Kruse via llvm-dev
Oct 19, 2021, 1:45:42 PM10/19/21
to Louis Dionne, llvm-dev, Libc++ Dev
Am Di., 19. Okt. 2021 um 11:37 Uhr schrieb Louis Dionne <ldio...@gmail.com>:
> - GitHubActions: Nobody's using that yet AFAICT.
> - GitHubActions: Nobody's using that yet AFAICT.
https://github.com/llvm/llvm-project/actions
Seems to be used for the release branch. Similarly, I didn't know that
we were using BuildKite for anything else than pre-merge checks before
you mentioned it.
Louis Dionne via llvm-dev
Oct 20, 2021, 12:56:57 PM10/20/21
to Mehdi AMINI, llvm-dev, Libc++ Dev
On Tue, Oct 19, 2021 at 1:39 PM Mehdi AMINI <joke...@gmail.com> wrote:
On Tue, Oct 19, 2021 at 9:37 AM Louis Dionne via llvm-dev <llvm...@lists.llvm.org> wrote:On Tue, Oct 19, 2021 at 12:45 AM Michael Kruse <llv...@meinersbur.de> wrote:Am Mo., 18. Okt. 2021 um 15:19 Uhr schrieb Louis Dionne <ldio...@gmail.com>:
> Nowadays, we handle all libc++'s own build bots through BuildKite at https://buildkite.com/llvm-project/libcxx-ci.
I didn't know. Good that there is systematic testing, but I am a bit
concerned about the fracturing of the CI infrastructure (Buildbot,
Green dragon, GitHub actions, Buildkite, ...)For libc++/libc++abi/libunwind, it's actually quite simple. There's exactly one place to look, and it's BuildKite. And it runs systematically on all changes.Did you find a solution for having blamelist and blame emails on BuildKite?
No, but I haven't tried. My experience so far has been that I've never had the need for a blamelist. Since we test everything pre-commit, you always know when one of your patch fails because you see it before it's been committed, in Phabricator. Actual breakage on main is incredibly rare nowadays and it pretty much only happens when someone (usually myself) is too keen on checking something in before CI is finished (oops!). There's never a lot of suspicious commits on main since the last build, so this isn't a problem we've encountered.
I guess your experience can vary based on the commit traffic to the project and how diligent people are at testing everything pre-commit, of course.
Louis
Mehdi AMINI via llvm-dev
Oct 20, 2021, 1:03:40 PM10/20/21
to Louis Dionne, llvm-dev, Libc++ Dev
There are other aspects:
- as the commit traffic increases, you can't necessarily test everything in sequence (Rust does, but their maintainers also have to manually bundle pull-request together: they are limited to ~7 merges per day), so concurrent merges can interact with each other and break master.
- as the matrix of configurations increase, it does not seems possible to just test everything (HW: PPC, SystemZ, X86, Arm, Arm64, Risc TIMES the OSes Mac, Windows, Linux, FreeBSD TIMES the host compilers gcc 5/6/7/..., clang, MSVC, ... TIMES sanitizers build), plus some things like the multi-stages bootstrap and such.
So in practice yeah I believe blamelist and notification are a must for LLVM and the monorepo, libc++ may be "too small" to be representative of the scale there.
Andrzej Warzynski via llvm-dev
Oct 28, 2021, 4:22:37 AM10/28/21
to Louis Dionne, llvm-dev, cfe...@lists.llvm.org, Libc++ Dev
Hi Louis,
This sounds like a huge simplification, thank you!
With this change, will we still be able to use LLVM_ENABLE_PROJECTS to
build libc++, libc++abi and libunwind? IIUC, you are disabling this
option entirely, right? Will CMake warn us about the
incorrect/deprecated usage? Should the LLVM documentation be updated [1]?
Thank you,
-Andrzej
[1] https://llvm.org/docs/CMake.html
Louis Dionne via llvm-dev
Oct 28, 2021, 9:35:03 AM10/28/21
to Andrzej Warzynski, llvm-dev, clang developer list, Libc++ Dev
On Thu, Oct 28, 2021 at 4:22 AM Andrzej Warzynski <andrzej....@arm.com> wrote:
Hi Louis,
This sounds like a huge simplification, thank you!
With this change, will we still be able to use LLVM_ENABLE_PROJECTS to
build libc++, libc++abi and libunwind? IIUC, you are disabling this
option entirely, right? Will CMake warn us about the
incorrect/deprecated usage? Should the LLVM documentation be updated [1]?
Hi,
The goal is to deprecate and then entirely remove building the runtimes with LLVM_ENABLE_PROJECTS. Libc++'s own documentation has already been updated, but I will update the documentation you linked to and add a warning in the CMake -- thanks for the suggestion!
Cheers,
Louis
David Blaikie via llvm-dev
Oct 28, 2021, 2:41:00 PM10/28/21
to Louis Dionne, llvm-dev, clang developer list, Libc++ Dev
On Thu, Oct 28, 2021 at 6:35 AM Louis Dionne via llvm-dev <llvm...@lists.llvm.org> wrote:
On Thu, Oct 28, 2021 at 4:22 AM Andrzej Warzynski <andrzej....@arm.com> wrote:Hi Louis,
This sounds like a huge simplification, thank you!
With this change, will we still be able to use LLVM_ENABLE_PROJECTS to
build libc++, libc++abi and libunwind? IIUC, you are disabling this
option entirely, right? Will CMake warn us about the
incorrect/deprecated usage? Should the LLVM documentation be updated [1]?Hi,The goal is to deprecate and then entirely remove building the runtimes with LLVM_ENABLE_PROJECTS.
Glad to hear that's on the plan - looking forward to removing the ambiguity/traps.
Reply all
Reply to author
Forward
0 new messages