Minimum supported version of LLVM

[Nathan Chancellor]

Hi all,

I am starting this thread to spur some discussion about our support
model for the minimum supported version of LLVM and how that might
impact bumping the minimum supported version of LLVM/Clang in the
future. I am adding what I hope are relevant parties (individual testers
and maintainers/lists of CI systems). Feel free to add anyone that I
might have missed.

Some history:

For the longest time, ClangBuiltLinux as a project rejected the idea of
a minimum supported version as we did not have as many resources for
testing or a good idea of "what works" and "what doesn't work". After
Linux Plumbers 2020, we decided that we would support from clang 10.0.1
on, which resulted in commit 1f7a44f63e6c ("compiler-clang: add build
check for clang 10.0.1").

The problem:

I would say there are three different levels of support:

1. CC=clang: Compiling with clang, GNU binutils for everything else
2. LLVM=1: Compiling with clang, LLVM binutils, GNU as
3. LLVM=1 LLVM_IAS=1: Compiling with clang, LLVM binutils, and LLVM's
integrated assembler.

We have started to address these different levels of support in the LLVM
documentation in the kernel:

https://lore.kernel.org/linux-doc/20210114003447.736...@gmail.com/

The issue that has been coming up more recently is whether or not the
minimum supported version of clang (the first tier of support) equals
the minimum supported version of LLVM (the second and third tier of
support) when it comes to inserting workarounds or disabling configs
that are known broken.

Some considerations:

1. LLVM 10.0.1 has a few issues that are not present in LLVM 11.x that
will or have required workarounds in the kernel:

* https://github.com/ClangBuiltLinux/linux/issues/732
* https://github.com/ClangBuiltLinux/linux/issues/1187

2. There are still outstanding issues with LLVM utilities that prevent
recommending LLVM=1 for some architectures so committing to a minimum
supported version might mean we would need to insert more workarounds
as time goes on. For example:

* ld.lld for MIPS big-endian:
https://github.com/ClangBuiltLinux/linux/issues/1025
* ld.lld and llvm-objdump for PowerPC64 big-endian:
https://github.com/ClangBuiltLinux/linux/issues/602
https://github.com/ClangBuiltLinux/linux/issues/666
* ld.lld for PowerPC64 little-endian:
https://github.com/ClangBuiltLinux/linux/issues/811
* ld.lld for RISC-V:
https://github.com/ClangBuiltLinux/linux/issues/1020

Questions to address:

I believe the first place to start is answering the question is there a
minimum supported version of LLVM? Is it LLVM as a whole or one
particular utility (such as ld.lld)?

If there is a minimum supported version of LLVM or ld.lld, is it the
same as clang?

If there is not a minimum supported version of LLVM, do we insert
workarounds for issues that we know are fixed with newer versions of
LLVM or do we just tell people to upgrade the version of LLVM they are
using?

If we decide that it is worth defining a minimum supported version of
LLVM, I would vote that it be LLVM 11.0.0. If we decide that that
version should stay in sync with the supported clang version, then I
would vote that we update that for Linux 5.12. Taking a look across
various distributions:

archlinux/base:latest: clang version 11.0.1
debian:stable-slim: clang version 7.0.1-8+deb10u2 (tags/RELEASE_701/final)
debian:testing-slim: Debian clang version 11.0.1-2
debian:unstable-slim: Debian clang version 11.0.1-2
fedora:latest: clang version 11.0.0 (Fedora 11.0.0-2.fc33)
fedora:rawhide: clang version 11.1.0 (Fedora 11.1.0-0.3.rc2.fc34)
opensuse/leap:latest: clang version 9.0.1
opensuse/tumbleweed:latest: clang version 11.0.1
ubuntu:18.04: clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
ubuntu:20.04: clang version 10.0.0-4ubuntu1
ubuntu:latest: clang version 10.0.0-4ubuntu1
ubuntu:rolling: Ubuntu clang version 11.0.0-2
ubuntu:devel: Ubuntu clang version 11.0.1-2

I am not so concerned for Ubuntu and Debian, as there are updated
versions of LLVM available from apt.llvm.org and OpenSUSE Leap is
already unable to build the upstream kernel with its version of
clang. Everyone else should be unaffected by that bump.

Hopefully all of that makes sense and spurs some good discussion.

Cheers,
Nathan

[Tom Stellard]

I think it's going to be difficult to support a minimum version of LLVM for an
extended period of time. The reason being that upstream only 'supports' a
released version of LLVM for 3 months (i.e. 12.0.0 is released, 3 months later
12.0.1 is released, and then there are no more 12.x releases).

My recommendation would be to support 2 versions only: the latest released
versions (e.g. right now this is 11.0.1) and the latest snapshot build from
git. Any version older than that is just going to continue to accumulate
workarounds in the kernel, since upstream is no longer fixing bugs.

If you want to try to support more versions than 2, then I think it would be
worth starting a conversation with upstream LLVM about what it would take to
extend the support lifetime of the releases.

> I believe the first place to start is answering the question is there a
> minimum supported version of LLVM? Is it LLVM as a whole or one
> particular utility (such as ld.lld)?
>

I would recommend using the same minimum supported version for all of the
LLVM sub-projects (e.g. llvm, clang, and lld). Most of the time users have the
same versions of all these tools installed anyway, and toolchains that mismatch
versions of clang and lld don't likely get a lot of testing.

-Tom

[Miguel Ojeda]

On Fri, Jan 29, 2021 at 10:20 PM Nathan Chancellor <nat...@kernel.org> wrote:
>
> Questions to address:
>
> I believe the first place to start is answering the question is there a
> minimum supported version of LLVM? Is it LLVM as a whole or one
> particular utility (such as ld.lld)?

Would this be the minimum version for LLVM=1 (in supported
architectures)? If yes, then yes (eventually). In other words, I think
there should be one minimum version for LLVM=1 eventually, even if for
the moment we only have one for CC=clang.

> If there is a minimum supported version of LLVM or ld.lld, is it the
> same as clang?

I would try to aim for that, yes. I think having it undefined for the
moment is fine, but when LLVM=1 works for all architectures that we
care for, then the minimum should be the same for both CC=clang and
LLVM=1 to simplify things and avoid regressing.

> If there is not a minimum supported version of LLVM, do we insert
> workarounds for issues that we know are fixed with newer versions of
> LLVM or do we just tell people to upgrade the version of LLVM they are
> using?

Until the minimum is defined, I'd simply tell them to use the latest
one for LLVM=1.

> If we decide that it is worth defining a minimum supported version of
> LLVM, I would vote that it be LLVM 11.0.0. If we decide that that
> version should stay in sync with the supported clang version, then I
> would vote that we update that for Linux 5.12. Taking a look across
> various distributions:

Well, this depends on how we look at it. Ideally, we want to reach a
point where we don't bump LLVM/Clang minimum versions unless there is
a reason for it, i.e. the same policy like we do so for GCC. That is
why I think it wouldn't hurt to wait to define a minimum version for
LLVM=1 until we are reasonably sure we can keep it stable for a while.

Having said that, bumping versions for LLVM is not as big of a deal as
for GCC so far, since the support is recent and distributions are not
building the kernel with it just yet (AFAIK, at least).

Cheers,
Miguel

[Fangrui Song]

Hi Nathan,

Thanks for starting this topic.

[Facepalm] I know really little about mips...

> * ld.lld and llvm-objdump for PowerPC64 big-endian:

> https://github.com/ClangBuiltLinux/linux/issues/602

LLD has removed incomplete and legacy PPC64 ELFv1 support and may never
support it in the future.

> https://github.com/ClangBuiltLinux/linux/issues/666

Yes, this require llvm-objdump 11.0.0 (display style issue).

> * ld.lld for PowerPC64 little-endian:
> https://github.com/ClangBuiltLinux/linux/issues/811

This is due to GNU ld and LLD's different -z notext semantics.
It is not relevant to LLD versions.

I have a bit more decription in
https://maskray.me/blog/2020-12-19-lld-and-gnu-linker-incompatibilities

> * ld.lld for RISC-V:
> https://github.com/ClangBuiltLinux/linux/issues/1020

Not relevant to LLD versions.
R_RISCV_ALIGN support may take an extended period of time.
In the meantile, users can use -mno-relax as we already notified the CI
maintainers(?).


Other relevant commits between 10.x and 11.x:

LLD
https://github.com/ClangBuiltLinux/linux/issues/1186
https://github.com/ClangBuiltLinux/linux/issues/951
https://github.com/ClangBuiltLinux/linux/issues/953
https://github.com/ClangBuiltLinux/linux/issues/773
https://github.com/ClangBuiltLinux/linux/issues/812


There are some commits between 11.x -> 12.x but they are less concerned:

https://github.com/ClangBuiltLinux/linux/issues/1260 (R_PPC64_ADDR16_HIGH)
https://github.com/ClangBuiltLinux/linux/issues/1186 (only with LTO)
https://github.com/ClangBuiltLinux/linux/issues/1141 (the input was already corrupted)

>Questions to address:
>
>I believe the first place to start is answering the question is there a
>minimum supported version of LLVM? Is it LLVM as a whole or one
>particular utility (such as ld.lld)?
>
>If there is a minimum supported version of LLVM or ld.lld, is it the
>same as clang?
>
>If there is not a minimum supported version of LLVM, do we insert
>workarounds for issues that we know are fixed with newer versions of
>LLVM or do we just tell people to upgrade the version of LLVM they are
>using?
>
>If we decide that it is worth defining a minimum supported version of
>LLVM, I would vote that it be LLVM 11.0.0. If we decide that that
>version should stay in sync with the supported clang version, then I
>would vote that we update that for Linux 5.12. Taking a look across
>various distributions:

I vote 11.0.0 as a minimum supported version, too. I don't maintain a CI
system. Note LTO/R_PPC64_ADDR16_HIGH would be better with 12.0.0.


Note, LLD needs to be newer than Clang if you use LTO. Newer Clang may
emit bitcode files not recognizable by older LLD. If the bitcode files
are older, it may need IR auto upgrade. This serial process may take
some linker time. Old format + new linker also gets less testing.

If you don't LTO, an older LLD usually works. This combo has very little
testing because most LLD LTO users use lock-stepped Clang+LLD. Some
utilities llvm-nm/llvm-ar/llvm-ranlib understand bitcode files. While
the format is stable it is the best to make these binary utilities
lock-stepped with clang+LLD as well.

>archlinux/base:latest: clang version 11.0.1
>debian:stable-slim: clang version 7.0.1-8+deb10u2 (tags/RELEASE_701/final)
>debian:testing-slim: Debian clang version 11.0.1-2
>debian:unstable-slim: Debian clang version 11.0.1-2
>fedora:latest: clang version 11.0.0 (Fedora 11.0.0-2.fc33)
>fedora:rawhide: clang version 11.1.0 (Fedora 11.1.0-0.3.rc2.fc34)
>opensuse/leap:latest: clang version 9.0.1
>opensuse/tumbleweed:latest: clang version 11.0.1
>ubuntu:18.04: clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
>ubuntu:20.04: clang version 10.0.0-4ubuntu1
>ubuntu:latest: clang version 10.0.0-4ubuntu1
>ubuntu:rolling: Ubuntu clang version 11.0.0-2
>ubuntu:devel: Ubuntu clang version 11.0.1-2
>
>I am not so concerned for Ubuntu and Debian, as there are updated
>versions of LLVM available from apt.llvm.org and OpenSUSE Leap is
>already unable to build the upstream kernel with its version of
>clang. Everyone else should be unaffected by that bump.
>
>Hopefully all of that makes sense and spurs some good discussion.
>
>Cheers,
>Nathan
>

>--
>You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to clang-built-li...@googlegroups.com.
>To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/20210129212009.GA2659554%40localhost.

[Arnd Bergmann]

On Fri, Jan 29, 2021 at 10:20 PM Nathan Chancellor <nat...@kernel.org> wrote:
>

> 2. There are still outstanding issues with LLVM utilities that prevent
> recommending LLVM=1 for some architectures so committing to a minimum
> supported version might mean we would need to insert more workarounds
> as time goes on. For example:
>
> * ld.lld for MIPS big-endian:
> https://github.com/ClangBuiltLinux/linux/issues/1025
> * ld.lld and llvm-objdump for PowerPC64 big-endian:
> https://github.com/ClangBuiltLinux/linux/issues/602
> https://github.com/ClangBuiltLinux/linux/issues/666
> * ld.lld for PowerPC64 little-endian:
> https://github.com/ClangBuiltLinux/linux/issues/811
> * ld.lld for RISC-V:
> https://github.com/ClangBuiltLinux/linux/issues/1020

What about armv4 support and big-endian arm32/arm64?

I had disabled lld for those in my randconfig builds last year
and did not expect them to get fixed. Has this changed?

I think pretty much by definition there will always be corner
cases that require a newer version than the minimum, or that
might not work at all.

> If we decide that it is worth defining a minimum supported version of
> LLVM, I would vote that it be LLVM 11.0.0. If we decide that that
> version should stay in sync with the supported clang version, then I
> would vote that we update that for Linux 5.12. Taking a look across
> various distributions:

I agree requiring llvm-11 as the minimum starting with the following
releases makes sense. The last stable release is still fresh, and that
can be built with llvm-10 on many architectures, but requiring llvm-11
now would probably let us remove some workarounds and stop caring
about things breaking in the future.

In the long run, I think we'll want to support more versions,
so if the current stable kernel (5.10) can be built with three versions
(10, 11, 12pre), the next one (5.15?) would go with four ones (11, 12,
13, 14pre), and the one after that could build with five ones
(12, 13, 14, 15, 16pre), all depending on actual technical requirements
that will come up of course.

We probably won't ever want the same 7+ years of old compilers
that Linux supports with gcc, but I would certainly plan on more than
just one or two year old releases.

I'm actually more worried about the reverse: as there is six year
long-term support for kernels, do we expect to be able to build
the oldest kernel with the latest version of llvm at the end of that?

E.g. would a linux-5.4 that today can be built with clang-8 through
clang-12 allow being built with clang-8 through clang-21 at the
end of its life in 2025, or do we assume that LTS kernel users also
have to use old compilers?

Arnd

[Nathan Chancellor]

Thank you to everyone for their input so far. I am replying to Arnd
because I think we are all generally on the same page so far.

On Sat, Jan 30, 2021 at 12:35:43AM +0100, Arnd Bergmann wrote:
> On Fri, Jan 29, 2021 at 10:20 PM Nathan Chancellor <nat...@kernel.org> wrote:
> >
> > 2. There are still outstanding issues with LLVM utilities that prevent
> > recommending LLVM=1 for some architectures so committing to a minimum
> > supported version might mean we would need to insert more workarounds
> > as time goes on. For example:
> >
> > * ld.lld for MIPS big-endian:
> > https://github.com/ClangBuiltLinux/linux/issues/1025
> > * ld.lld and llvm-objdump for PowerPC64 big-endian:
> > https://github.com/ClangBuiltLinux/linux/issues/602
> > https://github.com/ClangBuiltLinux/linux/issues/666
> > * ld.lld for PowerPC64 little-endian:
> > https://github.com/ClangBuiltLinux/linux/issues/811
> > * ld.lld for RISC-V:
> > https://github.com/ClangBuiltLinux/linux/issues/1020
>
> What about armv4 support and big-endian arm32/arm64?
>
> I had disabled lld for those in my randconfig builds last year
> and did not expect them to get fixed. Has this changed?

I don't believe so (as I commented on the issue tracker) and I don't
expect it to. There is not a ton of bandwidth for lld to support older
architectures or less common endianness combinations, as Fangrui has
commented in the past.

> I think pretty much by definition there will always be corner
> cases that require a newer version than the minimum, or that
> might not work at all.

This is a fair point and I suppose that is what Kconfig is for.

> > If we decide that it is worth defining a minimum supported version of
> > LLVM, I would vote that it be LLVM 11.0.0. If we decide that that
> > version should stay in sync with the supported clang version, then I
> > would vote that we update that for Linux 5.12. Taking a look across
> > various distributions:
>
> I agree requiring llvm-11 as the minimum starting with the following
> releases makes sense. The last stable release is still fresh, and that
> can be built with llvm-10 on many architectures, but requiring llvm-11
> now would probably let us remove some workarounds and stop caring
> about things breaking in the future.

This is generally what I am getting at, namely with the first point in
the considerations section. I would like to avoid sending patches to
workaround lld-10 bugs when we know that lld-11 has them fixed. I would
rather tell people to just use lld-11.

I feel that we as a project have gotten a lot better about caring about
the quality of the stable releases of LLVM because we testing the main
branch while also testing the stable branches and help with getting
fixes in there that are relevant to the kernel. I put a lot of effort
forward to do that for LLVM 10.0.1 and LLVM 11.0.0. Now that our testing
matrix is growing, I think it will be easier to support older versions.

> In the long run, I think we'll want to support more versions,
> so if the current stable kernel (5.10) can be built with three versions
> (10, 11, 12pre), the next one (5.15?) would go with four ones (11, 12,
> 13, 14pre), and the one after that could build with five ones
> (12, 13, 14, 15, 16pre), all depending on actual technical requirements
> that will come up of course.
>
> We probably won't ever want the same 7+ years of old compilers
> that Linux supports with gcc, but I would certainly plan on more than
> just one or two year old releases.

Tom brings up a good point that due to the nature of LLVM's support
period, this is going to be really difficult. With GCC, it is possible
to get fixes into the compiler a couple of years down the line. With
clang, it will usually see two releases: the major release then a minor
stable release three months later. If there is something that is broken
with LLVM, we have no choice than to workaround it or tell people to
upgrade. I have been tempted to start a conversation upstream about
improving that situation but that is a much bigger issue.

> I'm actually more worried about the reverse: as there is six year
> long-term support for kernels, do we expect to be able to build
> the oldest kernel with the latest version of llvm at the end of that?

I think the answer is yes. For example, we can currently build a 4.4
arm64 and x86_64 kernel with clang 13.x.

> E.g. would a linux-5.4 that today can be built with clang-8 through
> clang-12 allow being built with clang-8 through clang-21 at the
> end of its life in 2025, or do we assume that LTS kernel users also
> have to use old compilers?

I think that this depends on if you see a newer compiler as a feature.
Given that fixes for newer versions of GCC are taken into the stable
kernels, I would argue that the same thing should happen for clang (and
currently, it does). However, we have had some people push back on us
for that because clang is the new kid on the block. There is also the
weird question of what is more of a fix versus a new feature. For
example, building an architecture with clang: is that a fix that should
be backported to older kernels or is it a new feature that people should
upgrade their kernel for? Ultimately, I would say that if a compiler
upgrade breaks a build that was working on an older kernel release, the
older kernel release should be fixed or patched in some way, unless the
compiler is clearly buggy or wrong.

Cheers,
Nathan

[Nick Desaulniers]

On Fri, Jan 29, 2021 at 3:36 PM Arnd Bergmann <ar...@kernel.org> wrote:
>
> I'm actually more worried about the reverse: as there is six year
> long-term support for kernels, do we expect to be able to build
> the oldest kernel with the latest version of llvm at the end of that?
>
> E.g. would a linux-5.4 that today can be built with clang-8 through
> clang-12 allow being built with clang-8 through clang-21 at the
> end of its life in 2025, or do we assume that LTS kernel users also
> have to use old compilers?

I would think so. The newer compiler should always be better or an
improvement. Can you think of a case where an old compiler must be
used for older versions of kernel sources? Stable has said they will
always accept patches for newer toolchain support.
--
Thanks,
~Nick Desaulniers

[Arnd Bergmann]

The most common reason is compiler warnings. New compiler versions
tend to add warnings for stuff that used to be accepted by older compilers,
and that's generally a good thing, but it can clutter up your builds if it
happens a lot. We generally backport those warning fixes.

Feature removal is another case, but there is clearly nothing to do
about that: If clang drops support for unmaintained architectures, those
architectures architectures are frozen in time, and presumably mainline
clang won't remove them before the kernel has already dropped them
first.

The hardest part would be kernel code that accidentally relies on
undefined behavior, which can get changed by the compiler. There
are sometimes regressions that cause a kernel not to boot when
built with a newer kernel. If the kernel fix is too complex, e.g. because
it requires dropping support for an older compiler in return, backporting
might get tricky.

Arnd

[Guenter Roeck]

v4.4.y, v4.9.y, and v4.14.y require old versions of gcc for some architectures.

New versions of gcc don't work for all architectures either, even

in the latest kernel, thanks to a variety of gcc bugs. See

https://github.com/groeck/linux-build-test/blob/master/bin/stable-build-arch.sh

for details.

Maybe clang/llvm is going to be better, but I would not bet on it. Maybe we

should archive this exchange and, 10 years from now, dig it out and have

a good laugh (sorry, my sarcasm is strong today).

Guenter

 

--
Thanks,
~Nick Desaulniers


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#1072): https://groups.io/g/kernelci/message/1072
Mute This Topic: https://groups.io/mt/80221965/955378
Group Owner: kernelc...@groups.io
Unsubscribe: https://groups.io/g/kernelci/unsub [gro...@google.com]
-=-=-=-=-=-=-=-=-=-=-=-

[Mathieu Acher]

Hi,

My 2 cents here. I've experimented a bit with older versions of the kernel (4.13, 4.15, 4.16) and gcc 8 over tinyconfig/defconfig.
It leads to some issues:
https://github.com/TuxML/tuxml/issues/30
https://github.com/TuxML/tuxml/issues/31
https://github.com/TuxML/tuxml/issues/32

There is also an interesting bug report by Nathan specifically about 4.14:
https://lkml.org/lkml/2019/4/22/794

My conclusion is that gcc 8 has difficulties for kernel version <= 4.16.
There are some patches here and there.

It seems working on 4.17 though.
Incidentally or not: Linux 4.17 has been released on 3 June 2018, gcc 8.1 on May 2018 ;)

> Can you think of a case where an old compiler must be
> used for older versions of kernel sources?

Yes (see above) and I've not tested on randconfig, which may lead to further bugs/incompatibilities.

I'm also wondering: is it worth testing 4.14 (LTS) with versions of gcc >= 8?
As it is a LTS, it seems useful/needed, but it's a bit technical I would say (patch integration, comprehension of older gcc incompatibilities/bugs, etc.)

Best,


--
Dr. Mathieu ACHER, Associate Professor
Univ Rennes, Inria, CNRS, IRISA, France (DiverSE team)
http://www.mathieuacher.com/

----- Mail original -----
> De: "Guenter Roeck via groups.io" <groeck=googl...@groups.io>
> À: "kernelci" <kern...@groups.io>, "Nick Desaulniers" <ndesau...@google.com>
> Cc: "Arnd Bergmann" <ar...@kernel.org>, "Nathan Chancellor" <nat...@kernel.org>, "Sedat Dilek" <sedat...@gmail.com>,
> "Miguel Ojeda" <oj...@kernel.org>, "Lukas Bulwahn" <lukas....@gmail.com>, "Kees Cook" <kees...@chromium.org>,
> "Marco Elver" <el...@google.com>, "Dmitry Vyukov" <dvy...@google.com>, "Arvind Sankar" <nive...@alum.mit.edu>, "Ilie
> Halip" <ilie....@gmail.com>, "clang-built-linux" <clang-bu...@googlegroups.com>
> Envoyé: Mardi 2 Février 2021 00:30:33
> Objet: Re: Minimum supported version of LLVM

> View/Reply Online (#1075): https://groups.io/g/kernelci/message/1075
> Mute This Topic: https://groups.io/mt/80221965/2531766
> Group Owner: kernelc...@groups.io
> Unsubscribe: https://groups.io/g/kernelci/unsub [mathie...@irisa.fr]
> -=-=-=-=-=-=-=-=-=-=-=-