[llvm-dev] Problems using Clang with LLD on embedded ARM

[Sean Vig via llvm-dev]

Hi,

I stopped into IRC to ask about a problem I've been having using Clang in conjunction with LLD to compile and link for an embedded project on Cortex-M ARM processor.

First, I am able to separately compile with a call to clang and link with a call to lld, but I cannot use clang to link using lld using the -fuse-ld=lld flag. I have the output from `clang -v -fuse-ld=lld -target arm-none-eabi main.c -o main` here [1], where calling `clang -target arm-none-eabi -c main.c -o main.o` and `ld.lld main.o -o main` work fine. Doing the compiling and linking for my host computer works just fine using -fuse-ld=lld, and I verified that it is using LLD by checking the .comments section in the generated elf.

Second, combining LTO with specifying any specific arm architecture seems to break linking with LLD. That is, I compile the object files with LTO (either -flto or -flto=thin) and, in addition to specifying `-target arm-none-eabi`, I specify an architecture specific flag (such as -march=armv7e-m, -mcpu=cortex-m4, or -mthumb, or instead I use -target armv7em-none-eabi), then while calling lld to link I get an error like:
ld.lld: error: main.o: could not infer e_machine from bitcode target triple thumbv7em-none--eabi
LLD 4.0.0

I am working on Linux and these two problems seem to exist on both 4.0rc2 and the current head.

I am really happy with the progress LLVM has made, with the upcoming 4.0 release, I have managed to compile and link programs for my embedded projects without any of the gcc arm toolchain! There are still some shortcomings with the linker scripts compared to ld that make those a bit trickier to work with, even with the added MEMORY section parsing in the current head. Other than the above, the only other bother I've hit is having to re-link with `--oformat binary` to get raw binary output, since there I haven't found an objcopy equivalent in llvm. I can add feature requests for these missing features, as well as the bug reports for the above two issues, on the bugzilla if that is appropriate.

Thanks for the help,

Sean

[1] https://gist.github.com/flacjacket/35d92ea619af96352d5ac06844f2f1b5

[Sean Silva via llvm-dev]

On Sat, Feb 25, 2017 at 9:38 PM, Sean Vig via llvm-dev <llvm...@lists.llvm.org> wrote:

Hi,

I stopped into IRC to ask about a problem I've been having using Clang in conjunction with LLD to compile and link for an embedded project on Cortex-M ARM processor.

First, I am able to separately compile with a call to clang and link with a call to lld, but I cannot use clang to link using lld using the -fuse-ld=lld flag. I have the output from `clang -v -fuse-ld=lld -target arm-none-eabi main.c -o main` here [1], where calling `clang -target arm-none-eabi -c main.c -o main.o` and `ld.lld main.o -o main` work fine. Doing the compiling and linking for my host computer works just fine using -fuse-ld=lld, and I verified that it is using LLD by checking the .comments section in the generated elf.

For that triple, Clang seems to be calling into GCC driver (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it doesn't work like that apparently).

@Renato: Do you know why Clang is forwarding to GCC like this for linking? If we need to live with this arrangement for a long time we should definitely invest in teaching GCC about -fuse-ld=lld

@Davide: is there any chance you could revive that GCC patch to add support for LLD?

(though it does seem like we still fail H.J.'s test case in https://bugs.llvm.org//show_bug.cgi?id=28414 has not yet been fixed. I further reduced it. Seems to be related to symbol versioning. We don't recognize that `.symver bar, foo@@VERS` is a definition of `foo`.

)

Second, combining LTO with specifying any specific arm architecture seems to break linking with LLD. That is, I compile the object files with LTO (either -flto or -flto=thin) and, in addition to specifying `-target arm-none-eabi`, I specify an architecture specific flag (such as -march=armv7e-m, -mcpu=cortex-m4, or -mthumb, or instead I use -target armv7em-none-eabi), then while calling lld to link I get an error like:
ld.lld: error: main.o: could not infer e_machine from bitcode target triple thumbv7em-none--eabi
LLD 4.0.0

If you are building your own LLD, can you see if the attached patch fixes it for you? If not, can you add `--reproduce /tmp/repro.tar` to your link command line and upload repro.tar somewhere for us?

@Renato, do you know if there is any way for us to implement getBitcodeMachineKind in a sane way so that we don't fill in a long tail of cases of inferring EM_* from the triple? It seems that the EM_* choices are buried very deep in the backend, so it might not be feasible :(

 

I am working on Linux and these two problems seem to exist on both 4.0rc2 and the current head.

I am really happy with the progress LLVM has made, with the upcoming 4.0 release, I have managed to compile and link programs for my embedded projects without any of the gcc arm toolchain! There are still some shortcomings with the linker scripts compared to ld that make those a bit trickier to work with, even with the added MEMORY section parsing in the current head.

What kind of issues have you been running into?

Linker scripts are certainly one area that we have been slowly filling in the long tail of compatibility, it would be great if you filed bugs with any issues you encounter! If the files are small enough and you can share them, it can be as simple as just adding `--reproduce /tmp/repro.tar` to your link command line (or setting LLD_REPRODUCE=/tmp/repro.tar in the environment) and attach repro.tar to a bug report whenever you encounter an issue (even if repro.tar is larger, you can try hosting on google drive / dropbox etc., compressing with xz also often helps). Or if it is a parsing problem just attaching the problematic linker script to the bug report might be enough.

 

Other than the above, the only other bother I've hit is having to re-link with `--oformat binary` to get raw binary output, since there I haven't found an objcopy equivalent in llvm. I can add feature requests for these missing features, as well as the bug reports for the above two issues, on the bugzilla if that is appropriate.

Regarding objcopy, that is something that we have wanted to write for a long time (but it has never quite been a priority). As objcopy supports a bewildering set of different options, it would be useful for you to file a bug about what specific options/features you need. We don't really have a bugzilla component for objcopy yet, so you can just throw it under llvm-nm (another one of the binutils, close enough) in https://bugs.llvm.org/enter_bug.cgi?product=tools 

-- Sean Silva

 

Thanks for the help,

Sean

[1] https://gist.github.com/flacjacket/35d92ea619af96352d5ac06844f2f1b5

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Rafael Avila de Espindola via llvm-dev]

Sean Silva via llvm-dev <llvm...@lists.llvm.org> writes:

> For that triple, Clang seems to be calling into GCC driver
> (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
> -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
> selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
> doesn't work like that apparently).

Gosh, that is an old issue. I was sure we had converted to used ld
everywhere and deleted the old code for using the gcc driver as a
linker.

Doing that is probably the correct way to fix this issue. Having gcc
itelf support -fuse-ld=lld would be an awesome independent improvement.

Cheers,
Rafael

[Davide Italiano via llvm-dev]

I can probably find some time to revive that support next week
(sorry it's GDC week this week and I won't likely have time
for anything compiler-related).

--
David

[Sean Vig via llvm-dev]

Hi Sean

On Sun, Feb 26, 2017 at 5:10 AM, Sean Silva <chiso...@gmail.com> wrote:

For that triple, Clang seems to be calling into GCC driver (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it doesn't work like that apparently).

@Renato: Do you know why Clang is forwarding to GCC like this for linking? If we need to live with this arrangement for a long time we should definitely invest in teaching GCC about -fuse-ld=lld

@Davide: is there any chance you could revive that GCC patch to add support for LLD?

(though it does seem like we still fail H.J.'s test case in https://bugs.llvm.org//show_bug.cgi?id=28414 has not yet been fixed. I further reduced it. Seems to be related to symbol versioning. We don't recognize that `.symver bar, foo@@VERS` is a definition of `foo`.

)

If you are building your own LLD, can you see if the attached patch fixes it for you? If not, can you add `--reproduce /tmp/repro.tar` to your link command line and upload repro.tar somewhere for us?

Thanks! I applied that patch and it links with -flto now, but it looks like there are now some other problems. First, it seems to be dropping things that I try to KEEP in the linker script, even when I don't use --gc-sections, namely my interrupt vector table. Second, when using thin LTO, it starts trying to link symbols __aeabi_memclr4 and __aeabi_uidiv, which I don't think it should since I am using -ffreestanding -fno-builtins when compiling (with full lto or no lto, it does not try to link these symbols).

I just sent an email to register for a bugzilla account, I can report there with a minimally reproducible example for these problems.

 

@Renato, do you know if there is any way for us to implement getBitcodeMachineKind in a sane way so that we don't fill in a long tail of cases of inferring EM_* from the triple? It seems that the EM_* choices are buried very deep in the backend, so it might not be feasible :(

What kind of issues have you been running into?

Linker scripts are certainly one area that we have been slowly filling in the long tail of compatibility, it would be great if you filed bugs with any issues you encounter! If the files are small enough and you can share them, it can be as simple as just adding `--reproduce /tmp/repro.tar` to your link command line (or setting LLD_REPRODUCE=/tmp/repro.tar in the environment) and attach repro.tar to a bug report whenever you encounter an issue (even if repro.tar is larger, you can try hosting on google drive / dropbox etc., compressing with xz also often helps). Or if it is a parsing problem just attaching the problematic linker script to the bug report might be enough.

I haven't hit anything that looked like a bug, just features that aren't implemented, or that work slightly differently between ld linker scripts. I'd be happy to make additional feature requests for those on the bug tracker, too.

 

Regarding objcopy, that is something that we have wanted to write for a long time (but it has never quite been a priority). As objcopy supports a bewildering set of different options, it would be useful for you to file a bug about what specific options/features you need. We don't really have a bugzilla component for objcopy yet, so you can just throw it under llvm-nm (another one of the binutils, close enough) in https://bugs.llvm.org/enter_bug.cgi?product=tools 

Sounds good.

Thanks for the patch and the help.

Sean

 

-- Sean Silva

[Joerg Sonnenberger via llvm-dev]

On Sun, Feb 26, 2017 at 04:25:57PM -0600, Sean Vig via llvm-dev wrote:
> Thanks! I applied that patch and it links with -flto now, but it looks like
> there are now some other problems. First, it seems to be dropping things
> that I try to KEEP in the linker script, even when I don't use
> --gc-sections, namely my interrupt vector table. Second, when using thin
> LTO, it starts trying to link symbols __aeabi_memclr4 and __aeabi_uidiv,
> which I don't think it should since I am using -ffreestanding -fno-builtins
> when compiling (with full lto or no lto, it does not try to link these
> symbols).

You still need a runtime library, even with freestanding. Both
__aeabi_memclr4 and __aeabi_uidiv are helpers as defined by EABI and
their presence is required even with freestanding.

Joerg

[Renato Golin via llvm-dev]

On 26 February 2017 at 22:49, Joerg Sonnenberger via llvm-dev

<llvm...@lists.llvm.org> wrote:
> You still need a runtime library, even with freestanding. Both
> __aeabi_memclr4 and __aeabi_uidiv are helpers as defined by EABI and
> their presence is required even with freestanding.

IIRC, most users (kernel, android) just implement them or hook into
external libraries. It's not complicated and there's a lot of prior
art.

--renato

[Renato Golin via llvm-dev]

On 26 February 2017 at 11:10, Sean Silva <chiso...@gmail.com> wrote:
> For that triple, Clang seems to be calling into GCC driver
> (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
> -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd selects
> ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it doesn't
> work like that apparently).

This is really odd, but since LLD is not officially supported on ARM
yet (Peter is almost there!), we haven't got around using it for
anything serious.

It smells like yet-another clang driver kerfuffle when cross-compiling.

cheers,
--renato

[Joerg Sonnenberger via llvm-dev]

On Mon, Feb 27, 2017 at 10:51:09PM +0000, Renato Golin via llvm-dev wrote:
> On 26 February 2017 at 11:10, Sean Silva <chiso...@gmail.com> wrote:
> > For that triple, Clang seems to be calling into GCC driver
> > (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
> > -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd selects
> > ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it doesn't
> > work like that apparently).
>
> This is really odd, but since LLD is not officially supported on ARM
> yet (Peter is almost there!), we haven't got around using it for
> anything serious.
>
> It smells like yet-another clang driver kerfuffle when cross-compiling.

Not really. The generic none target has a fallback in clang to GCC,
because it can't know by definition how to link for such a target.
At this point in time it might be better to just error out...

Joerg

[Sean Silva via llvm-dev]

On Sun, Feb 26, 2017 at 2:25 PM, Sean Vig <sean....@gmail.com> wrote:

Hi Sean

On Sun, Feb 26, 2017 at 5:10 AM, Sean Silva <chiso...@gmail.com> wrote:

For that triple, Clang seems to be calling into GCC driver (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it doesn't work like that apparently).

@Renato: Do you know why Clang is forwarding to GCC like this for linking? If we need to live with this arrangement for a long time we should definitely invest in teaching GCC about -fuse-ld=lld

@Davide: is there any chance you could revive that GCC patch to add support for LLD?

(though it does seem like we still fail H.J.'s test case in https://bugs.llvm.org//show_bug.cgi?id=28414 has not yet been fixed. I further reduced it. Seems to be related to symbol versioning. We don't recognize that `.symver bar, foo@@VERS` is a definition of `foo`.

)

If you are building your own LLD, can you see if the attached patch fixes it for you? If not, can you add `--reproduce /tmp/repro.tar` to your link command line and upload repro.tar somewhere for us?

Thanks! I applied that patch and it links with -flto now, but it looks like there are now some other problems. First, it seems to be dropping things that I try to KEEP in the linker script, even when I don't use --gc-sections, namely my interrupt vector table.

This sounds like it might be an issue in the LTO internalization logic. Can you file a bug? 

If that is what is happening, then you should be able to add `__attribute__((used))` to the interrupt vector table to avoid it from being lost by LTO.

Slightly longer description, if you're interested: As part of LTO, there are optimization passes that run on the IR which can have GC-like effects, and it is LLD's responsibility to ensure that symbols are correctly marked as needing to be kept. We want to tell the IR optimizer precise info about what is live. Even if a function `foo` is dead and will be stripped by the regular linker GC later in the link, if the IR optimizer doesn't know that `foo` is dead then `foo` can still prevent its callees from being profitable to inline somewhere else that is actually live (the inliner might see two calls when there is actually only one). It sounds like currently the KEEP's in the linker script are the only thing informing the linker that the interrupt vector table must be kept, and I don't think LLD factors that into what it communicates to the IR optimizer.

 

Second, when using thin LTO, it starts trying to link symbols __aeabi_memclr4 and __aeabi_uidiv, which I don't think it should since I am using -ffreestanding -fno-builtins when compiling (with full lto or no lto, it does not try to link these symbols).

I just sent an email to register for a bugzilla account, I can report there with a minimally reproducible example for these problems.

 

@Renato, do you know if there is any way for us to implement getBitcodeMachineKind in a sane way so that we don't fill in a long tail of cases of inferring EM_* from the triple? It seems that the EM_* choices are buried very deep in the backend, so it might not be feasible :(

What kind of issues have you been running into?

Linker scripts are certainly one area that we have been slowly filling in the long tail of compatibility, it would be great if you filed bugs with any issues you encounter! If the files are small enough and you can share them, it can be as simple as just adding `--reproduce /tmp/repro.tar` to your link command line (or setting LLD_REPRODUCE=/tmp/repro.tar in the environment) and attach repro.tar to a bug report whenever you encounter an issue (even if repro.tar is larger, you can try hosting on google drive / dropbox etc., compressing with xz also often helps). Or if it is a parsing problem just attaching the problematic linker script to the bug report might be enough.

I haven't hit anything that looked like a bug, just features that aren't implemented, or that work slightly differently between ld linker scripts. I'd be happy to make additional feature requests for those on the bug tracker, too.

I said "bug" because well it is "bug"zilla, but yes, feature requests / compatibility issues are definitely very welcome. "ticket" is probably a better word, but "bug" is too ingrained for me :)

 

 

Regarding objcopy, that is something that we have wanted to write for a long time (but it has never quite been a priority). As objcopy supports a bewildering set of different options, it would be useful for you to file a bug about what specific options/features you need. We don't really have a bugzilla component for objcopy yet, so you can just throw it under llvm-nm (another one of the binutils, close enough) in https://bugs.llvm.org/enter_bug.cgi?product=tools 

Sounds good.

Thanks for the patch and the help.

No problem. Thanks for helping us find and fix our bugs :)

-- Sean Silva

 

Sean

 

-- Sean Silva