[llvm-dev] libunwind is not configured with -funwind-tables when building it for ARM Linux?

[Sergej Jaskiewicz via llvm-dev]

There’s this bug: https://bugs.llvm.org/show_bug.cgi?id=38468.

I’ve managed to track it down to a configuration issue. The thing is that in order for libunwind to be usable on ARM Linux, it should be built with the -funwind-tables flag. This flag is conditionally set here: https://github.com/llvm/llvm-project/blob/master/libunwind/CMakeLists.txt#L294, if the compiler “supports” it.

However, the CMake check fails with the following error:

```
ld.lld: error: undefined symbol: __aeabi_unwind_cpp_pr0

>>> referenced by src.cxx

>>> CMakeFiles/cmTC_e9739.dir/src.cxx.o:(.ARM.exidx.text.main+0x0)

clang++: error: linker command failed with exit code 1 (use -v to see invocation)

ninja: build stopped: subcommand failed.

Source file was:
int main() { return 0; }
```

No wonder! __aeabi_unwind_cpp_pr0 is defined in libunwind itself, which we haven’t built yet.

If I instead set the -funwind-tables flag unconditionally using `add_compile_flags(-funwind-tables)` instead of `add_cxx_compile_flags_if_supported(-funwind-tables)`, everything is fine and the aforementioned bug is gone.

I’ve found a PR which seemed to address this: https://reviews.llvm.org/D31858 (cc’ing @phosek, @compnerd and @beanz as participants of this PR). It mentions that the __aeabi_unwind_cpp_pr0 symbol is provided by the compiler runtime. However, as I’ve already said, it lives in libunwind, so the problem doesn’t seem to be solved.

I’m very tempted to just set the -funwind-tables flag unconditionally, but I’m afraid it’ll break something. What would be the right solution for building libunwind with this flag for ARM Linux?

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Peter Smith via llvm-dev]

On Mon, 18 Nov 2019 at 12:32, Sergej Jaskiewicz via llvm-dev
<llvm...@lists.llvm.org> wrote:
>
> There’s this bug: https://bugs.llvm.org/show_bug.cgi?id=38468.
>
> I’ve managed to track it down to a configuration issue. The thing is that in order for libunwind to be usable on ARM Linux, it should be built with the -funwind-tables flag. This flag is conditionally set here: https://github.com/llvm/llvm-project/blob/master/libunwind/CMakeLists.txt#L294, if the compiler “supports” it.
>

Are you sure that libunwind being built without -funwind-tables is the
cause of the bug? I would only expect that to be a problem if an
exception were being propagated through a libunwind function, and that
shouldn't happen unless something has gone badly wrong. I tried the
example with the armv7l release of clang 8.0 which I happened to have
installed on an Armv8l machine and the program worked. I was able to
reproduce the problem with the PR with the default Ubuntu16.04 clang
(3.8) and libc++-dev package.

I also note that when looking at the link line for the example in the
PR, clang was linking libgcc_s and not libunwind so I think the
problem may be somewhere else. There have been quite a lot of fixes
since clang 3.8 and its libc++ so it may be worth trying a more recent
clang.

Can I ask that you try and narrow down what the problem is, what your
environment is, and comment on the PR if it helps reproduce? It is
definitely worth looking at the output of clang -v to see which
unwinder it is using, by default it will be libgcc_s on Linux.

Peter

[Sergej Jaskiewicz via llvm-dev]

Hi Peter,

Thanks for your response.

On 18 Nov 2019, at 17:44, Peter Smith <peter...@linaro.org> wrote:

On Mon, 18 Nov 2019 at 12:32, Sergej Jaskiewicz via llvm-dev
<llvm...@lists.llvm.org> wrote:

There’s this bug: https://bugs.llvm.org/show_bug.cgi?id=38468.

I’ve managed to track it down to a configuration issue. The thing is that in order for libunwind to be usable on ARM Linux, it should be built with the -funwind-tables flag. This flag is conditionally set here: https://github.com/llvm/llvm-project/blob/master/libunwind/CMakeLists.txt#L294, if the compiler “supports” it.

Are you sure that libunwind being built without -funwind-tables is the
cause of the bug?

Yep, pretty sure, I’ve found that the cause of the problem is the _Unwind_Backtrace function not executing the provided callback. It isn’t doing so because, since libunwind is compiled without the flag, the information about the stack frame is lost, so, when _Unwind_Backtrace looks for it, it can’t find it (since we’ve entered the _Unwind_Backtrace stack frame, which lives in libunwind, where no unwind info is present).

I’ve looked at the generated assembly of libunwind and found the .cantunwind directives all over the place.

I would only expect that to be a problem if an
exception were being propagated through a libunwind function, and that
shouldn't happen unless something has gone badly wrong.

Can you explain what you mean?

I tried the
example with the armv7l release of clang 8.0 which I happened to have
installed on an Armv8l machine and the program worked. I was able to
reproduce the problem with the PR with the default Ubuntu16.04 clang
(3.8) and libc++-dev package.

I also note that when looking at the link line for the example in the
PR, clang was linking libgcc_s and not libunwind so I think the
problem may be somewhere else. There have been quite a lot of fixes
since clang 3.8 and its libc++ so it may be worth trying a more recent
clang.

I’m using mainline just-built clang to build libunwind. Actually, I’m cross-compiling on Windows for Linux.

Also, I’m using compiler-rt instead of libgcc.

And yes, the problem is not with libgcc_s, because, as I’ve said, force-setting the -funwind-tables flag in libunwind configuration makes the problem go away.

So, the main question remains: when we’re configuring libunwind build, CMake checks the -funwind-tables flag and that check fails because the __aeabi_unwind_cpp_pr0 symbol is absent. This symbol should be defined in libunwind, which is not build yet. I’m having a hard time understanding how can this be even possible. Can it be that you are not experiencing the problem because your clang uses libgcc and not compiler-rt?

[Peter Smith via llvm-dev]

On Mon, 18 Nov 2019 at 15:23, Sergej Jaskiewicz <jaski...@icloud.com> wrote:
>
> Hi Peter,
>
> Thanks for your response.
>
> On 18 Nov 2019, at 17:44, Peter Smith <peter...@linaro.org> wrote:
>
> On Mon, 18 Nov 2019 at 12:32, Sergej Jaskiewicz via llvm-dev
> <llvm...@lists.llvm.org> wrote:
>
>
> There’s this bug: https://bugs.llvm.org/show_bug.cgi?id=38468.
>
> I’ve managed to track it down to a configuration issue. The thing is that in order for libunwind to be usable on ARM Linux, it should be built with the -funwind-tables flag. This flag is conditionally set here: https://github.com/llvm/llvm-project/blob/master/libunwind/CMakeLists.txt#L294, if the compiler “supports” it.
>
>
> Are you sure that libunwind being built without -funwind-tables is the
> cause of the bug?
>
>
> Yep, pretty sure, I’ve found that the cause of the problem is the _Unwind_Backtrace function not executing the provided callback. It isn’t doing so because, since libunwind is compiled without the flag, the information about the stack frame is lost, so, when _Unwind_Backtrace looks for it, it can’t find it (since we’ve entered the _Unwind_Backtrace stack frame, which lives in libunwind, where no unwind info is present).
>
> I’ve looked at the generated assembly of libunwind and found the .cantunwind directives all over the place.
>
> I would only expect that to be a problem if an
> exception were being propagated through a libunwind function, and that
> shouldn't happen unless something has gone badly wrong.
>
>
> Can you explain what you mean?
>

I'm going from the example provided in the PR:

int main(int argc, char* argv[])
{
try
{
throw 1;
}
catch(...)
{
std::cout << "Exception is caught\n";
}
}

When the exception is thrown via a call to __cxa_throw, what I would
expect is for the unwinder to look up the PC value at the callsite in
the .ARM.exidx table. This should contain information for unwinding
the stack should the exception propagate outside of main. This
propagation of the exception out of main shouldn't go through a
function in libunwind as libunwind should not be in the call stack for
main. This is the case if the search in the table starts from the
callsite at main, and in this case it shouldn't matter whether
libunwind has unwind tables or not.

One thing I've not done is go back to the source and check if the
search does not start in main, but instead starts in a function in
libunwind, which via a further unwind gets to main where the exception
is caught. If that were the case then yes we would need unwinding
information. We'd also need unwinding information in libunwind if it
called a function that could throw an exception.

> I tried the
> example with the armv7l release of clang 8.0 which I happened to have
> installed on an Armv8l machine and the program worked. I was able to
> reproduce the problem with the PR with the default Ubuntu16.04 clang
> (3.8) and libc++-dev package.
>
> I also note that when looking at the link line for the example in the
> PR, clang was linking libgcc_s and not libunwind so I think the
> problem may be somewhere else. There have been quite a lot of fixes
> since clang 3.8 and its libc++ so it may be worth trying a more recent
> clang.
>
>
> I’m using mainline just-built clang to build libunwind. Actually, I’m cross-compiling on Windows for Linux.
> Also, I’m using compiler-rt instead of libgcc.
> And yes, the problem is not with libgcc_s, because, as I’ve said, force-setting the -funwind-tables flag in libunwind configuration makes the problem go away.
>

I've just checked a native build on Arm v8l, apologies I've not got a
Windows machine to cross-compile on. In my case I am getting unwind
information generated for libunwind. I'm guessing you are cross
compiling with a standalone build? In the past I have cross-compiled
libunwind, in summary:
build compiler-rt builtins
build libuwind (statically, depending on compiler-rt)
build libcxxabi depending on libunwind and compiler-rt
build libcxx depending on libcxxabi and compiler-rt

Are you trying to build libunwind into a shared library? I don't think
that is a supported configuration due to the cyclical dependency.

Peter

[Sergej Jaskiewicz via llvm-dev]

No, I’m building libunwind as part of LLVM. Can you please provide the options that you’re using to build libunwind?

In the past I have cross-compiled
libunwind, in summary:
build compiler-rt builtins
build libuwind (statically, depending on compiler-rt)
build libcxxabi depending on libunwind and compiler-rt
build libcxx depending on libcxxabi and compiler-rt

That’s what I’m doing as well.

Are you trying to build libunwind into a shared library? I don't think
that is a supported configuration due to the cyclical dependency.

I’m building a static library.

I’m building llvm & clang & lld first using the native compiler (MSVC), and then the just-built clang acts as a cross-compiler for the runtime libraries that you’ve mentioned.

[Peter Smith via llvm-dev]

The one I just checked was a native build where CC and CXX where clang 8.0
cmake -G Ninja ../llvm \
-DCMAKE_BUILD_TYPE=Release\
-DLLVM_ENABLE_ASSERTIONS=True\
-DLLVM_TARGETS_TO_BUILD="ARM"\
-DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;libc++;libc++abi;libunwind"\
-DLLVM_ENABLE_LLD=True

In that case I'd expect my clang -fuse-unwind-tables to work as it
would be running natively and any test would pick up the native
libgcc_s at link time.

readelf -u gave me amongst other things:
Unwind table index '.ARM.exidx.text._Unwind_Backtrace' at offset 0x49c
contains 1 entries:

0x0: 0x809b46af
Compact model index: 0
0x9b vsp = r11
0x46 vsp = vsp - 28
0xaf pop {r4, r5, r6, r7, r8, r9, r10, r11, r14}

>
> In the past I have cross-compiled
> libunwind, in summary:
> build compiler-rt builtins
> build libuwind (statically, depending on compiler-rt)
> build libcxxabi depending on libunwind and compiler-rt
> build libcxx depending on libcxxabi and compiler-rt
>
>
> That’s what I’m doing as well.

Apologies for the delay in responding, I thought I'd better dig out
build my script to check to see what unwind tables were present in
libunwind. I found that when building libunwind I needed
-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY for the CMake test for
unwind tables to succeed when cross-compiling. If I took that out the
test failed and I ended up with no exceptions.

Using qemu-arm I was able to reproduce the failure when there were no
unwind tables in libunwind. So empirically it seems like unwind tables
are required. If I get some time tomorrow I'll check the libunwind
source to see if it starts the unwinding from libunwind itself. If it
is required for a significant part of libunwind on Arm, then ideally
we would want to fail the build if the flag were not supported.

Peter

[Sergej Jaskiewicz via llvm-dev]

Thank you for your time!

I’ve tried CMAKE_TRY_COMPILE_TARGET_TYPE and it works. But setting this option manually seems suboptimal to me, maybe it makes sense to enable it whenever we’re cross-compiling like you suggested?

The CMake documentation, however, says that setting this option to STATIC_LIBRARY avoids running the linker in try_compile() checks. I’m not sure we _always_ want that. Do we?

Using qemu-arm I was able to reproduce the failure when there were no
unwind tables in libunwind. So empirically it seems like unwind tables
are required. If I get some time tomorrow I'll check the libunwind
source to see if it starts the unwinding from libunwind itself. If it
is required for a significant part of libunwind on Arm, then ideally
we would want to fail the build if the flag were not supported.

Peter

They are probably required only when we’re compiling for ARM, and not just any ARM, but when we’re using ARM EHABI, e. g. on Linux (AFAIU Darwin doesn’t use EHABI). But I don’t know a way to detect this at configuration time, only at build time.

I have an idea of the way that we could detect the -funwind-tables flag at build time: the check_cxx_compiler_flag function in CMake sets the LIBUNWIND_SUPPORTS_${flagname}_FLAG variable. Now, if the value of that variable is true, we pass a preprocessor definition like _LIBUNWIND_HAS_UNWIND_TABLES to the compiler when building libunwind. In the libunwind source code, we check if that macro is defined, and if not, raise an #error.

I’m going to prepare a patch so that this could be discussed further. What do you think?

[Peter Smith via llvm-dev]

I think it is fine when the output of the build is a static library as
there is no link step involved in it. So for linking libunwind.a I
think it is reasonable to add the flag. I agree that having to
remember to set it when cross-compiling is not ideal.

>
> Using qemu-arm I was able to reproduce the failure when there were no
> unwind tables in libunwind. So empirically it seems like unwind tables
> are required. If I get some time tomorrow I'll check the libunwind
> source to see if it starts the unwinding from libunwind itself. If it
> is required for a significant part of libunwind on Arm, then ideally
> we would want to fail the build if the flag were not supported.
>
> Peter
>
>
> They are probably required only when we’re compiling for ARM, and not just any ARM, but when we’re using ARM EHABI, e. g. on Linux (AFAIU Darwin doesn’t use EHABI). But I don’t know a way to detect this at configuration time, only at build time.
>
> I have an idea of the way that we could detect the -funwind-tables flag at build time: the check_cxx_compiler_flag function in CMake sets the LIBUNWIND_SUPPORTS_${flagname}_FLAG variable. Now, if the value of that variable is true, we pass a preprocessor definition like _LIBUNWIND_HAS_UNWIND_TABLES to the compiler when building libunwind. In the libunwind source code, we check if that macro is defined, and if not, raise an #error.
>
> I’m going to prepare a patch so that this could be discussed further. What do you think?

That sounds reasonable, although I'm no CMake expert and there may be
a better way of detecting cross-compilation for EHABI and either
unconditionally adding -funwind-tables or throwing an error.
Compiler-rt has got quite a lot of target detection logic in it,
although it is difficult to follow. It may be worth starting a new
thread as I suspect that many of the experts won't be looking at this
thread anymore.

[Sergej Jaskiewicz via llvm-dev]

I’ve submitted a patch that aims to fix this https://reviews.llvm.org/D70815, can I ask you to review it?