[llvm-dev] Undefined hidden symbol when mixing non-CFI and CFI-enabled static libraries using ThinLTO

[Игорь Балдуев via llvm-dev]

Hello,

I've stumbled upon a linkage error when mixing static libraries built with and without CFI support. I haven't found any information related to the correctness of such mixing, but here are some insights on the LTO optimization process leading to the errors (ThinLTO version is used). Non-CFI library sources are compiled with flags: 

--param ssp-buffer-size=4 -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC -fPIE -fgnu-keywords -fstack-protector-all -fstack-protector-strong -fstack-protector-strong -fstrict-aliasing -g -std=c++11 -stdlib=libc++

CFI library sources are compiled with flags:

--param ssp-buffer-size=4 -DNDEBUG -D_FORTIFY_SOURCE=2 -D_GLIBCXX_USE_CXX11_ABI=1 -D_LIBCPP_ENABLE_CXX17_REMOVED_AUTO_PTR=1 -D_LIBCPP_ENABLE_CXX17_REMOVED_BINDERS=1 -D_LIBCPP_ENABLE_CXX17_REMOVED_RANDOM_SHUFFLE=1 -O2 -fPIE -fgnu-keywords -flto=thin -fno-sanitize-trap=all -fsanitize-blacklist=/path/to/blacklist/blacklist.txt -fsanitize-cfi-cross-dso -fsanitize=cfi -fsanitize=safe-stack -fstack-protector-all -fstack-protector-strong -fstandalone-debug -fvisibility-inlines-hidden -fvisibility=hidden -g -std=c++17 -stdlib=libc++

Resulting executable is linked against two static libraries: non-CFI libbenchmark.a, which is Google's Benchmark library, and CFI-enabled libdsl.a, which is mine. Depending on the order of those two libraries in the link command, either linking completes successfully, or multiple undefined hidden symbol errors are emitted:

ld.lld: error: undefined hidden symbol: std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) (.cfi)

>>> referenced by ld-temp.o

>>>               lto.tmp:(std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long))

ld.lld: error: undefined hidden symbol: std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> > std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) (.cfi)

>>> referenced by ld-temp.o

>>>               lto.tmp:(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> > std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char))

An attempt to trace undefined symbol resolution using linker option -Wl,--trace-symbol gives the following result for the failing build:

.contribs/google_benchmark/lib/libbenchmark.a(benchmark.cc.o): definition of _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

.contribs/google_benchmark/lib/libbenchmark.a(benchmark_register.cc.o): reference to _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

...more references from libbenchmark.a...

lib/libdsl.a(markup_translator.cpp.o): reference to _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

/usr/toolchain/x86_64-pc-linux-gnu_gcc10.2.0_glibc2.11_clang11.0.0/x86_64-pc-linux-gnu/lib/libc++.so.1: shared definition of _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

lto.tmp: definition of _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

lto.tmp: reference to _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

Successfull build looks like this:

lib/libdsl.a(markup_translator.cpp.o): definition of _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

.contribs/google_benchmark/lib/libbenchmark.a(benchmark.cc.o): reference to _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

...more references from libbenchmark.a...

/usr/toolchain/x86_64-pc-linux-gnu_gcc10.2.0_glibc2.11_clang11.0.0/x86_64-pc-linux-gnu/lib/libc++.so.1: shared definition of _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

<internal>: reference to _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

lto.tmp: definition of _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

lto.tmp: reference to _ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m

It looks like linker fails  to resolve symbol's corresponding .cfi definition if symbol definition is initially found in a library built without CFI support, and succeeds otherwise. Also, additional <internal> reference is present in the latter case. Rebuilding all static libraries with CFI support solves the problem, but it is unclear to me whether this behavior is correct or this is some kind of a bug. Unfortunately, I was unable to make a minimal example reproducing the issue.