Symbol lookup errors while building C++ source code

[Niels Bergsma]

Hi all,

I've been trying to build simple C++ apps for the past few weeks on OSV. I frequently receive "failed looking up symbol" errors and I can't figure out how to solve them properly. Simple programs like HelloWorld or variants which include simple shared library work correctly, but more sophisticated libraries like leveldb, restbed report errors at runtime. Even when I manual compile those libraries and link the object files with my own code. 

The error I receive look like:

OSv v0.24

eth0: 192.168.122.15

/tools/example.so: failed looking up symbol _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1EPKcRKS3_ (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&))

[backtrace]

0x00000000003447cd <elf::object::symbol(unsigned int)+205>

0x0000000000346148 <elf::object::resolve_pltgot(unsigned int)+136>

0x0000000000346325 <elf_resolve_pltgot+69>

0x0000000000391e64 <???+3743332>

0x00000000004c12df <???+4985567>

0x000010000584cd54 <main+84>

0x0000000000421153 <osv::application::run_main(std::string, int, char**)+371>

0x0000000000423220 <osv::application::run_main()+624>

0x000000000020ce43 <osv::application::main()+83>

0x0000000000423588 <???+4339080>

0x000000000044d465 <???+4510821>

0x00000000003f10c7 <thread_main_c+39>

0x00000000003931b5 <???+3748277>

My capstan file looks like is:

base: cloudius/osv-base

cmdline: /tools/example.so

build: make

files:

  /tools/example.so: example.so

What am I doing wrong?

Thanks!

Niels 

[Avi Kivity]

It could be the new ABI, see https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html.

Try compiling your code with _GLIBCXX_USE_CXX11_ABI=0.

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nadav Har'El]

On Sun, Apr 23, 2017 at 2:06 PM, Avi Kivity <a...@scylladb.com> wrote:

It could be the new ABI, see https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html.

Try compiling your code with _GLIBCXX_USE_CXX11_ABI=0.

Indeed. Check out also commit 6a3bff38a281e65ee715bab4fadef63e0918f7d3 where we encountered this problem (and its fix) previously.

This problem is an example of what happens when you compile the OSv kernel itself with one compiler and libraries (in this case, the C++ standard library) and then try to run on it software compiled on a very different version of the compiler and/or library. The general issue is tracked in https://github.com/cloudius-systems/osv/issues/821, "Combining pre-compiled OSv kernel with pre-compiled executable".

In this case, the specific problem is that you're taking a very old compiled OSv kernel (Capstan takes an antique OSv kernel which we haven't recompiled in a very long time, as a result was also compiled with old compilers and libraries) and try to run on it code compiled on newer compilers. The newer compiler assumes new C++ ABI (in particular the implementation of std::string) which the old C++ library embedded in OSv did not include. If you use the macro which Avi suggested (-D_GLIBCXX_USE_CXX11_ABI=0), it will cause the C++ header files to use the old ABI and work on the old) library. Of course, this will not help if you're trying to run on OSv a shared library or an entire application which uses C++ and you cannot recompile.

A different approach would be to compile OSv yourself with the same compiler that built your application.

Pekka, another question we should ask ourselves is what is the plan with Capstan, and more importantly, the binary OSv releases. The OSv v0.24 version he used is two years old (!). If we're not going to produce any more pre-compiled releases (are we?) should we at least remove the antique ones and force people to compile OSv on their own before using Capstan? Or should we make another release? Or what?

Nadav.

[Rick Payne (Offshore)]

> On 23 Apr 2017, at 05:14, Nadav Har'El <n...@scylladb.com> wrote:
>
> Pekka, another question we should ask ourselves is what is the plan with Capstan, and more importantly, the binary OSv releases. The OSv v0.24 version he used is two years old (!). If we're not going to produce any more pre-compiled releases (are we?) should we at least remove the antique ones and force people to compile OSv on their own before using Capstan? Or should we make another release? Or what?

On this front, I have a rebar_osv plugin for the erlang ‘rebar’ build tool that will take an erlang release and build an OSv image. Its simpler to port an erlang OTP application to OSv that way than using the apps/erlang setup. It will also include a recent OSv base image (as I use the same method of CPIO-ing in the files). Hope to release that soon - are you ok with me including an OSv base image?

Cheers,
Rick

[Nadav Har'El]

Yes.

However note that I think that Capstan's notion of "base image" is misdirected. The Capstan "base image" is an image with a particular size (e.g., 10 GB) plus the OSv kernel and some collection of "default" files. If you want a 100 MB or 100 GB image instead, you're out of luck.

What I think is a better approach is to take separatey the OSv kernel with just cpiod.so and mkfs.so (i.e., take build/release/loader.img) and the files, and compose the image with the size you want and uploading the files you want to it.  This is more-or-less what the Mikelangelo project's fork of Capstan is doing. Maybe that's what you are also planning to do?

Nadav.

[Rick Payne (Offshore)]

> What I think is a better approach is to take separatey the OSv kernel with just cpiod.so and mkfs.so (i.e., take build/release/loader.img) and the files, and compose the image with the size you want and uploading the files you want to it. This is more-or-less what the Mikelangelo project's fork of Capstan is doing. Maybe that's what you are also planning to do?

That is indeed the intention. Right now, I’m building an Erlang/OTP with the patches from your modules/ directory and reference that from the rebar build (just as you would if you were cross-compiling an erlang/otp application). The ‘base’ image I’m using is created image with libncurses, ssl etc. Once a few Erlang/OTP build issues are sorted, I may move to using loader.img, but it is convenient to have an image with cloud-init and the httpserver installed too...

Cheers,
Rick

[Niels Bergsma]

Yes that did the trick. Awesome! I switched to the build scripts in the meanwhile, to make sure I have the latest version.

[miha.ple...@gmail.com]

Hi,

as Nadav said it might be best to use Capstan from MIKELANGELO repository. The Capstan that is there is under active development and contains much more recent OSv base image (we've last compiled it 2 weeks ago or so).
The Capstan there works so that it always uses same basic OSv image (it downloads it from the repository) and uploads all the files from your current directory to it. The way you would use it in your case is:

1. compile your C++ application and copy the result (one or more `.so` files and possible other files) into a new folder "mypackage"
2. add the two configuration files meta/package.yaml and meta/run.yaml into "mypackage" folder (consult README for details, should be fairly simple)
   
3. use command `capstan package compose <unikernel_name>` - a new unikernel will be composed with your application files
4. use command `capstan run <unikernel_name>` - the unikernel will be run

For details and a step-by-step guide please see README at the Capstan GithubRepository.

Dne nedelja, 23. april 2017 12.41.51 UTC+2 je oseba Niels Bergsma napisala: