seg fault in the middle of executing DT_INIT_ARRAYSZ

[zhiting zhu]

Hey,

I'm hitting this error:

ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT function
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Finished executing DT_INIT function
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 56 DT_INIT_ARRAYSZ functions
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func 0x1000388ad500
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func 0x1000388ac8a0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func 0x1000388ac8d0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func 0x1000388ac900
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func 0x1000388ac930
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func 0x1000388ac960
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func 0x1000388ac990
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func 0x1000388ac9c0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func 0x1000388ac9f0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func 0x1000388aca20
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init func 0x1000388aca50
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init func 0x1000388aca80
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init func 0x1000388acab0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init func 0x1000388acae0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init func 0x1000388acb10
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init func 0x1000388acb40
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init func 0x1000388acb70
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init func 0x1000388acc50
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init func 0x1000388acc80
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init func 0x1000388accb0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init func 0x1000388acce0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init func 0x1000388acd10
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init func 0x1000388acd40
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init func 0x1000388acd70
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init func 0x1000388acda0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init func 0x1000388acdd0
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init func 0x1000388ace00
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init func 0x1000388ace50
ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init func 0x1000388ace80
Aborted

[backtrace]
0x0000000040463abb <osv::generate_signal(siginfo&, exception_frame*)+59>
0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int, exception_frame*)+26>
0x000000004032f3e9 <mmu::vm_fault(unsigned long, exception_frame*)+185>
0x000000004038f7b6 <page_fault+166>
0x000000004038e5f6 <???+1077470710>
0x0000000040341042 <elf::program::init_library(int, char**)+402>
0x00000000403473db <elf::program::get_library(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)+715>
0x0000000040462f19 <dlopen+153>
0x0000100000937228 <_PyImport_FindSharedFuncptr+376>
0x006567617373656c <???+1936942444>
2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/lib.rs:1658] Failed to log metrics: Logger was not initialized.

Is there any clue how to debug this? The function pointer seems to point to a valid address but I get a seg fault when executing it.

Best,

Zhiting

[Waldek Kozaczuk]

Can you send us the full output? I wonder if there are any warnings before it?

I see you are running this on firecracker. Can you run it under qemu and connect to it with gdb and see if you get better stacktrace?

You can add another debug statement like that:

if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) {
...
// Put breakpoint here
}

and try to see what statement causes the fault. Make sure to do 'osv syms" to get as much debug info resolved as possible. (see https://github.com/cloudius-systems/osv/wiki/Debugging-OSv).

I wonder if this has to with the order of initializing the ELF objects when called by dlopen(). 

Waldek

PS. If no more clue next step would be to add an app so we can build it and reproduce it? 

[Nadav Har'El]

On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Can you send us the full output? I wonder if there are any warnings before it?

I see you are running this on firecracker. Can you run it under qemu and connect to it with gdb and see if you get better stacktrace?

You can add another debug statement like that:

if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) {
...
// Put breakpoint here
}

and try to see what statement causes the fault. Make sure to do 'osv syms" to get as much debug info resolved as possible. (see https://github.com/cloudius-systems/osv/wiki/Debugging-OSv).

I hope that "osv syms" will find the newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't, maybe we can fix the order of when the array that "osv syms" uses gets written during loaded.

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com.

[zhiting zhu]

Here's the output log. See the file attached.

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com.

[zhiting zhu]

Unfortunately, I can't boot the vm with qemu. It's hanging at the beginning.

I'm seeing this:

bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
vga: Add VGA device instance
eth0: ethernet address: 52:54:00:12:34:56
virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096
random: virtio-rng registered as a source.
random: intel drng, rdrand registered as a source.
random: <Software, Yarrow> initialized
VFS: unmounting /dev
VFS: mounting rofs at /rofs
random: device unblocked.

[Waldek Kozaczuk]

Are you passing any parameters to ./scripts/run.py? I would also suggest to specify just single vCPU - it should make debugging easier.

Also does it hang with this app only or others as well? Can you try this:

./scripts/build image=native-example
./scripts/run.py

Waldek

To unsubscribe from this group and stop receiving emails from it, send an email to osv...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osv...@googlegroups.com.

[zhiting zhu]

native-example and python image works with qemu. It seems it only hangs on my custom tensorflow image. I'm only passing --verbose /python3 to run.py

To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com.

[Waldek Kozaczuk]

It seems like it got stuck while trying to mount the filesystem. The next boot message would normally be 'VFS: mounting devfs at /dev".

I wonder if the image (usr.img) is somehow locked or something. Have you tried to rebuild the image? Try zfs.

Waldek

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com.

[zhiting zhu]

Yeah, Zfs image doesn't hang. I don't know why rofs image hangs. I need to increase the qemu memory in upload_manifest.py otherwise it hangs on building zfs images.

To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com.

[Waldek Kozaczuk]

BTW how big is you image?

I wounder if ROFS hangs because it runs out of memory when trying to load files into memory.

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com.

[zhiting zhu]

The image is 2.5 G.

To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com.

[Waldek Kozaczuk]

Yeah that is pretty big image given that the default memory size if 4GB. I wonder if increasing the memory to something like 8GB would make rofs work. rofs is not very memory efficient - see https://github.com/cloudius-systems/osv/issues/979. This is one of the issues I would want to work next.

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com.

[zhiting zhu]

I get the dump from gdb:

#0  0x00000000403965f2 in processor::cli_hlt () at arch/x64/processor.hh:247
#1  arch::halt_no_interrupts () at arch/x64/arch.hh:48
#2  osv::halt () at arch/x64/power.cc:26
#3  0x0000000040221b14 in abort (fmt=fmt@entry=0x405ff34b "Aborted\n") at runtime.cc:132
#4  0x0000000040221b32 in abort () at runtime.cc:98
#5  0x0000000040463aec in osv::generate_signal (siginfo=..., ef=0xffff800102932068) at libc/signal.cc:124
#6  0x0000000040463b5b in osv::handle_mmap_fault (addr=<optimized out>, sig=<optimized out>, ef=<optimized out>) at libc/signal.cc:139
#7  0x000000004032f3ea in mmu::vm_fault (addr=<optimized out>, addr@entry=1, ef=ef@entry=0xffff800102932068) at core/mmu.cc:1336
#8  0x000000004038f787 in page_fault (ef=0xffff800102932068) at arch/x64/mmu.cc:42
#9  <signal handler called>
#10 0x000010003896052a in google::protobuf::internal::AddDescriptors(google::protobuf::internal::DescriptorTable const*) ()
#11 0x00001000388aceb6 in ?? ()
#12 0x000000004033ff9a in elf::object::run_init_funcs (this=0xffffa0011ad8c800, argc=argc@entry=0, argv=argv@entry=0x0) at core/elf.cc:1175
#13 0x000000004034102b in elf::program::init_library (this=this@entry=0xffffa000001fbdf0, argc=argc@entry=0, argv=argv@entry=0x0) at core/elf.cc:1486
#14 0x00000000403473ac in elf::program::get_library (this=this@entry=0xffffa000001fbdf0, name="/lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so",
    extra_path=std::vector of length 0, capacity 0, delay_init=delay_init@entry=false) at core/elf.cc:1465
#15 0x0000000040462f4a in dlopen (filename=0x200006f47520 "/lib/python3.6/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so", flags=<optimized out>) at libc/dlfcn.cc:54
#16 0x0000100000937229 in _PyImport_FindSharedFuncptr ()
#17 0x000010000095dba7 in _PyImport_LoadDynamicModuleWithSpec ()

It seems the problem is this function: google::protobuf::internal::AddDescriptors. Unfortunately, tensorflow library strips the debug info even through I add -g when I compile it.

To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/34332f2f-7a70-4711-bc34-ce19bccaedcd%40googlegroups.com.

[zhiting zhu]

I feel it might be not the specific library's problem but osv's problem. If I add

from google.protobuf.pyext import _message

before import TensorFlow, that library could be loaded without any problem. But it then fails at the other places which could be successfully loaded before the addition:

ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing DT_INIT function
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Finished executing DT_INIT function
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 565 DT_INIT_ARRAYSZ functions
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 0 init func 0x1000246e00f0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 1 init func 0x1000246e04b0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 2 init func 0x1000246d31e0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 3 init func 0x1000246d3740
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 4 init func 0x1000246d3770
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 5 init func 0x1000246d37a0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 6 init func 0x1000246d37d0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 7 init func 0x1000246d3800
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 8 init func 0x1000246d3830
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 9 init func 0x1000246d3860
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 10 init func 0x1000246d3890
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 11 init func 0x1000246d38c0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 12 init func 0x1000246d38f0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 13 init func 0x1000246d3920
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 14 init func 0x1000246d3950
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 15 init func 0x1000246d3980
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 16 init func 0x1000246d39b0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 17 init func 0x1000246d39e0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 18 init func 0x1000246d3a10
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 19 init func 0x1000246d3a40
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 20 init func 0x1000246d3a70
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 21 init func 0x1000246d3aa0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 22 init func 0x1000246d3ad0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 23 init func 0x1000246d3b00
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 24 init func 0x1000246d3b30
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 25 init func 0x1000246d3b60
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 26 init func 0x1000246d3b90
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 27 init func 0x1000246d3bc0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 28 init func 0x1000246d3bf0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 29 init func 0x1000246d3c20
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 30 init func 0x1000246d3c50
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 31 init func 0x1000246d3c80
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 32 init func 0x1000246d3cc0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 33 init func 0x1000246d3cf0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 34 init func 0x1000246d3d20
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 35 init func 0x1000246d3d50
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 36 init func 0x1000246d3d80
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 37 init func 0x1000246d3db0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 38 init func 0x1000246d3de0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 39 init func 0x1000246d3e10
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 40 init func 0x1000246d3e40
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 41 init func 0x1000246d3e70
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 42 init func 0x1000246d3ea0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 43 init func 0x1000246d3ed0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 44 init func 0x1000246d3f00
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 45 init func 0x1000246d3f30
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 46 init func 0x1000246d3f60
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 47 init func 0x1000246d3f90
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 48 init func 0x1000246d3fc0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 49 init func 0x1000246d3ff0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 50 init func 0x1000246d4020
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 51 init func 0x1000246d4050
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 52 init func 0x1000246d4080
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 53 init func 0x1000246d40b0
ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: Executing 54 init func 0x1000246d42a0
Aborted

[backtrace]
0x0000000040463aeb <osv::generate_signal(siginfo&, exception_frame*)+59>
0x0000000040463b5a <osv::handle_mmap_fault(unsigned long, int, exception_frame*)+26>

0x000000004032f3e9 <mmu::vm_fault(unsigned long, exception_frame*)+185>

0x000000004038f786 <page_fault+166>
0x000000004038e5c6 <???+1077470662>

gdb outputs:

#0  0x000000004039100a in processor::cli_hlt () at arch/x64/processor.hh:247
#1  nmi (ef=0xffff8001000c8068) at arch/x64/exceptions.cc:306
#2  <signal handler called>
#3  sched::cpu::do_idle (this=this@entry=0xffff80000001a040) at core/sched.cc:405
#4  0x00000000403efd48 in sched::cpu::idle (this=0xffff80000001a040) at core/sched.cc:423
#5  0x00000000403efd6c in sched::cpu::<lambda()>::operator() (__closure=<optimized out>) at core/sched.cc:165
#6  std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...)
    at /usr/include/c++/7/bits/std_function.h:316
#7  0x00000000403efeb7 in sched::thread_main_c (t=0xffff8001000c3040) at arch/x64/arch-switch.hh:321
#8  0x000000004038f543 in thread_main () at arch/x64/entry.S:113

which doesn't correspond to a specific place in the library.

The full log is attached.

[zhiting zhu]

Hmm, this might be a protobuf problem. I switch to another version of protobuf that matches what tensorflow uses. That seems to make the problem go away.