grpc application crashes in protobuf init after dlopen()

[MMMike]

( linux C++ grpc protobuf static library init crash ) (keywords)

My application/service has a grpc API, so it is also a grpc service.

It has plugins, one of which is a grpc client of another service.

By plugin I mean features that can be loaded when needed (maybe hours later) by loading the shared library with dlopen().

The plugin and my application both use the same version/build of grpc.

When the plugin is loaded via dlopen(), the application crashes, apparently re-initializing the protobuf.

See the stack below.

When I make grpc on linux, the build includes libprotobuf.a which we install with the application.

I posit the re-init could be solved if the application and the plugin didn't both link with libprotobuf.a

Is this correct?  Is there a better way for me to do this?  

Is there a better explanation for the stack below? 

I think if the grpc build created libprotobuf.so instead of libprotobuf.a, and everything used a single instance of libprotobuf.so, the re-init wouldn't be a problem.

If necessary, the application startup could dlopen("libprotobuf.so", RTLD_NOW|RTLD_NODELETE) libprotobuf.so with flags to never unload the shared lib. 

If it is never unloaded I hope the re-init wouldn't happen.

Grpc source includes protobuf source via git submodules

I tried modifying grpc's Makefile to produce libprotobuf.so.

The grpc Makefile builds protobuf using: ./configure --disable-shared --enable-static $(PROTOBUF_CONFIG_OPTS)

The make fails if I try --enable-shared

Flags --enable-shared and --disable-static are common but don't seem to be fully implemented in protobuf.

I don't know how to convince protobuf to build libprotobuf.so for me.

Question 1: Am I on the right track with the cause of the crash? How do people prevent grpc's protobuf from crashing in one-time init code?

Question 2: How do I replace libprotobuf.a with libprotobuf.so in the grpc make or in the protobuf automake/configure? (assuming question 1 is on the right track.)  

Help I need advice. Please and thank you.

(gdb) bt

#0  0x00007fc4af68e875 in raise () from /lib64/libc.so.6

#1  0x00007fc4af68fe51 in abort () from /lib64/libc.so.6

#2  0x00007fc4b00ec325 in __gnu_cxx::__verbose_terminate_handler() ()

   from /usr/lib64/libstdc++.so.6

#3  0x00007fc4b00ea4c6 in ?? () from /usr/lib64/libstdc++.so.6

#4  0x00007fc4b00ea4f3 in std::terminate() () from /usr/lib64/libstdc++.so.6

#5  0x00007fc4b00ea712 in __cxa_throw () from /usr/lib64/libstdc++.so.6

#6  0x00007fc4b44c70d8 in google::protobuf::internal::LogMessage::Finish (this=0x7fc49a7f46d0)

    at google/protobuf/stubs/common.cc:269

#7  0x00007fc4b44d992a in google::protobuf::DescriptorPool::InternalAddGeneratedFile (

    encoded_file_descriptor=0x7fc49c755de0 <protobuf_ghost_2eproto::AddDescriptorsImpl()::descriptor>, size=489) at google/protobuf/descriptor.cc:1394

#8  0x00007fc49c684cfc in protobuf_ghost_2eproto::AddDescriptorsImpl ()

    at /tcwork/swbuilder/7182982361a3dd28/cmic_core/build/src/Grpc/ghostGrpc/ghost.pb.cc:155

#9  0x00007fc4b44c7f90 in google::protobuf::internal::FunctionClosure0::Run (this=0x7fc49a7f4760)

    at ./google/protobuf/stubs/callback.h:129

#10 0x00007fc4b44c9491 in google::protobuf::GoogleOnceInitImpl (

    once=0x7fc49c9a90e0 <protobuf_ghost_2eproto::AddDescriptors()::once>, closure=<optimized out>)

    at google/protobuf/stubs/once.cc:83

#11 0x00007fc4b44b6f6b in google::protobuf::GoogleOnceInit (

    once=0x7fc49c9a90e0 <protobuf_ghost_2eproto::AddDescriptors()::once>,

    init_func=0x7fc49c684ce2 <protobuf_ghost_2eproto::AddDescriptorsImpl()>)

    at /var/opt/teradata/swrepository/local/SM_SWREPOSITORY.SLES_11_SP3-devel/grpc-1.11.0-0.x86_64/usr/include/google/protobuf/stubs/once.h:128

#12 0x00007fc49c684d2e in protobuf_ghost_2eproto::AddDescriptors ()

    at /tcwork/swbuilder/7182982361a3dd28/cmic_core/build/src/Grpc/ghostGrpc/ghost.pb.cc:162

#13 0x00007fc49c68ac67 in protobuf_ghost_2eproto::StaticDescriptorInitializer::StaticDescriptorInitializer (this=0x7fc49c9a90c0 <protobuf_ghost_2eproto::static_descriptor_initializer>)

    at /tcwork/swbuilder/7182982361a3dd28/cmic_core/build/src/Grpc/ghostGrpc/ghost.pb.cc:167

#14 0x00007fc49c6893cf in __static_initialization_and_destruction_0 (__initialize_p=1,

    __priority=65535)

    at /tcwork/swbuilder/7182982361a3dd28/cmic_core/build/src/Grpc/ghostGrpc/ghost.pb.cc:169

#15 0x00007fc49c6893e4 in _GLOBAL__sub_I_ghost.pb.cc(void) ()

    at /tcwork/swbuilder/7182982361a3dd28/cmic_core/build/src/Grpc/ghostGrpc/ghost.pb.cc:1472

#16 0x00007fc49c755756 in __do_global_ctors_aux () from /opt/teradata/cmic/lib/libghostGrpc.so

#17 0x00007fc49c67c28b in _init () from /opt/teradata/cmic/lib/libghostGrpc.so

#18 0x00007fc49c9ef298 in ?? () from /opt/teradata/cmic/lib/libMEPlugin_Disk_Array_SDHK.so

#19 0x00007fc4b6f7f268 in call_init () from /lib64/ld-linux-x86-64.so.2

#20 0x00007fc4b6f7f397 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2

#21 0x00007fc4b6f836b6 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2

#22 0x00007fc4b6f7eef6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2

#23 0x00007fc4b6f82eab in _dl_open () from /lib64/ld-linux-x86-64.so.2

#24 0x00007fc4b071ef9b in dlopen_doit () from /lib64/libdl.so.2

#25 0x00007fc4b6f7eef6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2

#26 0x00007fc4b071f33c in _dlerror_run () from /lib64/libdl.so.2

#27 0x00007fc4b071ef01 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2

#28 0x00007fc4b6ca1adc in PluginLibraryLoad::Load (this=0xaf80a8)

    at /tcwork/swbuilder/7182982361a3dd28/cmi_core/src/managedelements/MEPlugin/MEPluginHost.cpp:108

etc...etc...from my application