How to deal with "File already exists in database:" error correctly?

[{ peetonn }]

I'm getting a runtime error:

[libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already exists in database: content-meta-info.proto

[libprotobuf FATAL google/protobuf/descriptor.cc:1358] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 

libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 

Which, I believe, happens because my macOS plugin code links against two .dylibs which use protobuf and, apparently, use same protobuf objects. (Is this assumption correct? How can I check it?)

How one shall debug and fix this error (given dylibs are third-party)?

Thanks,

[Adam Cozzette]

Based on the error message I think you have the right diagnosis of the problem. There are a couple ways you could try to fix it:

1. Move the generated protobuf code into its own dylib so that the original two dylibs don't include it themselves.

2. Alternatively, if you don't rely on protobuf reflection then you can set option optimize_for = LITE_RUNTIME; in the .proto file. That will remove reflection support, and as a side effect the generated protos will not register themselves in the process-wide descriptor pool (which is where the crash came from). I think you would still have to be careful to avoid ODR violations, so you might need to carefully set up the dylibs to hide their generated protobuf symbols.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/274860bb-6027-4324-9674-4ac9d21d1ca2%40googlegroups.com.

[Peter Gusev]

Thanks! I think I've seen this kind of solution somewhere online before...

but what if I don't have any edit access to the .proto files?

In my debugger, I noticed the crash originates in 

void AddDescriptors() {

  static ::google::protobuf::internal::once_flag once;

  ::google::protobuf::internal::call_once(once, AddDescriptorsImpl);

}

So I search all dylibs my plugin uses (and the plugin itself) for this symbol (as ' nm <binary> | grep "AddDescriptorsImpl" ') and the only place I found this symbol was in one dylib:

0000000000000f70 T __ZN38protobuf_content_2dmeta_2dinfo_2eproto18AddDescriptorsImplEv

0000000000013fc0 s __ZZN38protobuf_content_2dmeta_2dinfo_2eproto18AddDescriptorsImplEvE10descriptor

I'm trying to understand, how this code might be called twice?...

However, I can see various "ContentMetaInfo" in three different binaries (two dylibs and plugin binary).

--

Peter Gusev

gusev.io

pe...@gusev.io

+1 213 587-27-48

Research Scholar @ REMAP UCLA
Video streaming/ICN networks/Creative Coding/Interactive Media

[Adam Cozzette]

If you can't edit the .proto files then I think the only solution is to make sure the code for content-meta-info.proto only appears in one dylib (or in the main binary only).

I'm not sure I would fully trust nm to report which dylibs are including the code, because I can imagine that some or all of the symbols could get stripped depending on how they're built. If the string "ContentMetaInfo" appears in multiple binaries then they might all be including the generated code for that .proto file.

[Peter Gusev]

Thanks for your insights!

It may also gets complicated with the environment I'm testing it in.

Hence, I'd like to know when does protobuf invoke this method? Is it just when static variable "static_descriptor_initializer" gets initialized?

Thanks,

[Adam Cozzette]

Right, this registration will happen when that static variable's initialization occurs. Ordinarily this will happen pre-main, though if you're dynamically loading these dylibs then it will happen at that loading time as well.

[Peter Gusev]

This makes me think of what could potentially cause this code to execute twice?...

[Adam Cozzette]

If the generated code appears in two dylibs then there will be two of those static variables performing the same initialization.

[Peter Gusev]

I checked for "static_descriptor_initialized" variable using "nm" and it shows only one dylib for that. You mentioned don't fully trust nm -- is there a better tool to check that?

0000000000019de0 S protobuf_content_2dmeta_2dinfo_2eproto::static_descriptor_initializer

[Adam Cozzette]

Right, since that variable is static I suspect that the compiler doesn't have to output a symbol for it and so "nm" won't reliably report its existence. I'm not sure if there is a great way to tell whether a given pb.cc file was linked into a given dylib, but the pb.cc file will store its descriptor as a serialized protobuf and you can look for that. If you call strings on the dylib and grep for the .proto file name or particular message names (e.g. "ContentMetaInfo" in your case) then you should be able to find the serialized descriptor if it's present.