problem building marian w/ sentencepiece on AWS

[Steven Bradtke]

I am trying to build the latest Marian on an AWS platform, with the SentencePiece option turned ON. This require protobuf, which I have compiled and installed in a local directory. Building seems to go fine until time to link marian-decoder. At that time I get the error shown below. 

Has anyone seen this before? How can I get the marian build to pick up my locally installed protobuf?

Errors:

[ 72%] Linking CXX executable ../marian-decoder
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::MessageLite::SerializeAsString[abi:cxx11]() const'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::io::EpsCopyOutputStream::WriteStringMaybeAliasedOutline(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned char*)'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::Reflection::SetString(google::protobuf::Message*, google::protobuf::FieldDescriptor const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::Get(int) const'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::Add()'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::Message::Utf8DebugString[abi:cxx11]() const'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::Reflection::AddString(google::protobuf::Message*, google::protobuf::FieldDescriptor const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::EnumDescriptor::FindValueByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::begin() const'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::Descriptor::FindFieldByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::RepeatedPtrField(google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::RepeatedPtrField(google::protobuf::Arena*)'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::Message::GetTypeName[abi:cxx11]() const'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::end() const'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::internal::fixed_address_empty_string[abi:cxx11]'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::Clear()'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::MergeFrom(google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::Message::InitializationErrorString[abi:cxx11]() const'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::io::EpsCopyOutputStream::WriteStringOutline(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned char*)'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::Descriptor::DebugString[abi:cxx11]() const'
3rd_party/sentencepiece/src/libsentencepiece_train.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::size() const'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::~RepeatedPtrField()'
3rd_party/sentencepiece/src/libsentencepiece.so.0.0.0: undefined reference to `google::protobuf::internal::InlineGreedyStringParser(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, char const*, google::protobuf::internal::ParseContext*)'
collect2: error: ld returned 1 exit status
make[2]: *** [marian-decoder] Error 1
make[1]: *** [src/CMakeFiles/marian_decoder.dir/all] Error 2
make: *** [all] Error 2

[Hieu Hoang]

you usually have to set these env variables to make the linker find the libraries at compile and run time

   LD_LIBRARY_PATH

  LIBRARY_PATH 

Hieu Hoang

http://statmt.org/hieu

--
You received this message because you are subscribed to the Google Groups "marian-nmt" group.
To unsubscribe from this group and stop receiving emails from it, send an email to marian-nmt+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/marian-nmt/a221bf47-8a54-4b44-aa1f-eed6aaf86569o%40googlegroups.com.

[Steven Bradtke]

Hmmm, I've tried that, but no luck.

Inspecting the link.text that is generated for marian-decoder, the complete path to libprotobuf.so is included in the link command, bypassing the search path. That is, instead of -lprotobuf, we have /home/ec2-user/tools/lib/libprotobuf.so

Steve

To unsubscribe from this group and stop receiving emails from it, send an email to maria...@googlegroups.com.

[Roman Grundkiewicz]

Do you set all required CMake flags as described in https://marian-nmt.github.io/docs/#sentencepiece? Did it not work?

[Steven Bradtke]

Yes, I've tried that. I've tried letting cmake find protobuf for itself after setting CMAKE_PREFIX_PATH, and explicitly telling cmake where to find protobuf as described in the marian docs. 

I'm using cmake 3.17.3. Earlier versions of cmake (I forget exactly which) could not find a locally installed protobuf for themselves, even after setting CMAKE_PREFIX_PATH, but 3.17.3 has no problem.

I am trying to use protubuf 3.12.3. Is there an earlier, required version?

[Steven Bradtke]

It turns out that the problem really wasn't w/ specifying the correct protoc. The problem was that I wanted to use a version of GCC/G++ that I compiled and installed locally, not in the system paths. CMAKE gave every indication that it was using my version of the compiler during configuration, but it seems to have tried to link with the libraries of the system compiler, at least when building sentencepiece. Other tools built and linked just fine.