How to properly do static linking within CMake project

[Marcin Lewandowski]

Hello,

I am working on a library where I would use protobuf. 

I need to link it statically. The target platform is macOS. 

The usual way of fetching dependencies (brew install protobuf) is no go, as it comes only with shared libraries.

I tried forcing homebrew to rebuild protobuf (brew install protobuf --build-from-source) but it fails as some tests are not passing (logs [1] at the end of this message).

I fetched the source code (27.3) release from github, run 

bazel build :portico :protobuf 

inside the source directory, but while it compiles protoc compiler, there's no CMake/protobuf-lite library in the output.

Apparently the runtime is based on CMake, so I run 

cmake -S . -B build -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_CXX_STANDARD=17 -Dprotobuf_ABSL_PROVIDER=package -Dprotobuf_JSONCPP_PROVIDER=package

and then issued compilation in build/ dir, and yay, it compiled protobuf.

However, when I added the build/cmake to CMAKE_PREFIX_PATH it complains about missing build/cmake/protobuf/protobuf-targets.cmake

So I tried adding the repository as git submodule, and adding it via add_subdirectory CMake command, then it fails with

CMake Error: install(EXPORT "protobuf-targets" ...) includes target "libprotobuf-lite" which requires target "absl_absl_check" that is not in any export set.
CMake Error: install(EXPORT "protobuf-targets" ...) includes target "libprotobuf-lite" which requires target "absl_absl_log" that is not in any export set.
CMake Error: install(EXPORT "protobuf-targets" ...) includes target "libprotobuf-lite" which requires target "absl_algorithm" that is not in any export set.
CMake Error: install(EXPORT "protobuf-targets" ...) includes target "libprotobuf-lite" which requires target "absl_base" that is not in any export set.

...

abseil is added as add_subdirectory as well.

I run into conclusion that I must be doing something fundamentally wrong, as I don't expect such popular piece of code to be so hard to integrate.

Any suggestions on how to properly embed protobuf within such a project?

Thanks in advance,

Marcin

[1] Failed tests:

2: [ RUN      ] TextFormatParserTest.ParseFieldValueFromString
2: E0000 00:00:1723926116.158054 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158063 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158079 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158082 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158084 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: -
2: E0000 00:00:1723926116.158085 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158087 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158089 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: -
2: E0000 00:00:1723926116.158090 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158092 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158094 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: -
2: E0000 00:00:1723926116.158095 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158097 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158101 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:5: Invalid value for boolean field "optional_bool". Value: "tRue".
2: E0000 00:00:1723926116.158106 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:6: Invalid value for boolean field "optional_bool". Value: "faLse".
2: E0000 00:00:1723926116.158108 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (2)
2: E0000 00:00:1723926116.158109 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected identifier, got: -
2: E0000 00:00:1723926116.158111 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:3: Invalid value for boolean field "optional_bool". Value: "on".
2: E0000 00:00:1723926116.158113 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:2: Invalid value for boolean field "optional_bool". Value: "a".
2: E0000 00:00:1723926116.158130 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected double, got: a
2: E0000 00:00:1723926116.158132 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected double, got: a
2: E0000 00:00:1723926116.158134 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expect a decimal number, got: 0xf
2: E0000 00:00:1723926116.158136 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expect a decimal number, got: 012
2: E0000 00:00:1723926116.158138 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected string, got: hello
2: E0000 00:00:1723926116.158141 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:7: Unknown enumeration value of "FOOBAR" for field "optional_nested_enum".
2: E0000 00:00:1723926116.158143 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected "{", found "any".

[Claude Robitaille]

I have been using protobuf with static linking for years and moved to cmake a few months ago.

Since I always cross-compile I never use the system installed library and tools so my flow is this

1 - Build, using cmake anything that must be using the local compiler (in the case of protobuf because protoc is needed for later). ATM it is only protobuf

2 - Build all the dependencies, also using cmake

3 - Build my project, including internal libraries.

For the first step, ie building protobuf and produce a static archive (a file ending with .a) I use this excerpt from the corresponding cmake:
include(ExternalProject)

ExternalProject_Add(
    protobuf
    GIT_REPOSITORY https://github.com/protocolbuffers/protobuf.git
    GIT_TAG v3.20.3
    SOURCE_SUBDIR cmake
    CMAKE_ARGS
        -Dprotobuf_BUILD_TESTS=OFF
        -Dprotobuf_BUILD_EXAMPLES=OFF
        -DCMAKE_INSTALL_PREFIX=$ENV{INSTALL}/protobuf
        -Dprotobuf_BUILD_CONFORMANCE=OFF
        -Dprotobuf_BUILD_EXAMPLES=OFF
        -Dprotobuf_BUILD_LIBPROTOC=ON
        -Dprotobuf_BUILD_SHARED_LIBS=OFF
        -Dprotobuf_MSVC_STATIC_RUNTIME=OFF
        -Dprotobuf_WITH_ZLIB=OFF
    BUILD_BYPRODUCTS
        $ENV{INSTALL}/lib/libprotobuf.a
        $ENV{INSTALL}/bin/protoc
)

Note that the install prefix comes from an environment variable. It is normally set to ./build  In other word, protobuf and the cmake files are in a child directory off the main build location (which is used by step 3 above). Also, SHARED_LIBS is Off meaning that a static library is created.

And then in my main project (so step 3 above), I use this to process my proto files (I use to only deal with c++ hence the name; I added python at a later stage):

set(GLOBAL_PROTO_FILES "" CACHE INTERNAL "")

# Define the append function used by other CMakeList files
function(append_proto_file filename)
    set(GLOBAL_PROTO_FILES ${GLOBAL_PROTO_FILES} ${filename} CACHE INTERNAL "")
endfunction()

#
# The proto file processing function
#

function(CREATE_CPP_PROTO SRCS HDRS)
    list(REMOVE_DUPLICATES GLOBAL_PROTO_FILES)
   
    set(${SRCS})
    set(${HDRS})  

    #printList(VAR ${PROTO_SRC_TREE} TITLE "Proto location location")
    #printList(VAR ${GLOBAL_PROTO_FILES} TITLE "Proto files")
 
    add_library(proto-objects OBJECT ${GLOBAL_PROTO_FILES})
    target_link_libraries(proto-objects PUBLIC protobuf::libprotobuf)
    target_include_directories(proto-objects PUBLIC "$<BUILD_INTERFACE:$ENV{GENERATED}>")
   
    make_directory($ENV{GENERATED})
    make_directory($ENV{GENERATED}/python)
   
    # This is a function provided by the cmake binding for protobuf
    # C++
    protobuf_generate(
      TARGET proto-objects
      LANGUAGE cpp
      PROTOC_OUT_DIR $ENV{GENERATED}
      IMPORT_DIRS ${MODELS_DIR};${PROTO_SRC_TREE}
      #IMPORT_DIRS ${MODELS_DIR}
      #PROTOC_OPTIONS "--cpp_opt=paths=source_relative"
      #PROTOC_OPTIONS "--proto_path=${MODELS_DIR}"
    )
   
    # Python    
    protobuf_generate(
      TARGET proto-objects
      LANGUAGE python
      PROTOC_OUT_DIR $ENV{GENERATED}/python
      IMPORT_DIRS ${MODELS_DIR};${PROTO_SRC_TREE}
    )

    target_include_directories(proto-objects PRIVATE $ENV{GENERATED})
       
    # Probably useless but other targets depends on it, so we keep it but it does nothing. In future we could probably remove
    # reference to it.
    add_custom_target(PROTO_FILES_READY
      COMMENT "Moving .pb.h and _pb2.py files to replicate source directory structure"
    )  
 
    set_source_files_properties(${${SRCS}} ${${HDRS}} PROPERTIES GENERATED TRUE)
    set(${SRCS} ${${SRCS}} PARENT_SCOPE)
    set(${HDRS} ${${HDRS}} PARENT_SCOPE)
endfunction()

[Claude Robitaille]

I forgot that this is required in my main project top CMakeList.txt:

find_package(protobuf REQUIRED PATHS $ENV{INSTALL}/protobuf/lib/cmake/protobuf)
include($ENV{INSTALL}/protobuf/lib/cmake/protobuf/protobuf-module.cmake)

Also, I am on Linux but I do not see why it would be different on macOS.

[Petras Vestartas]

Hi,

Do you have cmake that works for the current version of protubuf?

I also do not understand the versioning logic, you are using v3.20 version while one year later we have v32 version.

[Em Rauch]

> I also do not understand the versioning logic, you are using v3.20 version while one year later we have v32 version.

https://protobuf.dev/support/version-support/

 C++Proto has gone from 3.20 to 6.32 in the time period you are looking at.

We use a monotonically increasing minor that is the same across all languages, which is the release or protoc version which is "32.0". Each language uses that for its minor/point but has a language-specific major version: the 32.0 protoc is the one for C++ 6.32.0, C# 3.32.0, java 4.32.0. 

In the past few years we have shifted around where we shifted some places that were cross-language topics from calling it "3.20.0" to referring to it as "20.0" which is what you are noticing, but 3.20.0 was already cross-language-version=20.0.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/protobuf/50117e7f-799a-4019-97c2-8817cf45e929n%40googlegroups.com.