How to properly do static linking within CMake project

[Marcin Lewandowski]

Hello,

I am working on a library where I would use protobuf. 

I need to link it statically. The target platform is macOS. 

The usual way of fetching dependencies (brew install protobuf) is no go, as it comes only with shared libraries.

I tried forcing homebrew to rebuild protobuf (brew install protobuf --build-from-source) but it fails as some tests are not passing (logs [1] at the end of this message).

I fetched the source code (27.3) release from github, run 

bazel build :portico :protobuf 

inside the source directory, but while it compiles protoc compiler, there's no CMake/protobuf-lite library in the output.

Apparently the runtime is based on CMake, so I run 

cmake -S . -B build -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_CXX_STANDARD=17 -Dprotobuf_ABSL_PROVIDER=package -Dprotobuf_JSONCPP_PROVIDER=package

and then issued compilation in build/ dir, and yay, it compiled protobuf.

However, when I added the build/cmake to CMAKE_PREFIX_PATH it complains about missing build/cmake/protobuf/protobuf-targets.cmake

So I tried adding the repository as git submodule, and adding it via add_subdirectory CMake command, then it fails with

CMake Error: install(EXPORT "protobuf-targets" ...) includes target "libprotobuf-lite" which requires target "absl_absl_check" that is not in any export set.
CMake Error: install(EXPORT "protobuf-targets" ...) includes target "libprotobuf-lite" which requires target "absl_absl_log" that is not in any export set.
CMake Error: install(EXPORT "protobuf-targets" ...) includes target "libprotobuf-lite" which requires target "absl_algorithm" that is not in any export set.
CMake Error: install(EXPORT "protobuf-targets" ...) includes target "libprotobuf-lite" which requires target "absl_base" that is not in any export set.

...

abseil is added as add_subdirectory as well.

I run into conclusion that I must be doing something fundamentally wrong, as I don't expect such popular piece of code to be so hard to integrate.

Any suggestions on how to properly embed protobuf within such a project?

Thanks in advance,

Marcin

[1] Failed tests:

2: [ RUN      ] TextFormatParserTest.ParseFieldValueFromString
2: E0000 00:00:1723926116.158054 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158063 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158079 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158082 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158084 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: -
2: E0000 00:00:1723926116.158085 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158087 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158089 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: -
2: E0000 00:00:1723926116.158090 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158092 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158094 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: -
2: E0000 00:00:1723926116.158095 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected integer, got: a
2: E0000 00:00:1723926116.158097 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (999999999999999999999999999999999999)
2: E0000 00:00:1723926116.158101 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:5: Invalid value for boolean field "optional_bool". Value: "tRue".
2: E0000 00:00:1723926116.158106 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:6: Invalid value for boolean field "optional_bool". Value: "faLse".
2: E0000 00:00:1723926116.158108 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Integer out of range (2)
2: E0000 00:00:1723926116.158109 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected identifier, got: -
2: E0000 00:00:1723926116.158111 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:3: Invalid value for boolean field "optional_bool". Value: "on".
2: E0000 00:00:1723926116.158113 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:2: Invalid value for boolean field "optional_bool". Value: "a".
2: E0000 00:00:1723926116.158130 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected double, got: a
2: E0000 00:00:1723926116.158132 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected double, got: a
2: E0000 00:00:1723926116.158134 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expect a decimal number, got: 0xf
2: E0000 00:00:1723926116.158136 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expect a decimal number, got: 012
2: E0000 00:00:1723926116.158138 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected string, got: hello
2: E0000 00:00:1723926116.158141 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:7: Unknown enumeration value of "FOOBAR" for field "optional_nested_enum".
2: E0000 00:00:1723926116.158143 5519321 text_format.cc:417] Error parsing text-format protobuf_unittest.TestAllTypes: 1:1: Expected "{", found "any".

[Claude Robitaille]

I have been using protobuf with static linking for years and moved to cmake a few months ago.

Since I always cross-compile I never use the system installed library and tools so my flow is this

1 - Build, using cmake anything that must be using the local compiler (in the case of protobuf because protoc is needed for later). ATM it is only protobuf

2 - Build all the dependencies, also using cmake

3 - Build my project, including internal libraries.

For the first step, ie building protobuf and produce a static archive (a file ending with .a) I use this excerpt from the corresponding cmake:
include(ExternalProject)

ExternalProject_Add(
    protobuf
    GIT_REPOSITORY https://github.com/protocolbuffers/protobuf.git
    GIT_TAG v3.20.3
    SOURCE_SUBDIR cmake
    CMAKE_ARGS
        -Dprotobuf_BUILD_TESTS=OFF
        -Dprotobuf_BUILD_EXAMPLES=OFF
        -DCMAKE_INSTALL_PREFIX=$ENV{INSTALL}/protobuf
        -Dprotobuf_BUILD_CONFORMANCE=OFF
        -Dprotobuf_BUILD_EXAMPLES=OFF
        -Dprotobuf_BUILD_LIBPROTOC=ON
        -Dprotobuf_BUILD_SHARED_LIBS=OFF
        -Dprotobuf_MSVC_STATIC_RUNTIME=OFF
        -Dprotobuf_WITH_ZLIB=OFF
    BUILD_BYPRODUCTS
        $ENV{INSTALL}/lib/libprotobuf.a
        $ENV{INSTALL}/bin/protoc
)

Note that the install prefix comes from an environment variable. It is normally set to ./build  In other word, protobuf and the cmake files are in a child directory off the main build location (which is used by step 3 above). Also, SHARED_LIBS is Off meaning that a static library is created.

And then in my main project (so step 3 above), I use this to process my proto files (I use to only deal with c++ hence the name; I added python at a later stage):

set(GLOBAL_PROTO_FILES "" CACHE INTERNAL "")

# Define the append function used by other CMakeList files
function(append_proto_file filename)
    set(GLOBAL_PROTO_FILES ${GLOBAL_PROTO_FILES} ${filename} CACHE INTERNAL "")
endfunction()

#
# The proto file processing function
#

function(CREATE_CPP_PROTO SRCS HDRS)
    list(REMOVE_DUPLICATES GLOBAL_PROTO_FILES)
   
    set(${SRCS})
    set(${HDRS})  

    #printList(VAR ${PROTO_SRC_TREE} TITLE "Proto location location")
    #printList(VAR ${GLOBAL_PROTO_FILES} TITLE "Proto files")
 
    add_library(proto-objects OBJECT ${GLOBAL_PROTO_FILES})
    target_link_libraries(proto-objects PUBLIC protobuf::libprotobuf)
    target_include_directories(proto-objects PUBLIC "$<BUILD_INTERFACE:$ENV{GENERATED}>")
   
    make_directory($ENV{GENERATED})
    make_directory($ENV{GENERATED}/python)
   
    # This is a function provided by the cmake binding for protobuf
    # C++
    protobuf_generate(
      TARGET proto-objects
      LANGUAGE cpp
      PROTOC_OUT_DIR $ENV{GENERATED}
      IMPORT_DIRS ${MODELS_DIR};${PROTO_SRC_TREE}
      #IMPORT_DIRS ${MODELS_DIR}
      #PROTOC_OPTIONS "--cpp_opt=paths=source_relative"
      #PROTOC_OPTIONS "--proto_path=${MODELS_DIR}"
    )
   
    # Python    
    protobuf_generate(
      TARGET proto-objects
      LANGUAGE python
      PROTOC_OUT_DIR $ENV{GENERATED}/python
      IMPORT_DIRS ${MODELS_DIR};${PROTO_SRC_TREE}
    )

    target_include_directories(proto-objects PRIVATE $ENV{GENERATED})
       
    # Probably useless but other targets depends on it, so we keep it but it does nothing. In future we could probably remove
    # reference to it.
    add_custom_target(PROTO_FILES_READY
      COMMENT "Moving .pb.h and _pb2.py files to replicate source directory structure"
    )  
 
    set_source_files_properties(${${SRCS}} ${${HDRS}} PROPERTIES GENERATED TRUE)
    set(${SRCS} ${${SRCS}} PARENT_SCOPE)
    set(${HDRS} ${${HDRS}} PARENT_SCOPE)
endfunction()

[Claude Robitaille]

I forgot that this is required in my main project top CMakeList.txt:

find_package(protobuf REQUIRED PATHS $ENV{INSTALL}/protobuf/lib/cmake/protobuf)
include($ENV{INSTALL}/protobuf/lib/cmake/protobuf/protobuf-module.cmake)

Also, I am on Linux but I do not see why it would be different on macOS.

[Petras Vestartas]

Hi,

Do you have cmake that works for the current version of protubuf?

I also do not understand the versioning logic, you are using v3.20 version while one year later we have v32 version.