Compiling kafka c++ with bazel failed

[yanlei yu]

BUILD ：

load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library")

cc_library(
    name = "rdkafkacpp",
    hdrs = ["rdkafkacpp.h"],
)

cc_library(
    name = "rdkafka",
    hdrs = ["rdkafka.h"],
)

cc_library(
    name = "rdkafka_mock",
    hdrs = ["rdkafka_mock.h"],
)

cc_binary(
    name = "producer",
    srcs = ["producer.cpp"],
    deps = [
        ":rdkafkacpp",
        ":rdkafka",
        ":rdkafka_mock",
    ],
)

error info:

bazel build //main:producer
Starting local Bazel server and connecting to it...
INFO: Analyzed target //main:producer (15 packages loaded, 63 targets configured).
INFO: Found 1 target...
ERROR: /home/kafka/software/kafkac/main/BUILD:18:10: Linking main/producer failed: (Exit 1): gcc failed: error executing command /bin/gcc @bazel-out/k8-fastbuild/bin/main/producer-2.params

Use --sandbox_debug to see verbose messages from the sandbox
bazel-out/k8-fastbuild/bin/main/_objs/producer/producer.pic.o:producer.cc:function main: error: undefined reference to 'RdKafka::Conf::create(RdKafka::Conf::ConfType)'
bazel-out/k8-fastbuild/bin/main/_objs/producer/producer.pic.o:producer.cc:function main: error: undefined reference to 'RdKafka::Producer::create(RdKafka::Conf const*, std::string&)'
bazel-out/k8-fastbuild/bin/main/_objs/producer/producer.pic.o:producer.cc:function main: error: undefined reference to 'RdKafka::Topic::PARTITION_UA'
bazel-out/k8-fastbuild/bin/main/_objs/producer/producer.pic.o:producer.cc:function main: error: undefined reference to 'RdKafka::err2str(RdKafka::ErrorCode)'
collect2: error: ld returned 1 exit status
Target //main:producer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 4.004s, Critical Path: 0.33s
INFO: 6 processes: 5 internal, 1 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

• Does anyone have this problem?

[Fædon Jóhannes Sinis]

This might not be your issue, but did you look at the following discussion? At some point you might run into the missing config.h:

https://github.com/bazelbuild/rules_foreign_cc/issues/243#issue-430040687

Where are you linking in the srcs for rdkafka? I'd expect to see something like this (note this example doesn't quite work because of the issue in the above link), but in your BUILD file I don't see any reference to 'src-cpp/*.cpp' as follows:

http_archive(

    name = "librdkafka",

    build_file_content =

        """

cc_library(

    name = 'librdkafka',

    srcs = glob(['src-cpp/*.cpp']),

    includes = ['.'],

    hdrs = glob(['src-cpp/*.h', "src/rdkafka.h"]),

    visibility = ['//visibility:public'],

)

""",

    strip_prefix = "librdkafka-2.1.1",

    urls = ["https://github.com/confluentinc/librdkafka/archive/refs/tags/v2.1.1.tar.gz"],

)

[Fædon Jóhannes Sinis]

In this meantime, the following feels a bit hacky but seems to work, if you install the libraries using something like

sudo apt install librdkafka-dev

and then in your WORKSPACE:

new_local_repository(

    name = "librdkafka",

    build_file_content =

        """

cc_import(

    name = "librdkafka",

    hdrs = glob(["include/librdkafka/*.h"]),

    includes = ['include/librdkafka'],

    shared_library = "lib/x86_64-linux-gnu/librdkafka++.so",

    visibility = ["//visibility:public"],

)

""",

    path = "/usr",

)

and then in your BUILD file:

cc_binary(

    name = "producer",

    srcs = ["producer.cc"],

    deps = [

        "@librdkafka",

    ],

)

[Austin Schuh]

Bazel likes to check that intermediate libraries have all the symbols
required in either themselves or their dependencies. That looks like
what you are running into. I'd recommend looking into what files
provide RdKafka::Conf::create (and the other missing symbols) and make
sure they are in your cc_library.

Austin

> --
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/539515d6-7946-4e62-844f-6d9e0593cfb4n%40googlegroups.com.

[yanlei yu]

I tried this configuration and still had problems：

my WORKSPACE：

new_local_repository(
    name = "librdkafka",
    build_file_content =
        """
cc_import(
    name = "librdkafka",
    hdrs = glob(["include/librdkafka/*.h"]),
    includes = ['include/librdkafka'],
    shared_library = "lib/x86_64-linux-gnu/librdkafka++.so",
    visibility = ["//visibility:public"],
)
""",
   path = "/usr",
)

my  main/BUILD:

load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library")

cc_binary(
    name = "producer",
    srcs = ["producer.cc"],
    deps = [
        "@librdkafka",
    ],
)

After executing bazel build //main:producer error message:

Starting local Bazel server and connecting to it...

ERROR: /home/kafka/.cache/bazel/_bazel_kafka/8c7613a1577cc298cb554cf9b5102207/external/librdkafka/BUILD.bazel:2:10: @librdkafka//:librdkafka: no such attribute 'includes' in 'cc_import' rule
ERROR: /home/kafka/software/kafkac/main/BUILD:3:10: Target '@librdkafka//:librdkafka' contains an error and its package is in error and referenced by '//main:producer'
ERROR: Analysis of target '//main:producer' failed; build aborted: Analysis failed
INFO: Elapsed time: 4.054s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (13 packages loaded, 18 targets configured)

[yanlei yu]

• After I remove includes from cc_import, execute it 

•  bazel build //main:producer error message: 
  

Starting local Bazel server and connecting to it...

INFO: Analyzed target //main:producer (16 packages loaded, 63 targets configured).
INFO: Found 1 target...
ERROR: /home/kafka/software/kafkac/main/BUILD:3:10: Compiling main/producer.cc failed: (Exit 1): gcc failed: error executing command /bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 24 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox

main/producer.cc:5:21: fatal error: rdkafka.h: No such file or directory
 #include "rdkafka.h"
                     ^
compilation terminated.

Target //main:producer failed to build
Use --verbose_failures to see the command lines of failed build steps.

INFO: Elapsed time: 3.836s, Critical Path: 0.18s
INFO: 6 processes: 6 internal.

FAILED: Build did NOT complete successfully

• Maybe my BUILD configuration is wrong?

[Fædon Jóhannes Sinis]

It's strange that includes is not part of the core documentation for cc_import:

https://bazel.build/reference/be/c-cpp#cc_import

but there is this comment appended to the end of that section that "cc_import supports an include attribute" (see screenshot below). 

Which version of bazel are you running and which platform?

[yanlei yu]

My bazel version:

Build label: 4.2.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Aug 30 00:00:00 2021 (1630281600)
Build timestamp: 1630281600
Build timestamp as int: 1630281600

[yanlei yu]

• The machine system is CentOS7

[Fædon Jóhannes Sinis]

The comment in question was only checked in a few months ago, so I wonder if this was not supported in Bazel 4 (see https://github.com/bazelbuild/bazel/issues/15222)

Your version of Bazel seems quite old; I am on version 6.2.1 . Why don't you switch to using Bazelisk and see if this fixes your issue? https://bazel.build/install/bazelisk

[yanlei yu]

I am in accordance with the https://bazel.build/install/redhat? hl=zh-cn, bazel4 installed on centos7, I have tried to install version 5 and version 6 on my machine, the installation was not successful

Is there any other way to install bazel6 on centos7 besides yum

[yanlei yu]

The yum repo provided at this address also does not have a version 6 version

https://download.copr.fedorainfracloud.org/results/vbatts/bazel/epel-8-x86_64/

（epel-8、epel-9 There is no version 6）

在2023年7月10日星期一 UTC+8 17:45:45<phaedo...@gmail.com> 写道：

[yanlei yu]

I am using 6.2.1 version after the build is still not successful, error：

bazel clean --expunge
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

[root@hdp01v kafkac]# bazel build //main:producer

Starting local Bazel server and connecting to it...

INFO: Analyzed target //main:producer (37 packages loaded, 162 targets configured).
INFO: Found 1 target...
ERROR: /home/kafka/software/kafkac/main/BUILD:3:10: Compiling main/producer.cc failed: (Exit 1): gcc failed: error executing command (from target //main:producer) /bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 26 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging

main/producer.cc:5:21: fatal error: rdkafka.h: No such file or directory
 #include "rdkafka.h"
                     ^
compilation terminated.
Target //main:producer failed to build
Use --verbose_failures to see the command lines of failed build steps.

INFO: Elapsed time: 5.588s, Critical Path: 0.23s

INFO: 6 processes: 6 internal.
FAILED: Build did NOT complete successfully

main/BUILD:

load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library")

cc_binary(
    name = "producer",
    srcs = ["producer.cc"],
    deps = [
        "@librdkafka",
    ],
)

producer.cc:

#include <iostream>
#include <string>
#include <list>
#include <stdint.h>
#include "rdkafka.h"
#include "rdkafkacpp.h"
#include <list>
#include <cpp2sky/config.pb.h>
using namespace std;

......

在2023年7月10日星期一 UTC+8 17:45:45<phaedo...@gmail.com> 写道：

[Fædon Jóhannes Sinis]

This looks like progress, it just can't find the header.

Have you verified that all the paths in the cc_import section are correct? I'm not familiar with CentOS, and it sounds like you had to install the current version of bazel manually, so I imagine the install locations might be different than the defaults on Ubuntu.

[yanlei yu]

I used 6.2.1 and copied the.h file of librdkafka into my project's workspace and added cc_library to BUILD：

load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library")

cc_library(

    name = "rdkafka",
    hdrs = ["rdkafka.h"],
)

cc_library(

    name = "rdkafkacpp",
    hdrs = ["rdkafkacpp.h"],
)

It now compiles successfully，thank you