Trouble with ZeroMQ Go client with built-in C dependency

[Haixin Tie]

Hi Cgo experts,

I've been struggling for hours trying to make github.com/pebbe/zmq4 (ZeroMQ Go client) build with C dependency in Bazel. Even though `cgo = True` and `cdeps` on `libzmq` are used, Bazel seems to always have trouble seeing the zmq.h header file from libzmq C dependency.

What am I missing? I've created a minimal code example to show it: https://github.com/thx123/bazel-cgo (and README for steps).

I would greatly appreciate some guidance from Cgo experts! This is how the BUILD file look like:

```

load("@rules_go//go:def.bzl", "go_binary")

go_binary(
    name = "zmq_client",
    srcs = ["zmq_client.go"],
    deps = [
        "@com_github_pebbe_zmq4//:go_default_library",
    ],
    cgo = True,
    cdeps = [
        # Including the full library doesn't make it work, either.
        # "@libzmq//:libzmq",
        "@libzmq//:libzmq_headers_only",
    ],
)

```

Regards,

HaiXin

[Filip Filmar]

The command line for cgo does not contain the `-I` flag for the correct directory it would seem. 

This is the problem build step:

bazel build @@gazelle~~go_deps~com_github_pebbe_zmq4//:zmq4 --sandbox_debug

So it's not even in your source.  Looking at the cmdline, it does not contain the correct include dir flag at all.  Look for `-cppflags` in the long text below. I'll check why, but chances are it's not you.

```

  (cd /home/fmil/.cache/bazel/_bazel_fmil/82a7bf6c7aa74efc21460f76993e3263/sandbox/linux-sandbox/15/execroot/_main && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GODEBUG='winsymlink=0' \
    GOEXPERIMENT=nocoverageredesign \
    GOOS=linux \
    GOPATH='' \
    GOROOT=bazel-out/k8-fastbuild/bin/external/rules_go~/stdlib_ \
    GOROOT_FINAL=GOROOT \
    GOTOOLCHAIN=local \
    PATH=/usr/bin:/bin \
    TMPDIR=/usr/local/google/tmp \
    ZERO_AR_DATE=1 \
  /home/fmil/.cache/bazel/_bazel_fmil/install/ec66cb4ed623de0db7a7f89408b9b468/linux-sandbox -t 15 -w /dev/shm -w /home/fmil/.cache/bazel/_bazel_fmil/82a7bf6c7aa74efc21460f76993e3263/sandbox/linux-sandbox/15/execroot/_main -w /tmp -w /usr/local/google/tmp -M /home/fmil/.cache/bazel/_bazel_fmil/82a7bf6c7aa74efc21460f76993e3263/sandbox/linux-sandbox/15/_hermetic_tmp -m /tmp -S /home/fmil/.cache/bazel/_bazel_fmil/82a7bf6c7aa74efc21460f76993e3263/sandbox/linux-sandbox/15/stats.out -D /home/fmil/.cache/bazel/_bazel_fmil/82a7bf6c7aa74efc21460f76993e3263/sandbox/linux-sandbox/15/debug.out -- bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_go~~go_sdk~bazel-cgo__download_0/builder_reset/builder compilepkg -sdk external/rules_go~~go_sdk~bazel-cgo__download_0 -goroot bazel-out/k8-fastbuild/bin/external/rules_go~/stdlib_ -installsuffix linux_amd64 -src external/gazelle~~go_deps~com_github_pebbe_zmq4/auth.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/ctxoptions_unix.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/ctxoptions_windows.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/doc.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/errors.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/polling.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/reactor.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/socketget.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/socketget_unix.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/socketget_windows.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/socketset.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/utils.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/wrappers_unix.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/wrappers_windows.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/zmq4.go -src external/gazelle~~go_deps~com_github_pebbe_zmq4/dummy.c -src external/gazelle~~go_deps~com_github_pebbe_zmq4/zmq4.h -embedroot '' -embedroot bazel-out/k8-fastbuild/bin -embedlookupdir external/gazelle~~go_deps~com_github_pebbe_zmq4 -importpath github.com/pebbe/zmq4 -p github.com/pebbe/zmq4 -package_list bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_go~~go_sdk~bazel-cgo__download_0/packages.txt -lo bazel-out/k8-fastbuild/bin/external/gazelle~~go_deps~com_github_pebbe_zmq4/zmq4.a -o bazel-out/k8-fastbuild/bin/external/gazelle~~go_deps~com_github_pebbe_zmq4/zmq4.x -gcflags '' -asmflags '' -cppflags '-I external/gazelle~~go_deps~com_github_pebbe_zmq4 -iquote .' -cflags '-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fPIC' -cxxflags '-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++14 -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fPIC' -objcflags '-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fPIC' -objcxxflags '-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++14 -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fPIC' -ldflags '-fuse-ld=gold -B/usr/bin -Wl,-no-as-needed -Wl,-z,relro,-z,now -pass-exit-codes -Wl,--push-state,-as-needed -lstdc++ -Wl,--pop-state -Wl,--push-state,-as-needed -lm -Wl,--pop-state')

```

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/e91d9c23-e22f-4e5d-aa6c-ccdcbf04f6dan%40googlegroups.com.

[Filip Filmar]

Fix might be easy, give me a moment.

[Filip Filmar]

Ok, so the fix isn't obvious to me sadly.

The issue is that go_deps extension generates a BUILD file for the dependency com_github_pebbe_zmq4 that does *not* depend on `libzmq` at all. Presumably it relies on libzmq being preinstalled on your machine by default.

I am not well versed in bzlmod so I don't know how you could teach it about @libzmq that you compiled.

On ubuntu, if you do `sudo apt install libczmq-dev`, you will get past the `zmq.h` error, but you will get stuck in the linking phase.

However, the symbols in the libczmq-dev aren't the same as the wrapper expects. The ones used in the wrapper have a `zmq4_` prefix, whereas the ones actually offered in the library available in Ubuntu do not.

So the link step is completely broken. It's not you.

Since this is not a hermetic library, it's a "works for me" sort of situation.

F

[Filip Filmar]

Friends don't let friends compile non-hermetically.

F

[Filip Filmar]

OK, two bugs, none yours, really.

(1) go_mods extension is not hermetic, and depends on the library zmq4 being installed on your machine. I would call this a bug.

(2) go_binary rule seems to *ignore* all c-related options: clinkopts, cxxopts, cdeps, etc.  I tested it even by adding completely bogus strings there, and they had no effect on compilation whatsoever. Adding `cdeps = [ "@libzmq//:libzmq" should cause a link option `-L <dir_to_zmq_library>` to appear in the linker command line, but it doesn't. I even added `clinkopts = [ "oogabooga" ]`, and no new errors happened.

F

[Filip Filmar]

Here's the result, but I think no fixes, just exposing bugs. Sorry.

https://github.com/thx123/bazel-cgo/pull/1/files

F

[Filip Filmar]

On Mon, Sep 30, 2024 at 2:48 PM HaiXin Tie <thx...@gmail.com> wrote:

Thanks Filip for looking into this. Is there any practical way to patch https://github.com/pebbe/zmq4 such that it can link libzmq C++ dependency properly?

Yes I do it all the time, but only based on WORKSPACE. The general approach for me is: (1) fork the repo; (2) use the trick from below to use that code instead of the original; (3) use `git diff HEAD^..HEAD` to formulate a patch, then add it to the `patches = [...]` arg so bazel automatically fixes your repo up when it gets downloaded.

I tried to figure out how one would do that in bzlmod world, and it's not obvious to me.  This is why I thought first "this will be easy to fix" thinking yeah I'll just hack the WORKSPACE file. But then I discovered bzlmod and it was "aw shucks".

I was able to discover all of this by using the repo override so I could make local changes. See here for the trick: https://hdlfactory.com/post/2023/07/30/bazel-tips-and-tricks/#:~:text=Use%20an%20alternate%20repository

You need to figure out the correct repo name in bzlmod world, which is nontrivial. And I have no idea how it will interact with `go_mod` extension.

 

If this problem cannot be solved, I guess I'd have to leave any go-redis related code outside the Bazel-managed repository, which would be a great pity...

It can, it's just annoying.

Probably someone who's spent more on bazel modules might advise better.

F

[Filip Filmar]

On Mon, Sep 30, 2024 at 3:06 PM HaiXin Tie <thx...@gmail.com> wrote:

Could you elaborate which patch would solve this problem? 

A patch to the `go_binary` rule to cause it not to ignore its cgo-related arguments.  That would remove the main current issue, but could reveal more issues.

As I said there are two bugs.

One bug is that to build the go wrapper of libczmq you must have it installed on the host machine, and must be lucky for the host to be the same platform as target.

Now we mostly run these things on x86_64, so if you `sudo apt-get install libczmq4-dev` you will have the library installed and we can forget about it for now. For now. Ideally, the go wrapper would be built with a hermetic libzmq.

The other bug is, when you add `cdeps = [ "@libzmq//:libzmq"]`, this should cause the command line that builds your `//:zmq_server` and `//:zmq_client` to contain a flag `-L <path to libzmq.a that was built in the @libzmq repo>`.   It should also cause the files comprising target `@libzmq//:libzmq` to be in the sandbox of the rule, so that the linker can actually find those files.

But, for some reason this does not happen. This is why linking fails.

But also, even if we fixed that, there is the problem that the cgo wrapper for the zmq library is built with the host's libzmq, while your code is built with the hermetic libzmq. This could break if the two are different enough.  I didn't verify if this is the case, or we just got lucky, since it got late last night for me when I tried all of this and I had to stop trying.

F

[Filip Filmar]

I also think if the go libzmq wrapper would have been made hermetic, then it would propagate with it the needed linker flags, and your code would link just fine.

E.g. I used quite a few cgo wrappers, such as sqlite3 for example, and they all worked this way and worked just fine.

F

[Filip Filmar]

Also, CC rules are really smart (sometimes too much?), and perhaps the reason we're not finding `cdeps` to work is because it notices that within *this* rule, nothing uses specifically the C bindings. This is a *wild* speculation based on a similar-seeming, but quite different issue I had earlier.

All these are answered by staring at code long enough, but I didn't do that legwork. Things seem too broken for me to sink that time in for free.

[Filip Filmar]

On Mon, Sep 30, 2024 at 3:28 PM HaiXin Tie <thx...@gmail.com> wrote:

As for the bug of go wrapper of libzmq requiring local installation, I did try compiling it inside Bazel using Cmake (in the `fcc` branch of the same demo repo): https://github.com/thx123/bazel-cgo/blob/fcc/BUILD.bazel. But it didn't work, either.

Nah, I compiled it just fine, but forget if I had to nudge something around or not.

I sent you a pull request that shows the repo that I used to make it happen, so you may be able to replay it.

 

As for the linking bug. I also tried declaring a `go_library` target with `cgo` and `cdeps`, then a `go_binary` target on top of it, but it didn't seem to help, either.

The ordering should be irrelevant. Bazel never worked that way.

 

I really hope the official Bazel-Go team can provide some guidance on this. I've spent quite a number of hours on this, and is on the brink of giving it all up and falling back to using another repo based on Make, which is not hermetic, but works.

If all else fails, you can, in fact, build the binaries externally, and simply bring them into a bazel build if you must use bazel.

F

[Filip Filmar]

On Mon, Sep 30, 2024 at 3:52 PM HaiXin Tie <thx...@gmail.com> wrote:

IIUC, the linking still needs to be fixed to make the solution work?

Yes.

See this for example: https://github.com/thx123/bazel-cgo/pull/1/files#diff-7fc57714ef13c3325ce2a1130202edced92fcccc0c6db34a72f7b57f60d552a3R40

I just added a completely bogus string to the link args, and got no errors.

F

[Filip Filmar]

If this is important to you, I recommend checking out the source to the rules go locally, pointing your `--override_repository` flag to the local checkout, and adding some well placed `print`s. Rinse and repeat until you figure out what the problem is. rules_go are big, but the code of interest is not all that scary.

Sorry, but this is (still) a necessary skill for a bazel user. DIY is practically a requirement.

F

On Mon, Sep 30, 2024 at 4:00 PM HaiXin Tie <thx...@gmail.com> wrote:

Yes, I saw your code, hence that question :)

I guess until some Bazel Go team experts can come out to help us, there won't be clear fixes to the problems discovered so far.

Anyway, Thanks for your extended assistance, Filip!

Regards,

HaiXin

[HaiXin Tie]

Thanks Filip for looking into this. Is there any practical way to patch https://github.com/pebbe/zmq4 such that it can link libzmq C++ dependency properly?

If this problem cannot be solved, I guess I'd have to leave any go-redis related code outside the Bazel-managed repository, which would be a great pity...

Regards,

HaiXin

[HaiXin Tie]

Thanks for spending the time late Sunday night to troubleshoot this. Really appreciate it. Totally agreed with you that the go zmq wrapper should have been made hermetic.

As for the bug of go wrapper of libzmq requiring local installation, I did try compiling it inside Bazel using Cmake (in the `fcc` branch of the same demo repo): https://github.com/thx123/bazel-cgo/blob/fcc/BUILD.bazel. But it didn't work, either.

As for the linking bug. I also tried declaring a `go_library` target with `cgo` and `cdeps`, then a `go_binary` target on top of it, but it didn't seem to help, either.

For cross-compilation, we could use Bazel's conditions to build different targets based on target OS, but that would require some additional starlark coding. But we'd have to solve the first two harder bugs first before doing this.

I really hope the official Bazel-Go team can provide some guidance on this. I've spent quite a number of hours on this, and is on the brink of giving it all up and falling back to using another repo based on Make, which is not hermetic, but works.

Regards,

HaiXin

[HaiXin Tie]

Actually I have a working version on how to apply a local patch in bzlmod. It's actually pretty simple (by looking into bzlmod code examples, as the official doc did not cover it):

https://github.com/thx123/bazel-examples/blob/main/MODULE.bazel#L8

https://github.com/thx123/bazel-examples/blob/main/cfg-bazel/third_party/abseil-cpp/abseil-cpp.patch

Could you elaborate which patch would solve this problem? I went through your earlier replies but not 100% sure which code snippet could do the magic. Thanks!

Regards,

HaiXin

[HaiXin Tie]

On Mon, Sep 30, 2024 at 3:31 PM Filip Filmar <fil...@gmail.com> wrote:

On Mon, Sep 30, 2024 at 3:28 PM HaiXin Tie <thx...@gmail.com> wrote:

As for the bug of go wrapper of libzmq requiring local installation, I did try compiling it inside Bazel using Cmake (in the `fcc` branch of the same demo repo): https://github.com/thx123/bazel-cgo/blob/fcc/BUILD.bazel. But it didn't work, either.

Nah, I compiled it just fine, but forget if I had to nudge something around or not.

I sent you a pull request that shows the repo that I used to make it happen, so you may be able to replay it.

Yes I saw your PR (https://github.com/thx123/bazel-cgo/pull/1). Thanks. IIUC, the linking still needs to be fixed to make the solution work? Sorry if I missed anything important.

 

As for the linking bug. I also tried declaring a `go_library` target with `cgo` and `cdeps`, then a `go_binary` target on top of it, but it didn't seem to help, either.

The ordering should be irrelevant. Bazel never worked that way.

Ack'ed.

 

I really hope the official Bazel-Go team can provide some guidance on this. I've spent quite a number of hours on this, and is on the brink of giving it all up and falling back to using another repo based on Make, which is not hermetic, but works.

If all else fails, you can, in fact, build the binaries externally, and simply bring them into a bazel build if you must use bazel.

I was hoping to build with Bazel so that I can also use protobufs defined in the repo. But it'd be more troublesome if the code lives outside Bazel...

Regards,

HaiXin

[HaiXin Tie]

Thanks for the tip, Filip. I will try diving deeper into rules_go to identify the problems and improve my Bazel DIY skills when I have more time to spare on this... I've already spent quite a number of hours stuck on this...

Regards,

HaiXin

[HaiXin Tie]

Yes, I saw your code, hence that question :)

I guess until some Bazel Go team experts can come out to help us, there won't be clear fixes to the problems discovered so far.

Anyway, Thanks for your extended assistance, Filip!

Regards,

HaiXin

[Filip Filmar]

Heh, I'm trying to make a cgo binding for a different library today, and am seeing the same issues (cdeps settings not propagated to the cmdline)...

F

[Filip Filmar]

What I found seems to be a bug indeed. I'm not quite sure how it's supposed to be fixed, however.

See: https://github.com/bazelbuild/rules_go/pull/4134

Your code fix:  https://github.com/thx123/bazel-cgo/pull/1/

With that fix applied to `rules_go`, (and with prior changes), your binaries build and run as one would expect.

See: https://github.com/bazelbuild/rules_go/pull/4134

There are other seemingly illogical things going on about propagating CcInfo in rules_go, but that's a story for another time.

F

[Filip Filmar]

Video proof: https://photos.app.goo.gl/FeVhkqHNnngMQTv77

[HaiXin Tie]

This is awesome. Thanks a lot, Filip! For now I can apply a local patch in Bzlmod to rules_go (like this one) to make my project build.

Regards,

HaiXin

[Filip Filmar]

Yes, but note that what I proposed there is strictly speaking incorrect.  I wasn't looking for a fix, but a change that makes it apparent the current behavior is buggy.

My change tries to link everything with `libstdc++`, and there is a note in `link.bzl` just above my change that talks about *not* doing that unless you actually link C++ code.

F

[HaiXin Tie]

Ack'ed. Hopefully the rules_go team can come up with an official PR to fix this issue cleanly.

Regards,

HaiXin

[Filip Filmar]

Maybe not?

I'm amazed a bit that assuming the link flags construction has a bug we're the only two people having the problem... (I tried to compile libfst in a similar way and got hit with a similar bug)

F