Use protobuf plugins with bazel

[Ramil Shaimardanov]

Hi!

I am trying to start using protobuf with bazel and I have run into a big issue. I cannot seem to find any good guides/examples for how to use uncommon protoc plugins for bazel generation. For example, I would like to use plugins like these:

https://github.com/danielgtaylor/python-betterproto

https://github.com/alpancs/protoc-gen-pubsub-schema

(and some more)

I am not experienced enough with bazel to read rules like go_proto_library so reverse engineer how to do what they do (and the fact that I feel the need to do so, i think, is an issue which greatly hurts bazel adoption. would be nice to have a lot more tutorials on bazel.build. I have heard many a times from people wishing to adopt bazel and being scared off by perceived complexity)

I have also tried using rules-proto-grpc which appears to have support for custom plugins, but I cannot install it, as I keep getting errors, when trying to add it to my workspace.

Can anyone point me to documentation/projects that would help me figure this out?

Thanks!

[Jason Schein]

Hi Rami,

I'm pretty new to doing bazel extensions/customization myself, but I'm currently working on something similar (integrating a proto plugin for rust instead of python). My current state is that I have my plugins working and now I'm trying to write a rule to wrap them so I can share my work as a bazel package. So I'm hopeful I know enough now to at least help you get your proto plugin toolchain setup in bazel. It has been a bit painful getting to this point, so I can commiserate.

Regarding your comment:

I have also tried using rules-proto-grpc which appears to have support for custom plugins, but I cannot install it, as I keep getting errors, when trying to add it to my workspace.

I recommend sticking with this approach, I tried a few others before I stumbled upon this one, and this one was the least painful to setup. I think remember running into some errors importing rules_proto_grpc into my WORKSPACE file, but I can't remember exactly what they were. I do remember they were related to the order of the various things I was importing in my WORKSPACE. This is something I don't understand extremely well, but its currently importing correctly for me so you're welcome to look at my WORKSPACE file and see if I'm importing it any differently than you were.

Here's a list of steps I recommend you try so you can verify each piece works as you go

1. Make sure your plugin works outside bazel
   1. pip install betterproto[compiler]
   2. protoc -I . --betterproto_proto_plugin_out=tmp/ --plugin=protoc-gen-betterproto_proto_plugin=$HOME/Code/python/venvs/default/bin/protoc-gen-python_betterproto
   3. Double check and make sure the output looks like it used your plugin and not the default python proto compiler.
2. Add a binary plugin target to bazel for your plugin
   1. I used a tool which integrates rusts package manager into bazel and automatically created this target for me.
   2. rules_python and pip_parse can make this target for you, see this gist.
   3. NOTE: pip freeze doesn't include the [compiler] part in betterproto[compiler], I'm not sure why but make sure this is in your requirements_lock.txt file.
3. Build the binary for your plugin in bazel, and test it again manually using protoc (or better yet, the protoc in your bazel workspace)
   1. build protoc: bazel build @com_google_protobuf//:protoc
   2. build your plugin: bazel build @pip_deps_betterproto//:rules_python_wheel_entry_point_protoc-gen-python_betterproto
   3. Call protoc with your plugin: ${BZBIN}/external/com_google_protobuf_protoc_linux_x86_64/protoc.exe --plugin=protoc-gen-betterproto_proto_plugin=${BZBIN}/external/pip_deps_betterproto/rules_python_wheel_entry_point_protoc-gen-python_betterproto -I path/to/your/proto/dir/ --betterproto_proto_plugin_out=tmp/ someproto.proto
4. If everything above worked, your proto plugin should work (any issue you have from here just means you need to tinker with the options you pass to your proto_plugin via bazel attributes).
   1. Finish following the guide from rules-proto-grpc.
   2. Try building a target that uses your proto rule.

Here's my code where I managed to get this working for the rust plugin I'm using, in case thats helpful just to compare. Note that I had to set output_directory = True rather than outputing files, since the files my proto library generates aren't based on the names of the input files (and bazel rules can't look at the contents of their input files at the stage in the build process when the rules are expected to declare their expected output files).

My Custom Rule Definitions

Proto plugin targets

-Jason

On Friday, May 26, 2023 at 1:03:24 AM UTC-7 rami...@gmail.com wrote:

Hi!

I am trying to start using protobuf with bazel and I have run into a big issue. I cannot seem to find any good guides/examples for how to use uncommon protoc plugins for bazel generation. For example, I would like to use plugins like these:

https://github.com/danielgtaylor/python-betterproto

https://github.com/alpancs/protoc-gen-pubsub-schema

(and some more)

I am not experienced enough with bazel to read rules like go_proto_library so Reverse Engineer how to do what they do (and the fact that I feel the need to do so, i think, is an issue which greatly hurts bazel adoption. would be nice to have a lot more tutorials on bazel.build. I have heard many a times from people wishing to adopt bazel and being scared off by perceived complexity)

[Ramil Shaymardanov]

I finally figured it out how to do this (sort of). I was able to install rules_proto_grpc.

My issue was with using packaged older version of rules_go and g...@1.20. I have used dependency override (described in rules_proto_grpc).

With that extension it's very simple, basically creating a BUILD + .bzl files, where you pass through all the parameters to the plugin binary.

Quick note: in order to install a golang binary in bazel you need to use go_repository. You can add it manually to WORKSPACE, or, better yet, use gazelle update-repos

gazelle(
    name = "gazelle-install",
    args = [
        "<package>@<version>",
    ],
    command = "update-repos",
)

You can then do:

bazel query "@<package_repo_name>//:*"

to find all generated targets and choose the one you need and use it as binary in proto_plugin(..., tool = "@<package_repo_name>//:<binary target>", ...)

Hopefully my findings prove useful to someone. If you've got questions, I'd be happy to share more of my experience