How to use gen-file outs from one rule as srcs in another genrule

[kgre...@gmail.com]

I'm implementing a build file for an external project that uses cmake.

This project has a bunch of proto files. Their CMakeLists.txt file runs protoc to create the pb.* srcs, and then it creates an umbrella header file called "msgs.hh" which simply has an include line for every proto header that was generated.

I'm trying to generate that header similarly in Bazel and for some reason I'm struggling.

I could simply hard-code all the files, but that wouldn't be as robust, and there are hundreds of proto files in this particular project, so I'd rather it be automated.

I'm trying to do essentially this in my BUILD file:

------------------
proto_library(
name = "my_proto",
srcs = ["my.proto"],
)

cc_proto_library(
name = "my_cc_proto",
deps = [":my_proto"],
)

genrule(
name = "msgs_hh",
srcs = [":my_cc_proto"],
outs = ["msgs.hh"],
cmd = """
echo '// Automatically generated' >> $@
for src in "$(SRCS)"
do
if [[ $${src} == *".h" ]]; then
echo '#include <ignition/msgs/'"$$(basename $$src)"'>' >> $@
fi
done
""",
visibility = ["//visibility:private"],
)
--------------------

Is it not correct so use ":my_cc_proto" as srcs? If not, then is there a better way I can achieve what I'm trying to do?

Any help appreciated!

Thanks
Kevin

[Kevin Greene]

To add a bit more context, when I try to build the msgs_hh target, it doesn't appear to be compiling the protos. Also, I added print statements in the cmd, and $(SRCS) is an empty list.

--

You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/lWpZKTE4i1s/unsubscribe.

To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/8b270132-b273-4181-8801-8b42fa270fc4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kevin Greene]

I should also add that I'm using the latest version of bazel from HEAD.

[Brian Silverman]

First, some explanation of what's happening: ":my_cc_proto" is the cc_library rule, which looks like it's not giving you any srcs for the genrule. Things like a genrule's srcs basically take the files transitive info provider of any inputs and use that. Because nothing in the genrule's inputs require the proto files to be read/compiled, Bazel isn't going to bother doing that when you ask it to build the genrule.

You're probably going to need to write a rule to get the information you want, instead of using a genrule. I suspect that doing the string manipulation in Skylark and using ctx.file_action is going to be easier to work with than the shell script too, as an added bonus. The cookbook has some good examples to get started if you're unsure, like Rule with attributes.

I'm not 100% sure which cc_proto_library/proto_library rules you're using, but assuming they're the ones from Bazel itself it looks like .transitive_sources will get you the information you're looking for. If not, I would recommend digging around in ctx.attr.my_proto_srcs[n] (where my_proto_srcs is an attr.label_list in your custom rule) using print(str(dir(ctx.attr.my_proto_srcs[0]))) etc and look at the contents helpful-sounding names for the proto files. That and `git grep` in the Bazel source tree tend to be helpful for figuring out what's where. NB: I've only done this with legacy providers and haven't seen how it interacts with modern providers, so YMMV.

To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/8b270132-b273-4181-8801-8b42fa270fc4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a58ab959-7952-0b57-f699-3caeafede57a%40mixmax.com.

[Kevin Greene]

Thank you, thank you! That was immensely helpful. I wrote up a little bazel skylark rule based on your feedback. Here is what I came up with: def _glob_cc_proto_hdr_impl(ctx):     output_hdr = ctx.outputs.out     output_content = "\n".join(ctx.attr.hdr_lines) + "\n"     for cc_proto in ctx.attr.cc_protos:         path_prefix = ctx.genfiles_dir.path + "/" + cc_proto.label.workspace_root + "/"         src_hdrs = sorted([hdr.path.replace(path_prefix, "")                            for hdr in cc_proto.cc.transitive_hdrs                            if hdr.path.startswith(path_prefix)])         output_content += "\n".join(["#include <" + hdr + ">" for hdr in src_hdrs]) + "\n"     ctx.file_action(output=output_hdr, content=output_content) glob_cc_proto_hdr = rule(     implementation=_glob_cc_proto_hdr_impl,     attrs={         "hdr_lines": attr.string_list(default=[]),         "cc_protos": attr.label_list(mandatory=True),         "out": attr.output(mandatory=True),     },     executable=False, ) ​I have 2 remaining questions: Is there a more robust way for me to get the path to the generated sources, rather than the hard-coded assumptions I'm making about cc_proto_library's output directory? Right now this creates the generated header in the bazel-out directory. How can I make this create the output file in bazel-genfiles instead? That seems like a more appropriate place for it to go. Should I be using ctx.new_file()? Thanks again for your help! Kevin

[Kevin Greene]

Correction: It creates the generated header in the bazel-bin directory (not bazel-out).

On Fri, May 12, 2017 11:15 PM, Brian Silverman bsilve...@gmail.com wrote:

[Kevin Greene]

Ah blurgh, and I just realized you can't use a *.bzl file from a BUILD file if that BUILD file is being used for an external repo. In my case, my build file is being passed to a "new_git_respository" rule. And when I try to build, it gives me this error: ERROR: /home/kevin/Projects/first_commit/WORKSPACE:55:1: error loading package '@ign_msgs_archive//': Extension file not found. Unable to load package for '@ign_msgs_archive//bzl:glob_cc_proto_hdr.bzl': BUILD file not found on package path and referenced by '//external:ign_msgs_factory'. It's searching for glob_cc_proto_hdr.bzl under the @ign_msgs_archive// workspace, which obviously fails.

[Brian Silverman]

#including hdr.short_path with quotes should work (not sure about <> in the general case).

To get your output header in genfiles (which you'll need for #including it to work I think), set output_to_genfiles=True in your rule.

That looks like a fine use for attr.output; the other thing I would consider (depending on your use case) is using '%{name}.h' in outputs instead so the header always matches the rule name. If you use ctx.new_file, things like `bazel build //my/package:my_header.h` and putting 'my_header.h' into the hdrs of a cc_library in the same package won't work because it's not an output from the rule, which does not seem like the right behavior here.

You want @//bzl:glob_cc_proto_hdr.bzl to load it from a BUILD file in another repository. //foo looks for foo in the "default" repository, aka the current one, while @//foo looks in the "main" repository, which is the top-level one where your Sklark file is.

[Kevin Greene]

It works! Thank you again so much for the help. I was really bashing my head against the wall. I used attr.output and it worked fine. I just needed to add output_to_genfiles=True to my rule.