Codegen produces unpredictable files. How to compile those?

[Konstantin]

I know that Bazel "does not like it" when some action produces the outputs (files) which names are not known at analysis phase. So far I was lucky to avoid such cases using different tricks, but unfortunately now I hit exactly this problem.

Some custom code generator produces C++ source files - .cpp and .h The number of produced files varies and their names are generated programmatically by the codegen and cannot be predicted by the Bazel rule which invokes the codegen. 

From what I could find so far it seems that the rule should use ctx.actions.declare_directory and then make the codegen to generate the sources to that directory. Then the rule can return it as the output. 

Generated files are C++ sources and the next build step supposed to compile them, for instance build a static library from them. That is where I hit the wall. Not a slightest idea how to achieve that. Help?

Thank you!

Konstantin

[Herrmann, Andreas]

Hi Konstantin,

Indeed, Bazel does not support dynamic dependencies. See e.g. the following thread for a related discussion:

https://groups.google.com/g/bazel-discuss/c/-MIUzCGQWJg/m/JVbthNW_BAAJ

A couple ideas how you could work around this in your case:

1. It sounds like you control the code generator. Would it be feasible to change it to produce easily predictable outputs? E.g. just a single .cpp and .h file.

2. You could use BUILD file generation, a la gazelle, to generate targets with predeclared outputs based on the sources that the code generator reads.

    This could potentially be done in repository rules, so that Bazel reruns the code generation automatically when the inputs change.

3. As you say you could use directory outputs and write a custom rule that determines the compiler/linker and compiler/linker flags from the cc toolchain and then runs an action that compiles the .cpp files into a static archive in one go. This way you don't have to track individual .cpp files in Starlark.

    The header files are a bit trickier. One option would be to merge the generated headers into one single header at a predictable output path.

    Examples for getting the compiler and flags: https://github.com/tweag/rules_haskell/blob/6e05343b196c396e55352b97c83a121c2d2f85ff/haskell/cc.bzl#L91 and https://github.com/tweag/rules_haskell/blob/6e05343b196c396e55352b97c83a121c2d2f85ff/haskell/private/cc_wrapper.bzl#L13

Best, Andreas

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/3a22ec7b-c0b5-48a8-89e6-6a64262588b9n%40googlegroups.com.

[Konstantin]

Andreas, thanks a lot for your response! I got a couple of great ideas from it.

I don't really have much control of the codegen - it's an old and hairy contraption and I don't want to mess with it and get on the hook to support it.

BUILD file generation looks like a winning solution to me. Instead of running codegen with the regular rule I can (probably) run it with repository rule and then let that rule generate the BUILD file accordingly to the sources actually being generated. Sounds easy enough and what is even more important - conceptually right.

Idea #2 is obviously much more involved. We already switched from the built-in C++ rules (cc_library, cc_binary) to our own custom implementations which are functionally equivalent but account for a lot of small quirks where built-in rules are not flexible enough to accommodate for our (not so unusual) requirements. So extending those rules to accept a directory of files is definitely possible although looks like much more work than the first solution.

I should be able to handle it from here. Thanks again!