Fun with filegroups. Combining multiple source directories and genrule output.

[Garrett Kajmowicz]

TL;DR:

Is there a way to create a filegroup which will contain, in the same directory, files from multiple source directories plus the output of a genrule()?

I have a project I'm slowly working to convert to Bazel. I think the last major remaining issues can be addressed if I can get filegroups (and some trivial genrules) to do what I want. But I'm not able to easily figure out how to get filegroups to do what I want/need.

The main challenge I have is in feeding a bunch of files to the Linux kernel kbuild system for building out-of-tree kernel modules.

I've managed to make most of this work by wrapping the Kbuild call into a foreign_cc make() wrapper call. With enough environment variable substitution I'm able to at least convert some .c files into .o files.

The challenge now is that the make() wrapper only accepts a single filegroup and that Kbuild requires all files to be in the same directory.

I have a directory structure that looks like:

common/file1.c

common/file2.c

kernel/file3.c

kernel/file4.c

kernel/file_requiring_preprocessing.in

kernel/Kbuild

In the existing make-based buildsystem, the file_requiring_preprocessing.in is handled by a few sed statements. That output, plus the files from the common/ and kernel/ directories are symlinked into a separate build directory. Inside the build directory the kernel kbuild system is triggered and the kernel module is built.

I can obviously replace the existing sed statements via genrule(). And calling the kbuild can be handled via the make() wrapper. I'm left with putting everything in the same directory.

Is there a way to create a filegroup which will contain, in the same directory, files from multiple source directories plus the output of a genrule()?

Thank you!

[Philipp Schrader]

I think you could do this via a genrule:

genrule(

    name = "foo",

    srcs = [

        "//common:files"

        "//kernel:files",

        "//path/to:genrule",

    ],

    outs = [

        "build/file1.c",

        "build/file2.c",

        "build/file3.c",

        "build/file4.c",

        "build/Kbuild",

    ],

    cmd = "mkdir build && cp $(SRCS) build/",

)

Then you can pass "foo" to your make() function.

I think the bit to compute the "outs" list might be a bit annoying, but maybe it won't be so bad with a "glob()" and a helper macro that performs that path manipulation.

skylib has some nice functions to manipulate paths in Starlark.

Hope I didn't misunderstand what you're looking for.

Phil   

[Garrett Kajmowicz]

That mostly works. Thank you!

The one issue I'm now facing is that the kernel module is somehow being built in the same directory as the output of the genrule() and not in the temporary directory which the make() rule would expect. 

That is, I end up with the following error lines:

ERROR: /var/tmp/gk/repo_bazel/usfp-firewall/kernel/usfp_rh/BUILD:83:5: output 'usfp-firewall/kernel/usfp_rh/usfp_kernel/build/usfp_rh.ko' was not created

ERROR: /var/tmp/gk/repo_bazel/usfp-firewall/kernel/usfp_rh/BUILD:83:5: CcGnumakeMakeRule usfp-firewall/kernel/usfp_rh/usfp_kernel/include failed: not all outputs were created or valid

The .ko is supposed to be generated by the make() rule named usfp_kernel. Instead, I end up with the kernel module present from the root of the sandbox:
bazel-out/k8-fastbuild/bin/usfp-firewall/kernel/usfp_rh/build/usfp_rh.ko

So it is present in the /build/ path, not the /usfp_kernel/ path.

Is there any guidance on how to make bazel happy with this?

Thank you again!

-     Garrett

[Garrett Kajmowicz]

I changed the output of the copying genrule to stuff the files in the directory which should contain the output of the Kbuild process so that .ko file gets produced in the correct directory. It's not elegant, but it works.