Consuming a generated file?

[irs...@gmail.com]

Hi,

I have a Go file that is used as a `src` for many Go targets across multiple packages. It's 95% identical across all packages, except a single line. However, replacing this line is a bit complex, as it require a logic that reads information from other files in the package. I noticed `repository_ctx` has template option, but after some reading, It look like it's possible to create a logic inside Bazel that inspects other workspace files.

So if that's not possible, what's the proper way to handle it? writing an external Python/Go script and use `gen_rule` to call it? can other Targets accept gen_rule as a `src`?  If there's an example in the wild it will be useful.

Thanks!

[Paul Johnston]

The go_library.srcs should accept any *.go file.  A genrule is certainly one way to generate a derived go file from some templated source.

A more obtuse example is how the go_embed_data rule works, by encoding data into a generated go source file: https://github.com/bazelbuild/rules_go/blob/master/extras/embed_data.bzl

[irs...@gmail.com]

Hello Paul,

Thank you for the answer.

I kinda like how it was handled in go_embed_data. It's pretty elegant. Setting an attribute directly run a Go code rather than calling a command line directly. I'm assuming I can probably do the same with `py_library`, but Go seems very popular among Bazel rule maintainers, perhaps I should instead teach myself some Go and reference it like in the example.

Thanks for the shove in the right direction!

[irs...@gmail.com]

Had a time to play with genrule, and there's something I'm not sure about. I wrote a small Bash script that expects a folder (where he scans for specific files), and output a generate file that is being saved on the same folder where the script running from. It looks like so:

sh_binary(
    name = "generate_file",
    srcs = ["//utils:generate_file.sh"],
)

genrule(
    name = "test",
    tools = [
        ":generate_file",
    ],
    outs = [
        "generate_file.txt",
    ],
    cmd = "$(location :tool) `pwd`",
)

This however will not work, because the file will get created in the root of the sandbox, rather than in bazel-out, and bazel will complain the out file is not being saved. I was checking examples in the wild, and almost all expeted the shell script to print the generated file into stdout like so:

Another thing I can probably do is to modify the shell script to accept a path to save the generated file at:

    cmd = "$(location :tool) `pwd` $@",


But is that the correct way of doing it? or there's a better way I'm missing here?


Thanks!

[Austin Schuh]

On Mon, Dec 23, 2019 at 1:10 AM <irs...@gmail.com> wrote:
>
> Had a time to play with genrule, and there's something I'm not sure about. I wrote a small Bash script that expects a folder (where he scans for specific files), and output a generate file that is being saved on the same folder where the script running from. It looks like so:
>
> sh_binary(
> name = "generate_file",
> srcs = ["//utils:generate_file.sh"],
> )
>
> genrule(
> name = "test",
> tools = [
> ":generate_file",
> ],
> outs = [
> "generate_file.txt",
> ],
> cmd = "$(location :tool) `pwd`",
> )
>
> This however will not work, because the file will get created in the root of the sandbox, rather than in bazel-out, and bazel will complain the out file is not being saved. I was checking examples in the wild, and almost all expeted the shell script to print the generated file into stdout like so:
>
>
> Another thing I can probably do is to modify the shell script to accept a path to save the generated file at:
>
> cmd = "$(location :tool) `pwd` $@",
>
> But is that the correct way of doing it? or there's a better way I'm missing here?
> Thanks!

You are very close. I would suggest ditching `pwd`.

This moves most/all of the path logic back into bazel where it is
easier to deal with.

[irs...@gmail.com]

Hello Austin :-)

I just understand something. `pwd` will point to the root of the project, which is not something I want. Here's how the structure looks like:

- Root_Folder
  - service
    - api
       - BUILD
       - file_a
       - file_b
  - utils
    - BUILD
    - generate_file.sh

The genrule is inside service/api BUILD file. What I want is to run the utils/generate_file.sh script file. The script needs a folder to scan, in this case service/api. and it will scan file_a and file_b and generated a file out of it. When I use genrule, it looks like the context directory is the root folder, so I need to state service/api as parameter to the script like so:

 cmd = "$(location :tool) service/api"

I don't want to explicitly state service/api, because this will change. I technically want to say - pass the folder where the BUILD file is at. Running the above will not work, because Bazel will complain it expected to find the output file inside bazel-out, and the script will save the file into the root_diretory - where the content is running from. So the other option I see is either make sure the script print to stdout the generated file and direct it into $@, or copy the output file into $@ with `cp`. 

Thanks!

[irs...@gmail.com]

OK, seems like I missed $(RULEDIR) :-)

So the three options are:

1. Make sure the script that generate the file prints the generate file into STDOUT:

cmd = "$(location :tool) $(RULEDIR) > $@"

2. Let the script generate the file in the root folder, and use mv to move it to the right folder Bazel excepts to find it in:

cmd = "$(location :tool) service/api && mv generate_file.txt  $@"

3. Modifying the script to get a parameter that tells it where to save the generated file:

cmd = "$(location :tool) service/api  $@"

I guess any can work really :-)

[irs...@gmail.com]

I figured the hard way genrule doesn't run on the source-tree. So $(RULEDIR) didn't work as expected. So I modified the script to accept a list of files, and an output directory. Sharing the solution in case someone will find it useful in the future:

filegroup(

    name = "txt-files",

    srcs = glob(["**/*.txt"]),

    visibility = ["//visibility:public"],

)

def generate_file(name, srcs, **kwargs):

    native.genrule(

        name = name,

        srcs = srcs,

        tools = [

            "//utils/generate_file.sh",

        ],

        outs = ["generated_file"],

        cmd = "$(location //utils/generate_file.sh) $(SRCS) $@",

        **kwargs

    )

The only thing that bothered me is that I wanted to avoid filegroup as stating explicitly all files. Instead I want to supply a directory - but it looks like Bazel prefer to work with files as you can't set folder as target. Maybe there's a better way to do it with a custom rule - as you probably not bound to work with targets like you are with genrule - but that's OK for right now.

[Austin Schuh]

Looks like you mostly figured it out. Bazel gives you more knobs if
you change from a genrule to a full rule, at the cost of having to
understand more about how bazel works.
https://docs.bazel.build/versions/master/skylark/lib/native.html#package_name
might be helpful in automating the "service/api" part of the command.

Bazel has directory support, but it has more quirks. I would
personally only use it as a last resort.

Austin

> --
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f40c1eaf-cc6b-4fd4-88eb-315b90d48a92%40googlegroups.com.

[irs...@gmail.com]

Yea, I think as for now I will be sticking with the example above. How Bazel handle directories out of curiosity? I couldn't find an official documentation about it.

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.