Calling other executable targets from my

[Oleg Tsarev]

Hello,

I am using "gvt" tool for manage vendor dependencies.

I wrote skylark extension for that, and use it like:

bazel run :gvt -- list

bazel run :gvt -- fetch some/external/repo

Now I want to execute gazelle just after execute gvt, and I can not understand how to rightly call another executable target from my own

https://gist.github.com/excavador/b3434f5ba82e1050f6a22ca525271487

Could you please advise me, now to correctly run some another executable target from my rule?

Best regards, Oleg

[László Csomor]

Hi Oleg,

Looks like you don't need to implement a custom rule here and could use a genrule to run all your commands:

genrule(

  name = "gvt",

  outs = ["gvt-output.txt"],

  # You must list all tools here that you wish to use in the `cmd`.

  tools = [

    "//:gazelle",

    "//vendor/github.com/FiloSottile/gvt:gvt"

  ],

  # You must use the "$(location //label/of:tool)" syntax to safely retrieve the runtime

  # path of the tools. You can only use $(location) for labels that you declared in

  # either the `srcs`, `tools`, or `outs` attribute.

  cmd = " ".join([

    "( $(location //vendor/github.com/FiloSottile/gvt:gvt) list",

    "    && $(location //vendor/github.com/FiloSottile/gvt:gvt) fetch some/external/repo",

    "    && $(location //:gazelle) --arg --to --gazelle",

    ") > $@",

  ]),

)

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPuPPcsQk9yL1XZ76i8SgWnPxNt1mYogCux%3Dub1q3v_MsmhH2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Oleg Tsarev]

gvt should write data to WORKSPACE directory

ср, 25 окт. 2017 г. в 14:30, László Csomor <laszlo...@google.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[László Csomor]

Why, what kind of data? Build rules typically shouldn't write to the source tree. Note that I'm familiar neither with gvt nor gazelle, so maybe my questions are obvious to someone who is, but not to me.

If you have to update files in the source tree, then Bazel is probably not the right tool. You could write a script that runs Bazel to build the gvt and gazelle binaries, then use them from "$(bazel info bazel-bin)/vendor/github.com/FiloSottile/gvt/gvt" and "$(bazel info bazel-bin)/gazelle". You'd have to run this script from the root directory of your workspace.

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

[Oleg Tsarev]

gvt is tool for manage "vendor/" directory

Command "gvt fetch githib.com/USERNAME/REPONAME" create directory "vendor/githib.com/USERNAME/REPONAME" and also load all dependencies missed in the "vendor"

You are right: I can not make it "build" rule, but my purpose is different - "bazel run :gvt -- fetch githib.com/USERNAME/REPONAME" is command which runned manually and update "vendor/".

It is NOT a build step.

It is reason why I make it as "run" rule. I implemented it exactly as implemented "gazelle" rule from rules_go.

It works perfectly, except the lack of ability to run "gazelle" after fetch data to "vendor/" in WORKSPACE.

[Oleg Tsarev]

Sorry, missed the point about gazelle.

"gazelle" is tool from "rules_go" which generated bazel BUILD files in WORKSPACE.

"bazel run :gazelle" is official way to work with golang from bazel.

Also gazelle used for external golang dependencies - it generated BUILD.bazel files for external repos.

gazelle can work in two modes - "external" (with bazel external repositories) and "vendored" (as I use it).

I use "vendored" because I need autocomplete to dependency in my IDE.

Consider case when I need to vendor some golang dependency.

How I can make it manually?

"gvt fetch github.com/USERNAME/REPONAME" # load data to "vendor" directory

"bazel run :gazelle" # generated BUILD.bazel file for everything, including newly added "vendor/github.com/USERNAME/REPONAME"

So, I do not want install gvt on my machine globally.

For that I wrote the my extersion, and for this task (vendor dependency) run following:

"bazel run :gvt -- fetch github.com/USERNAME/REPONAME"

"bazel run :gazelle"

After that I have added external dependency to my project and can commit these files.

I just want to remove second step "bazel run :gazelle", more precisely to say, I want to move it inside my extension

[László Csomor]

Thanks for your explanation! Let me loop in some folks from the regular rules_go contributors, maybe they have some thoughts too.

Meanwhile, can you explain why you can't run gazelle from the extension? I mean, is it that you don't know how to run it do it, or you do but gazelle reports some error?

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[László Csomor]

Ah, never mind, I see it in your original email:

"and I can not understand how to rightly call another executable target from my own"

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[Oleg Tsarev]

On Wed, Oct 25, 2017 at 3:28 PM, László Csomor <laszlo...@google.com> wrote:

Thanks for your explanation! Let me loop in some folks from the regular rules_go contributors, maybe they have some thoughts too.

Would be great! 

Hi Jay! I am "excavador" from github, we already discuss some issues :)

Meanwhile, can you explain why you can't run gazelle from the extension? I mean, is it that you don't know how to run it do it, or you do but gazelle reports some error?

Two major problems - 1) I do not understand how to extract path to gazelle script correctly 2) I do not understand how to construct correct runfiles for execute gazelle.

But probably it is the wrong way. I have some executable target "//:gazelle" and I want to execute it from my extenstion. I suppose I should NOT inspect the target in any way, I should just simple "call" it.

Imagine I want to call some another executable rule, not a gazelle....

If short, I want to know the "right" way to run any executable rule for instance, from "run_shell" command.

[László Csomor]

> If short, I want to know the "right" way to run any executable rule for instance, from "run_shell" command.

That's probably not what you need: actions.run creates a build action, i.e. you'd have to use "bazel build", not "bazel run".

But anyway, doing that should be easy from actions.run(), here's an example:

- declaring the dependency: https://github.com/bazelbuild/bazel/blob/master/third_party/protobuf/3.4.0/protobuf.bzl#L127

- creating an action with that tool as the executable: https://github.com/bazelbuild/bazel/blob/master/third_party/protobuf/3.4.0/protobuf.bzl#L109

actions.run just works, it sets up the runfiles correctly.

actions.run_shell may be unsuitable, because it's an action whose executable is Bash, and you can't tell it to set up runfiles for some other binary.

I think you could set up a sh_binary analogous to the genrule I showed earlier:

  sh_binary(

    name = "foo",

    srcs = ["foo.sh"],

    data = [

      "//:gazelle",

      "//vendor/github.com/FiloSottile/gvt:gvt"

    ],

  )

Since you have gvt and gazelle as data dependencies, Bazel stages them in the sh_binary's runfiles.

The script would be like:

  #!/bin/bash

  set -eu

  WORKSPACE_DIR=$1

  RUNFILES=${RUNFILES:-$0.runfiles}

  GAZELLE_BINARY="$(realpath "$(find $RUNFILES -name 'gazelle' | head -n1)")"

  GVT_BINARY="$(realpath "$(find $RUNFILES -path 'vendor/github.com/FiloSottile/gvt' -name 'gvt' | head -n1)")"

  cd "$WORKSPACE_DIR"

  $GVT_BINARY fetch github.com/USERNAME/REPONAME

  $GAZELLE_BINARY

And you run it like so:

  $ bazel run "//:foo" -- "$(bazel info workspace)"

For the record, I'm working on a change that'll add a runfiles.sh library to @bazel_tools (maybe in Bazel 0.8.0) that will define a `rlocation()` function that you can use to look up runfiles easily, without having to use the find|head hack. (https://bazel-review.googlesource.com/c/bazel/+/18412)

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[Oleg Tsarev]

Thank you for sharing!

Unfortunatelly, your script will not work.

Run target ":gazelle" execute not a binary, but shell script "gazelle_script.bash" which sensitive to location (resolve pathes in some way)

It is the original problem, why I wrote to mailing list.

Some time later I will create complete example repo with manual for reproduce, probably it would be simpler for discuss & find solution.

 

For the record, I'm working on a change that'll add a runfiles.sh library to @bazel_tools (maybe in Bazel 0.8.0) that will define a `rlocation()` function that you can use to look up runfiles easily, without having to use the find|head hack. (https://bazel-review.googlesource.com/c/bazel/+/18412)

So, would be I able to implement executable rule  "RULE_NAME" in any way, and then use it from another rule as "rlocation('RULE_NAME') <some my args>" ?

[Alex Humesky]

For tools which manage dependencies, it' probably best to create a workspace rule:

https://docs.bazel.build/versions/master/skylark/repository_rules.html

Workspace rules can be used in the WORKSPACE file, download files, and generate BUILD files.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPuPPcsQk9yL1XZ76i8SgWnPxNt1mYogCux%3Dub1q3v_MsmhH2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPuPPcsX1S6gumD_MX2Dw3cs0wqvJXxBO4Jh-hRY9wC_NoGbCw%40mail.gmail.com.

[Oleg Tsarev]

Does not work for vendored dependencies in golang

Please read

https://github.com/bazelbuild/rules_go/blob/master/Vendoring.md

I use vendored dependency for two reasons:

- I have patches to dependencies, too snall for fork and too specific for push to upstream

- I need some way to expose dependencies to IDE (I know about IDEA plugin, but it far from required - first, second - people use other IDEs as well)

ср, 25 окт. 2017 г. в 16:53, Alex Humesky <ahum...@google.com>:

[Jay Conrod]

So I think the fundamental problem you're running into is that "bazel run" executes programs in the execroot, not in the source tree. The gazelle rule cheats a bit here. You can see the implementation in gazelle.bzl. We add //:WORKSPACE as a data dependency to a generated bash script. When the script runs, it reads the symlink to the WORKSPACE file in the source tree, changes to that directory, then executes the Gazelle binary.

I would not recommend this approach. There are several disadvantages.

• It's not guaranteed to work in the future. There's no reason a data dependency has to be a symlink to the source tree.
• You won't know the user's working directory; you can only jump to a specific directory in the source tree.
• You won't have access to stdin. "bazel run" doesn't provide it.
• Bazel will hold a lock while your program is running. No other bazel commands can run concurrently.
• In general, this breaks expectations that things executed with "bazel run" are isolated from your source tree.

Because of these disadvantages, I'm thinking of switching Gazelle to another approach.

• A bazel rule would build a script that has all the Gazelle arguments baked in.
• The script would have to be manually copied to the source tree. This would need to be done once in a new workspace, and then whenever the script changes (not very often). The script would warn when it needs to be updated.
• When the script is run from the source tree, it would "bazel build" and then execute Gazelle with the baked in arguments.

So that might work for you. There's already a general script called bazel-run.sh. This runs "bazel build" on a target, then execs the resulting binary with arguments from the command line. This only works for one target though.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPuPPcsQk9yL1XZ76i8SgWnPxNt1mYogCux%3Dub1q3v_MsmhH2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

[László Csomor]

Alex and Jay, thanks for your detailed answers.

Just one thing I want to mention: it's already the case that on Windows, Bazel doesn't create symlinks, so dereferencing the WORKSPACE symlink only works on Linux/macOS, and it's brittle.

Oleg:

> So, would be I able to implement executable rule  "RULE_NAME" in any way, and then use it from another rule as "rlocation('RULE_NAME') <some my args>" ?

Maybe, although that's not the main use case for rlocation. Rlocation is just a function you can use to look up data dependencies in a platform-independent way. So yes, it can look up path/to/rule_name, but the main binary generated by that rule may be called differently (particularly on Windows), and you would have to tell that binary where it finds its runfiles.

Anyway, an example of `rlocation` is to look up the runfile for which there'd be a symlink under $RUNFILES/io_bazel/foo/bar -- you'd access $(rlocation "io_bazel/foo/bar") instead and that works on Windows and Linux/macOS.

FYI, I'll be on vacation for ~2 weeks starting tomorrow, so I won't be able to follow up on this issue.

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[Oleg Tsarev]

Hi Jay!

On Wed, Oct 25, 2017 at 5:28 PM, Jay Conrod <jayc...@google.com> wrote:

So I think the fundamental problem you're running into is that "bazel run" executes programs in the execroot, not in the source tree. The gazelle rule cheats a bit here. You can see the implementation in gazelle.bzl. We add //:WORKSPACE as a data dependency to a generated bash script. When the script runs, it reads the symlink to the WORKSPACE file in the source tree, changes to that directory, then executes the Gazelle binary.

Exactly! And in my gist-link in the first message I do the same. 

I would not recommend this approach. There are several disadvantages.

• It's not guaranteed to work in the future. There's no reason a data dependency has to be a symlink to the source tree.

Yes, but we have tasks where we need to update source tree.

If you restrict do it from "run" targets, it means, users would be forced to write some wrappers-scripts around bazel... Probably we should consider to allow write rules which modify source tree.

These rules should be banned from build execution, it is just utility actions, which always run manually.

  

• You won't know the user's working directory; you can only jump to a specific directory in the source tree.

Yes, it is exactly what I need. I can run "//:gazelle" or "//:gvt" from any subdirectory in WORKSPACE, it should work from WORKSPACE root. 

• You won't have access to stdin. "bazel run" doesn't provide it.

In case described by me I do NOT need this. I need just run manually some task, that is. 

• Bazel will hold a lock while your program is running. No other bazel commands can run concurrently.

It is manual execution, I don't care :)

 

• In general, this breaks expectations that things executed with "bazel run" are isolated from your source tree.

As I wrote above, consider the rules which work with source tree - sometime, rarely, you need these rules.

gazelle it good example, but I also have the same problems with some internal tools.

If you restrict work with WORKSPACE, it means I should wrap bazel and some my scripts in the root Makefile, and use make instead off bazel-only

 

Because of these disadvantages, I'm thinking of switching Gazelle to another approach.

• A bazel rule would build a script that has all the Gazelle arguments baked in.
• The script would have to be manually copied to the source tree. This would need to be done once in a new workspace, and then whenever the script changes (not very often). The script would warn when it needs to be updated.
• When the script is run from the source tree, it would "bazel build" and then execute Gazelle with the baked in arguments.

Weird! Ok, it means I would be forced to wrap bazel by GNU Make. Very sad.

[Jay Conrod]

> Yes, but we have tasks where we need to update source tree.

> If you restrict do it from "run" targets, it means, users would be forced to write some wrappers-scripts around bazel...

> Probably we should consider to allow write rules which modify source tree.

> These rules should be banned from build execution, it is just utility actions, which always run manually.

Personally, I agree with you here. I'd like to be able to define rules which operate in the workspace and don't hold the Bazel lock. There could be an attribute on binary rules that says whether they execute that way. It makes sense to disqualify these from being run as part of a normal build.

I brought up some of these ideas in another thread, and while there were lots of good ideas, it didn't seem like Bazel itself could support this before 1.0. The required changes are significant.

Until then, it seems like wrapper scripts are the best option.

> If you restrict work with WORKSPACE, it means I should wrap bazel and some my scripts in the root Makefile, and use make instead off bazel-only

You don't necessarily need to use make, but you will probably need some sort of wrapper script that builds some tools with Bazel then executes them with known arguments.

[Oleg Tsarev]

Thank you very much for your answer!

On Wed, Oct 25, 2017 at 7:17 PM, Jay Conrod <jayc...@google.com> wrote:

> Yes, but we have tasks where we need to update source tree.

> If you restrict do it from "run" targets, it means, users would be forced to write some wrappers-scripts around bazel...

> Probably we should consider to allow write rules which modify source tree.

> These rules should be banned from build execution, it is just utility actions, which always run manually.

Personally, I agree with you here. I'd like to be able to define rules which operate in the workspace and don't hold the Bazel lock. There could be an attribute on binary rules that says whether they execute that way. It makes sense to disqualify these from being run as part of a normal build.

I brought up some of these ideas in another thread, and while there were lots of good ideas, it didn't seem like Bazel itself could support this before 1.0. The required changes are significant.

Got it... So, the main problem with idea – re-design (conceptually) run targets. It is complicated task 

Until then, it seems like wrapper scripts are the best option.

Yes. Probably we can meet some problem with sandboxing on linux, but these manual actons run typicaly on developer machine (OSX) where sandbox is not supported, and hack would work fine.

> If you restrict work with WORKSPACE, it means I should wrap bazel and some my scripts in the root Makefile, and use make instead off bazel-only

You don't necessarily need to use make, but you will probably need some sort of wrapper script that builds some tools with Bazel then executes them with known arguments.

Yes. GNU make is usefull utility for wrap :)