Problem using a binary built from an external (bazel) dependency

[P. Oscar Boykin]

I'm working on getting a usable scala skylark rule. My current state of affairs is here:

https://github.com/twitter/scalding/blob/bazel-build/bazel_tools/scala.bzl

My problem is that I want to use to make interface jars. To be hermetic, I want to just depend on bazel's ijar. I do that with a WORKSPACE definition to the "0.1.1" tag here:

https://github.com/twitter/scalding/blob/bazel-build/WORKSPACE#L2

Then in the build rule, I try to access third_party/ijar:ijar

https://github.com/twitter/scalding/blob/bazel-build/bazel_tools/scala.bzl#L166

I tried a few things, but the ijar binary never gets built and never shows up here:

ls `bazel info execution_root`/bazel-out/local_darwin-fastbuild/bin

I do have it in the `baze info execution_root`/_bin/ijar but that seems to be something built-in, and not my dependency (which is actually fine, if it is in the contract of bazel that it exist).

The error I get is:

 external/bazel/third_party/ijar/ijar: No such file or directory

I tried using both .path and .short_path, neither seemed to work:

https://github.com/twitter/scalding/blob/bazel-build/bazel_tools/scala.bzl#L39

I can hack it currently by using "_bin/ijar", but it would be nice not depend on details of the version of bazel installed.

[Han-Wen Nienhuys]

in the latest version of Bazel, you should be able to reference
@bazel_tools://tools/jdk:ijar (the file should be under
bazel-bazel/external/bazel_tools/tools/jdk/ijar)

> --
> You received this message because you are subscribed to the Google Groups
> "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bazel-discus...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bazel-discuss/871a20ea-53fe-4af0-a606-97a16608e71a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Han-Wen Nienhuys
Google Munich
han...@google.com

[P. Oscar Boykin]

Thanks. Using bazel release 0.1.2rc2-jdk7 using @bazel_tools://tools/jdk:ijar works (I thought I had tried that on 0.1.1 and it didn't, but perhaps I made a mistake).

https://github.com/twitter/scalding/commit/659d68dc30861b865cb242ef4edeb288d4af0f37

1) Why doesn't it work to depend on a ijar from a git_repository in WORKSPACE? Is that the expected behavior?

2) Is @bazel_tools documented somewhere? Are there other built-in external repositories? Is it safe to depend on its contents (i.e. is this part of the bazel API?).

I've filed an issue here:

https://github.com/bazelbuild/bazel/issues/647

Thanks for your help.

--

P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin

[Han-Wen Nienhuys]

On Tue, Nov 24, 2015 at 4:21 PM, P. Oscar Boykin <oscar....@gmail.com> wrote:
> Thanks. Using bazel release 0.1.2rc2-jdk7 using
> @bazel_tools://tools/jdk:ijar works (I thought I had tried that on 0.1.1 and
> it didn't, but perhaps I made a mistake).
>
> https://github.com/twitter/scalding/commit/659d68dc30861b865cb242ef4edeb288d4af0f37
>
> 1) Why doesn't it work to depend on a ijar from a git_repository in
> WORKSPACE? Is that the expected behavior?
>
> 2) Is @bazel_tools documented somewhere? Are there other built-in external
> repositories? Is it safe to depend on its contents (i.e. is this part of the
> bazel API?).

Adding lukacs for @bazel_tools questions.

[Kristina Chodorow]

@local-jdk is also a built-in external repository (for accessing the locally-installed JDK). We should probably add documentation to http://bazel.io/docs/external.html on where the external repositories end up, @local-jdk and @bazel_tools, and maybe even some info on package paths.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFQ2z_PipCggmVD4TKCXVpcX35GpkgyYL0qF78GcZ8zY-4w%2BAA%40mail.gmail.com.

[Lukács T. Berki]

TL;DR for Laurent: how to add an executable and its runfiles to the inputs of a SpawnAction created from Skylark? Also, is there a way to create a ":ijar" implicit attribute on Skylark rules that would obey --ijar_top?

On Tue, Nov 24, 2015 at 4:21 PM, P. Oscar Boykin <oscar....@gmail.com> wrote:

Thanks. Using bazel release 0.1.2rc2-jdk7 using @bazel_tools://tools/jdk:ijar works (I thought I had tried that on 0.1.1 and it didn't, but perhaps I made a mistake).

https://github.com/twitter/scalding/commit/659d68dc30861b865cb242ef4edeb288d4af0f37

1) Why doesn't it work to depend on a ijar from a git_repository in WORKSPACE? Is that the expected behavior?

Kind of. The reason why it doesn't work is that you don't add ijar to the set of inputs of the action here . Regrettably, my Skylark is not very sharp, so I have to enlist Laurent's help to tell you how to do it properly. My best guess is using ctx.resolve_command, but that's been marked experimental, and this is something that's a quite trivial use so there must be a solution I don't know of.

2) Is @bazel_tools documented somewhere? Are there other built-in external repositories? Is it safe to depend on its contents (i.e. is this part of the bazel API?).

bazel_tools is here to stay. The set of tools in it may change, though, but that's not a problem with ijar. If you depend on ijar as @bazel_tools//tools/jdk:ijar (note the "tools" path segment there -- it seems to be missing from your .bzl file), you won't be affected by --ijar_top, though. Again, Laurent will tell if there is a way to do it better.

Also, thanks for pointing out that we add a lot of files into the _bin directory. It's not intended, so I'll eventually send out a change to remove it.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAE6AC96dGuZ1Pk93Tye25LC3vMkxHxS_6KBSzv8aGGsH3cGYqQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Maximillianstr. 11-15 | 80539 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[Han-Wen Nienhuys]

Does this need runfiles? For normal spawns, you can just add the
executable to inputs, see eg.

https://github.com/bazelbuild/bazel/blob/master/tools/build_rules/go/def.bzl#L221

(ctx.file.go_tool is part of ctx.files.toolchain).

[P. Oscar Boykin]

Thanks for the help.

It works if I replace the ijar with:

-      "_ijar": attr.label(executable=True, default=Label("@bazel_tools//tools/jdk:ijar"), single_file=True, allow_files=True),

+      "_ijar": attr.label(executable=True, default=Label("@bazel//third_party/ijar:ijar"), single_file=True, allow_files=True),

and add the ijar to the inputs of the action:

-      inputs=list(jars) + ctx.files.srcs + [ctx.outputs.manifest],

+      inputs=list(jars) + ctx.files.srcs + [ctx.outputs.manifest] + [ctx.file._ijar],

with

git_repository(

  name = "bazel",

  remote = "https://github.com/bazelbuild/bazel.git",

  tag = "0.1.1")

in my WORKSPACE.

That said, it sounds like there is no real reason to do this since ijar will be a part of bazel_tools for the foreseeable future.

Thanks again for the help.

[Lukács T. Berki]

Well, it was still a bug :) Glad that I could be of help! :)

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/ff80f118-db8b-41b9-b840-2cb7de6cf505%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Lukács T. Berki]

On Wed, Nov 25, 2015 at 10:29 AM, Lukács T. Berki <lbe...@google.com> wrote:

TL;DR for Laurent: how to add an executable and its runfiles to the inputs of a SpawnAction created from Skylark? Also, is there a way to create a ":ijar" implicit attribute on Skylark rules that would obey --ijar_top?

Note to self: there is a way.  We can define late bound attributes in Skylark like this:

def libc_from_config(attr_map, cfg):

  return cfg.cpp.libc

rule("edward", implementation=edward_impl, 

  attr = { "libc": attr("LABEL", default=libc_from_config) }

So making ijar_top accessible is only a few @SkylarkModule / @SkylarkCallable annotations away. Probably.

 

[Damien Martin-guillerez]

Sorry jumping into the conversation but accessing ijar_top is easy the labels //tools/defaults:ijar always point to it...

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOu%2B0LVo1GQFqFP_x-m5G7ZWJsstRBTwj--BFfeEGubK48SzNQ%40mail.gmail.com.

[Dhanalakshmi Durairaj]

Hi, 

Greetings to all,

I am working on custom GHS toolchain with custom rules.  Custom toolchain is also external dependency. We are referring custom toolchain tools from external directory. I registered toolchain in workspace.

In my custom rule,  I gave the below code:

cxx_binary = rule(

    _cxx_binary_impl,

    attrs = {

        "srcs": attr.label_list(

            allow_files = [".c", ".h"],

            doc = "Source files to compile for this binary",

        ),

        "deps": attr.label_list(

            providers = [CxxInfo],

        ),

        "_cc_toolchain": attr.label(default = Label("@bazel_tools//tools/cpp:current_cc_toolchain")),

    },

    doc = "Builds an executable program from C++ source code",

    executable = True,

    #    toolchains = ["//toolchain:qnx_suite"],

    toolchains = use_cpp_toolchain(),

)

def _cxx_compile(ctx, src, hdrs, out):

    cc_toolchain = find_cpp_toolchain(ctx)

    args = ctx.actions.args()

    args.add("-o", out)

    args.add("-c")

    args.add("-iquote", ".")

    args.add(src)

    ctx.actions.run(

        executable = cc_toolchain.compiler_executable,

        outputs = [out],

        inputs = [src] + hdrs,

        arguments = [args],

        mnemonic = "CxxCompile",

        use_default_shell_env = True,

    )

It works as expected in local. In sandbox execution, I get the below error:

external/ghs_compiler/toolchain/wrappers/ccarm -o bazel-out/k8-fastbuild/bin/main/main.c.o -c -iquote . main/main.c)
# Configuration: d20a3da66be757a87363ceeb80fc232cc65357a3a8fbf944e9ce35007bcdac55
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:518: "execvp(external/ghs_compiler/toolchain/wrappers/ccarm, 0x23c0380)": No such file or directory

Pls help me to fix this issue. 

[Filip Filmar]

At minimum, the target corresponding toyour `ccarm` binary wrapper needs to be mentioned in the `tools` parameter of `ctx.action.run`.

Without that, it will not be present in the sandbox where your `run` is executing, leading to "file not found".

F

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c0621ca6-83b8-42c3-976b-76c638c90ef0n%40googlegroups.com.

[Filip Filmar]

On Tue, Mar 21, 2023 at 3:48 AM Dhanalakshmi Durairaj <dhanad...@gmail.com> wrote:

It works as expected in local. In sandbox execution, I get the below error:

Ah, I missed this on the first pass.  Also check your `ccarm` file. If it is a symlink (as it might be since you need a wrapper to call a binary from external repo), make sure it is a symlink using a relative path, not an absolute path.

If the symlink is using an absolute path, it will refer to a file that is not accessible from inside the sandbox.

F
 

[Dhanalakshmi Durairaj]

Hi,

Thanks a lot. I added tools parameter in ctx.action.run as below:

ctx.actions.run(

        executable = cc_toolchain.compiler_executable,

        tools = cc_toolchain.all_files,

        outputs = [out],

        inputs = [src] + hdrs,

        arguments = [args],

        mnemonic = "CxxCompile",

        use_default_shell_env = True,

    )

Now sandbox execution is working as expected. 

external/ghs_compiler/toolchain/wrappers/ccarm bazel-out/k8-fastbuild/bin/main/main.c.o -o bazel-out/k8-fastbuild/bin/main/hello-world)
# Configuration: c11ebfa482b4c3d0d30fc5caaae6b9e9ef23ddddb61e052d62b04a16b6120789

# Execution platform: @local_config_platform//:host

Target //main:hello-world up-to-date:
  bazel-bin/main/hello-world
INFO: Elapsed time: 17.869s, Critical Path: 4.08s
INFO: 5 processes: 3 internal, 2 linux-sandbox.
INFO: Build completed successfully, 5 total actions

[Dhanalakshmi Durairaj]

Hi 

Greetings to all,

I am working on GHS toolchain.  Ccarm compiler generates “per object debug info file” by providing the flag -G and the extension of the file is .dbo. Debug info file for the archive is .dba. GCC toolchain generates per object debug info file with the extension of .dwo and .dwp (for archive) by providing the flag -gsplit-dwarf. Moreover, I am trying to produce .map file also. How to get these outputs (.dbo,.dba and .map)? Please guide me, is there any way to achieve this without using custom rules?