Moving or linking (hard/sym) built files into a different directory structure

[Kunal]

Hi all,

Sincere apologies for a long post.

I'm very new to bazel and trying to use it to build a golang based project that builds quite a few golang plugins (linkmode="plugin").

I have managed use gazelle to generate the BUILD.bazel files in the entire directory hierarchy and the builds happen correctly - but, ends up producing the final ".so" files deep within this hierarchy.

The app that loads these plugins expects them to be available in a certain directory structure with each plugin having its path encoded as base64 string (don't ask why - I'm not the owner/designer/developer of this app).

Earlier, these plugins were getting built using a serially running bash script and took a lot of time. Bazel helps us parallelise the builds nicely.

I wrote a custom rule to both generate the base64 encoded filename as well as fire the "ln" command to create the link. For some reason, the "ln" command always throws error with "<source-plugin-file>: No such file or directory".

I also printed the subcommand generated for this and when I try to execute the printed subcommand, the same one succeeds perfectly. So, I'm a bit confused as to what am I doing wrong here.

Here is the output from the build:
➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗ bazelisk build -s //...
DEBUG: /home/kunal/work/hotstar/hotstar-x-stack/build-systems/bazel/hs-core-widget-binder-query/tools/hotstar/bazel/link_artefacts.bzl:23:10: destination path: binders
DEBUG: /home/kunal/work/hotstar/hotstar-x-stack/build-systems/bazel/hs-core-widget-binder-query/tools/hotstar/bazel/link_artefacts.bzl:30:10: Linking file binders/Vm90aW5nV2lkZ2V0SW5zdGFuY2UvVm90aW5nV2lkZ2V0
INFO: Analyzed 1036 targets (311 packages loaded, 1657 targets configured).
INFO: Found 1036 targets...
SUBCOMMAND: # //:link-something [action 'Linking binder VotingWidgetInstance/VotingWidget..', configuration: 2196e5683698aea1297cc7025e556276264b5abd3954e88e8320391192cc438e, execution platform: @local_config_platform//:host]
(cd /home/kunal/.cache/bazel/_bazel_kunal/cd15c8988ce3af495ee0ee5280a12808/execroot/hs-core-widget-binder-query && \
  exec env - \
    PATH=/home/kunal/.cache/bazelisk/downloads/bazelbuild/bazel-6.0.0-linux-x86_64/bin:/home/kunal/.nodenv/shims:/home/kunal/.nodenv/bin:/home/kunal/bin:/usr/local/bin:/home/kunal/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:./node_modules/.bin:/home/kunal/.local/bin:/home/kunal/go/bin:/home/kunal/.please/bin \
  /bin/bash -c 'ln bazel-out/k8-fastbuild/bin/widgets/VotingWidgetInstance/VotingWidget/binder/binder_/binder.so bazel-out/k8-fastbuild/bin/binders/Vm90aW5nV2lkZ2V0SW5zdGFuY2UvVm90aW5nV2lkZ2V0')
# Configuration: 2196e5683698aea1297cc7025e556276264b5abd3954e88e8320391192cc438e
# Execution platform: @local_config_platform//:host
ERROR: /home/kunal/work/hotstar/hotstar-x-stack/build-systems/bazel/hs-core-widget-binder-query/BUILD.bazel:20:10: Linking binder VotingWidgetInstance/VotingWidget.. failed: (Exit 1): bash failed: error executing command (from target //:link-something) /bin/bash -c 'ln bazel-out/k8-fastbuild/bin/widgets/VotingWidgetInstance/VotingWidget/binder/binder_/binder.so bazel-out/k8-fastbuild/bin/binders/Vm90aW5nV2lkZ2V0SW5zdGFuY2UvVm90aW5nV2lkZ2V0'

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
ln: failed to access 'bazel-out/k8-fastbuild/bin/widgets/VotingWidgetInstance/VotingWidget/binder/binder_/binder.so': No such file or directory
INFO: Elapsed time: 145.057s, Critical Path: 7.00s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

You can see that the mentioned file actually exists:
➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗ ls -l "bazel-out/k8-fastbuild/bin/widgets/VotingWidgetInstance/VotingWidget/binder/binder_/binder.so"
-r-xr-xr-x 1 kunal kunal 28M Mar  8 15:53 bazel-out/k8-fastbuild/bin/widgets/VotingWidgetInstance/VotingWidget/binder/binder_/binder.so

Here is the code for the custom rule I wrote:

# some docstring

load("@aspect_bazel_lib//lib:base64.bzl", "base64")

def _link_file_impl(ctx):
    src = ctx.attr.src
    dep = src[DefaultInfo].files.to_list()[0]

    # print("dep = {dep}, rel_path = {rel_path}, path = {path}, dir = {dir}".format(dep=dep,
    #         path=dep.path,
    #         rel_path=dep.short_path,
    #         dir=dep.dirname,
    # ))

    ## Extract sub-path - only the name of the widget/space/page
    pieces = dep.short_path.split("/")[1:-3]

    artifact_rel_path = "/".join(pieces)
    # print("artifact rel path = {artifact_rel_path}".format(artifact_rel_path=artifact_rel_path))

    dest_path = ctx.attr.dest_path
    artifact_type = ctx.attr.artifact_type
    print("destination path: {dest_path}".format(dest_path=dest_path))

    dst_b64 = base64.encode(artifact_rel_path)
    dst = ctx.actions.declare_file("/".join([dest_path, dst_b64]))

    src_path = dep.path

    print("Linking file {dst}".format(dst=dst.short_path))

    _cmd = "ln {src} {target}".format(src=src_path, target=dst.path)
    # print(" --- Executing command: {_cmd}".format(_cmd=_cmd))

    ctx.actions.run_shell(
        mnemonic = "Link{artifact_type}".format(artifact_type=artifact_type.capitalize()),
        progress_message = "Linking {artifact_type} {artifact_rel_path}..".format(artifact_type=artifact_type, artifact_rel_path=artifact_rel_path),
        inputs = [],
        outputs = [dst],
        command = _cmd,
        # command = "ln %s %s".format(src_path, dst.path),
        # arguments = [src_path, dst.path],
        use_default_shell_env = True,
    )

    return [
        DefaultInfo(
            files = depset([dst]),
            runfiles = ctx.runfiles(files=[dst]),
        )
    ]

_link_file = rule(
    implementation = _link_file_impl,
    provides = [DefaultInfo],
    attrs = {
        'src': attr.label(mandatory = True, allow_files = True),
        'dest_path': attr.string(mandatory = True),
        'artifact_type': attr.string(mandatory = True),
    },
)

def link_file(name, src, dest_path, artifact_type, **kwargs):
    _link_file(
        name = name,
        src = src,
        dest_path = dest_path,
        artifact_type = artifact_type,
        **kwargs
    )

Just for completion sake, here is what happens when I run the SUBCOMMAND manually:

➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗ ls -l bazel-out/k8-fastbuild/bin/binders
total 0
➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗
➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗
➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗
➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗ (cd /home/kunal/.cache/bazel/_bazel_kunal/cd15c8988ce3af495ee0ee5280a12808/execroot/hs-core-widget-binder-query && \
  exec env - \
    PATH=/home/kunal/.cache/bazelisk/downloads/bazelbuild/bazel-6.0.0-linux-x86_64/bin:/home/kunal/.nodenv/shims:/home/kunal/.nodenv/bin:/home/kunal/bin:/usr/local/bin:/home/kunal/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:./node_modules/.bin:/home/kunal/.local/bin:/home/kunal/go/bin:/home/kunal/.please/bin \
  /bin/bash -c 'ln bazel-out/k8-fastbuild/bin/widgets/VotingWidgetInstance/VotingWidget/binder/binder_/binder.so bazel-out/k8-fastbuild/bin/binders/Vm90aW5nV2lkZ2V0SW5zdGFuY2UvVm90aW5nV2lkZ2V0')
➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗ ls -l bazel-out/k8-fastbuild/bin/binders
total 28M
-r-xr-xr-x 2 kunal kunal 28M Mar  8 15:53 Vm90aW5nV2lkZ2V0SW5zdGFuY2UvVm90aW5nV2lkZ2V0
➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗

As you can see, the first "ls -l ..." output shows the binders/ as empty directory and the same "ls -l ..." command shows the hard-linked file with base64 encoded filename.

In fact, here you can also see that their inode nos. match - indicating the successful hard-link:

➜  kunal@hydrogen hs-core-widget-binder-query git:(feature/bazel/stage-1) ✗ ls -il "bazel-out/k8-fastbuild/bin/binders/Vm90aW5nV2lkZ2V0SW5zdGFuY2UvVm90aW5nV2lkZ2V0" "bazel-out/k8-fastbuild/bin/widgets/VotingWidgetInstance/VotingWidget/binder/binder_/binder.so"
95842297 -r-xr-xr-x 2 kunal kunal 29139855 Mar  8 15:53 bazel-out/k8-fastbuild/bin/binders/Vm90aW5nV2lkZ2V0SW5zdGFuY2UvVm90aW5nV2lkZ2V0
95842297 -r-xr-xr-x 2 kunal kunal 29139855 Mar  8 15:53 bazel-out/k8-fastbuild/bin/widgets/VotingWidgetInstance/VotingWidget/binder/binder_/binder.so

Please help me understand where am I going wrong.

TIA,

Kunal

[Gregg Reynolds]

Your run_shell action has empty inputs.  I dunno if that will solve your problem, but I do know that bazel will ignore any files not listed as inputs.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0a7d185e-5f82-4417-ab4b-fab4ab58c204n%40googlegroups.com.

[Brian Silverman]

To expand on Gregg's point: Bazel runs actions (including ctx.actions.run_shell) in an environment where only their inputs are accessible. There are various sandboxing strategies that can be enabled to make the inputs even more inaccessible, but even the basic processwrapper-sandbox will make this fail. If you add src to your inputs, it should work.

Also, you should probably just use ctx.actions.symlink instead of run_shell here.

If you see ctx.actions.declare_symlink while you're looking around, that's not what you want here.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAO40Mi%3Dx6N1BCtR6Qz5gziDyExgp3NAJ6j6uA5rOALa1egC--g%40mail.gmail.com.

[Filip Filmar]

On Thu, Mar 9, 2023 at 8:55 PM Brian Silverman <bsilve...@gmail.com> wrote:

To expand on Gregg's point: Bazel runs actions (including ctx.actions.run_shell) in an environment where only their inputs are accessible. There are various sandboxing strategies that can be enabled to make the inputs even more inaccessible, but even the basic processwrapper-sandbox will make this fail. If you add src to your inputs, it should work.

IIUC, the issue in the example above is at least `input = []`. It should probably be at least `input = [src]`. This will have Bazel make the file `src` available to the sandbox that is executing your `run_shell`. Otherwise, despite the fact that your `src` exists, it will *not* be part of the sandbox, and you get the "file not found" behavior.

(Don't ask how I know this. :/ )

F

[Kunal]

Thanks for all the help, Gregg, Brian, Filip - I really appreciate it.

Empty input, indeed, turned out to be the issue.

I got this particular rule to work - but, now I have other problem of actually moving the files into the required build directory structure that needs to reside at the "outputPath" or "$(BINDIR)".

I might need to write a tool to do this linkage/movement that should be called from the top WORKSPACE/BUILD.bazel level - so as to have access to the "outputPath" and/or "$(BINDIR)".

From what I understand, I cannot walk up the directory tree from within inside of a package.
For example, I cannot move/generate a new file from //foo/bar/baz:target into $(BINDIR) directly - it seems to have access only to the subdirs of "foo/bar/baz".

Am I correct here? Or is there a way to walk up to parents of "foo/bar/baz"?

As an experiment, I called this above rule from the top-level BUILD.bazel file with a specific target as the input and that did indeed create the link in the $(BINDIR).

However, I need to somehow generate this list of inputs on-the-fly - so, I was trying to push it down into the generated BUILD.bazel files (generated by gazelle) with the "map-kind" directive.

That's where I got stuck with not having access to the top-level $(BINDIR).

I can think of 2 options that may work:
1. write a tool (maybe simple shell/python script) that does the linkage and is called from the top-level BUILD.bazel

    (this seems to be the approach used in https://github.com/google/bazel_rules_install)

2. somehow, figure out a way to generate the list of inputs to this rule at run-time - maybe, with some provider? or somehow explore how to use the depsets from the DefaultInfo - maybe, transitive depset?

Which way should I explore?

Thanks in advance,

 Kunal

[fabian.bra...@gmail.com]

Hi Kunal, 

to achieve a simple install you can indeed use a custom binary.

To make the linkage "automatic" You can use a sh_binary. I have a fitting example here from some time ago:
https://github.com/FaBrand/bazel-learning/blob/master/sh_binary_args/BUILD#L8-L19

Notice the `data` attribute. This tells bazel basically to add a "runtime" dependency. --> https://bazel.build/concepts/dependencies#data-dependencies

You can use them the easy way using the location arguments or resolve them using the runfiles library (neatly descripte here: https://stackoverflow.com/a/64035908)

You can even pass additional arguments via the `bazel run` commandline like so:

```

bazel run //:plugin_installer -- /install/target/path

```

The arguments will just get added to the existing ones that you might have passed via the `args` attribute.

See https://bazel.build/reference/be/make-variables#predefined_label_variables on what you have at your disposal to pass via command line.

Best Regards

Fabian

[Kunal Gangakhedkar]

On Thu, 16 Mar 2023 at 20:22, fabian.bra...@gmail.com <fabian.bra...@gmail.com> wrote:

Hi Kunal, 

to achieve a simple install you can indeed use a custom binary.

To make the linkage "automatic" You can use a sh_binary. I have a fitting example here from some time ago:
https://github.com/FaBrand/bazel-learning/blob/master/sh_binary_args/BUILD#L8-L19

Notice the `data` attribute. This tells bazel basically to add a "runtime" dependency. --> https://bazel.build/concepts/dependencies#data-dependencies

You can use them the easy way using the location arguments or resolve them using the runfiles library (neatly descripte here: https://stackoverflow.com/a/64035908)

You can even pass additional arguments via the `bazel run` commandline like so:

```

bazel run //:plugin_installer -- /install/target/path

```

The arguments will just get added to the existing ones that you might have passed via the `args` attribute.

See https://bazel.build/reference/be/make-variables#predefined_label_variables on what you have at your disposal to pass via command line.

Perfect. Thanks for the help, Fabian.

Unfortunately, I got pulled into some other higher priority work - so, I'll have to get back to this bazel experiment in some time.

I'll keep you updated about the progress as and when I get to spend some time on this.

Thanks,

Kunal

 

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/RU_rtujalL4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c66ac17a-9e3b-444e-b24d-f1d74a925c8fn%40googlegroups.com.