How to write a "symlink" rule? (also rules_go related)

[tom....@centralway.com]

Hi,

tl;dr How do you write a Bazel rule to symlink to an existing target? I'd like the invocation to look like:

# this should create a symbolic link from my-binary to some/deep/dir/elsewhere/my-binary:my-binary.
symlink(
name = "my-binary",
src = "//some/deep/dir/elsewhere/my-binary:my-binary",
)

I've read previous related discussions [1] and the Skylark examples [2] but am currently unable to come up with a solution.

Background:

I'm converting a large project from using the standard Go tools ("go build"/"go install) to Bazel. The project has several tens of binaries scattered across its directory hierarchy and I'd like to symlink or copy these to a single "bin" directory in "bazel-bin". Collecting everything in a single directory greatly simplifies further tooling (e.g. it's easy to add to my $PATH) and emulates the "go install" step that other tooling relies on.

My current best effort is:

def _impl(ctx):
output = ctx.outputs.out
# go_binary rules have two outputs, a library and the binary itself.
# The following is a horrible hack to find the binary, not the library.
for f in ctx.attr.src.files:
if not f.path.endswith(".a"):
input = f
break
ctx.action(
inputs=[input],
outputs=[output],
mnemonic="Symlink",
command="mkdir -p %s && ln -s %s %s" % (output.dirname, input.path, output.path))

symlink = rule(
implementation=_impl,
attrs={
# We can't use single_file=True because we'd like to use this with
# go_binary targets.
"src": attr.label(mandatory=True, allow_files=True),
},
outputs={
"out": "%{name}",
},
)

This fails in a number of ways:

1) I get a warning about the name of the rule conflicting with the name of the output:

WARNING: BUILD.bazel:3:1: target 'my-binary' is both a rule and a file; please choose another name for the rule.

I'd really like to avoid stuttering the name of the symlink (i.e. I'd like the output file to be the name of the rule). Is there any way to do this and avoid the warning?

2) The generated symlink is relative and does not point to the actual location of the src:

$ ls -l bazel-bin/bin
lrwxr-xr-x 1 user wheel 71 Jan 4 14:23 my-binary -> bazel-out/local-fastbuild/bin/some/deep/dir/elsewhere/my-binary:my-binary

Bazel warns about this:

ERROR: BUILD.bazel:3:1: output 'bin/my-binary' is a dangling symbolic link.

How can I get either a correct relative path or an absolute path to the actual file on disk? None of the File properties [3] seem relevant, and the correct relative path will depend on where I invoke the symlink() rule.

3) Is there any nice way to work around the double outputs (binary and library) of rules_go's go_binary?

Thanks very much for any help!
Tom

[1] https://groups.google.com/forum/#!topic/bazel-discuss/EoiHjBPLNuY
[2] https://bazel.build/versions/master/docs/skylark/cookbook.html
[3] https://bazel.build/versions/master/docs/skylark/lib/File.html

[John Cater]

Why are you creating a symlink? Bazel already has alias (https://bazel.build/versions/master/docs/be/general.html#alias) for referring to a target by another name. What is your use case?

--
This message is for the attention of the intended recipient(s) only. It may
contain confidential, proprietary and/or legally privileged information.
Use, disclosure and/or retransmission of information contained in this
email may be prohibited. If you are not an intended recipient, you are
kindly asked to notify the sender immediately (by reply e-mail) and to
permanently delete this message. Thank you.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/96dca868-3db2-4e5a-8cdd-bf02dea176dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Payne]

Thanks for the reply!

The use case is that I'd like to collect multiple binaries from across the directory structure into a single directory. I don't mind if it's a symlink or a copy, although some of the binaries are quite large so a symlink would probably be quicker.

The important end goal is that a single directory somewhere in "bazel-bin/" contains multiple compiled binaries from elsewhere in the tree. They need to be in a single directory because other tooling relies on this.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/96dca868-3db2-4e5a-8cdd-bf02dea176dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

[John Cater]

One option I've seen for this is to create a top-level pkg_tar (https://www.bazel.io/docs/be/pkg.html#pkg_tar) target that collects the various targets and generates a single archive, which can then be copied around to other locations as needed. Would that help you out?

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/96dca868-3db2-4e5a-8cdd-bf02dea176dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Payne]

I did try this, but I really need the individual binaries as I'd like to run them directly from "bazel-bin/".

The equivalent rule in make would be:

my-binary: some/deep/dir/elsewhere/my-binary

  cp $< $@

Surely this is possible in Bazel?

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/96dca868-3db2-4e5a-8cdd-bf02dea176dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

This message is for the attention of the intended recipient(s) only. It may contain confidential, proprietary and/or legally privileged information. Use, disclosure and/or retransmission of information contained in this email may be prohibited. If you are not an intended recipient, you are kindly asked to notify the sender immediately (by reply e-mail) and to permanently delete this message. Thank you.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

[John Cater]

In general, the path your are following is the way to go then, a custom rule that invokes "ln -s" with the right paths.

Unfortunately, I don't know enough about the way the go rules create the binaries to answer the question about the binary vs the .a file.

My suggestion for the relative symlink would be to add an extra $PWD/ in the path you are linking from to make it absolute, not relative.

Sorry I can't be more help!

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/96dca868-3db2-4e5a-8cdd-bf02dea176dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

This message is for the attention of the intended recipient(s) only. It may contain confidential, proprietary and/or legally privileged information. Use, disclosure and/or retransmission of information contained in this email may be prohibited. If you are not an intended recipient, you are kindly asked to notify the sender immediately (by reply e-mail) and to permanently delete this message. Thank you.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

[Tom Payne]

Thanks very much John! I'll explore this option.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/96dca868-3db2-4e5a-8cdd-bf02dea176dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

This message is for the attention of the intended recipient(s) only. It may contain confidential, proprietary and/or legally privileged information. Use, disclosure and/or retransmission of information contained in this email may be prohibited. If you are not an intended recipient, you are kindly asked to notify the sender immediately (by reply e-mail) and to permanently delete this message. Thank you.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

This message is for the attention of the intended recipient(s) only. It may contain confidential, proprietary and/or legally privileged information. Use, disclosure and/or retransmission of information contained in this email may be prohibited. If you are not an intended recipient, you are kindly asked to notify the sender immediately (by reply e-mail) and to permanently delete this message. Thank you.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

[Tom Payne]

To wrap up this thread, here's the solution that eventually worked for me:

def _impl(ctx):

    # Create an absolute symlink to a Go binary. We cannot use a relative

    # symlink as the output symlink is accessible via several different paths

    # (e.g. bazel-bin/, bazel-out/local-fastbuild/bin/) of different depths so

    # no single relative symlink can be correct.

    for input in ctx.attr.src.files:

        # go_binary rules have two outputs, a library and the binary itself.

        # The following is a horrible hack to avoid copying the library.

        if input.path.endswith(".a"):

            continue

        output = ctx.outputs.out

        input_root = str(input.root)

        # Compute the actual input root by stripping off the "[derived]" suffix

        # that Bazel insists on adding.

        if input_root.endswith("[derived]"):

            input_root = input_root[:-len("[derived]")]

        ctx.action(

            inputs=[input],

            outputs=[output],

            mnemonic="SymlinkGoBinary",

            command="mkdir -p %s && ln -sf %s/%s %s" % (output.dirname, input_root, input.short_path, output.path))

symlink_go_binary = rule(

    implementation=_impl,

    attrs={

        "out": attr.output(mandatory=True),

        # We can't use single_file=True because go_binary rules have two

        # outputs.

        "src": attr.label(mandatory=True, allow_files=True),

    },

)

I couldn't find a way to use the name of the rule as the output without getting a warning from Bazel, so you have to stutter when using it:

symlink_go_binary(
    name = "symlink-my-binary",

    out = "my-binary",

    src = "//some/deep/dir/elsewhere/my-binary:my-binary",
)

The $PWD trick to compute the absolute path didn't work because it includes the path to the sandbox. Instead I used input.root + "/" + input.short_path, except that input.root gets a "[derived]" suffix for no apparent reason that has to be stripped off. I suspect there's a cleaner way of computing the absolute path of the input.

Regards,

Tom