A rule to run a java_binary that depends on a java_library runfiles

[T H]

Hey,

I have a rule that runs a java_binary target in an action. Two mandatory attributes of the rule are a java_library target and the name of a class in that library which the java_binary should load and invoke a method on.

So far I have managed to include the output jar and dependent jars of the library in the classpath of the java_binary via the --main_advice_classpath flag. (https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/java/java_stub_template.txt#L31)

However, the binary requires the runfiles of the java library to be in its own runfiles. No matter what I try passing to the action's tools/inputs, I cannot get the files into the binary's runfiles.

I have put together an reproducible example here: https://github.com/tmhdgsn/bazel-java-lab/blob/main/def.bzl#L19

Additionally I came across this issue, which could potentially be related?

https://github.com/bazelbuild/bazel/issues/15486 

Thanks!

Tom

[pcl...@google.com]

Hi,

you can probably build and use the deploy jar of the java_binary so that it runs standalone in an action and won't require the runfiles.

Check https://docs.bazel.build/versions/main/tutorial/java.html#package-a-java-target-for-deployment

Cheers,

Yun

[rleva...@google.com]

(James Duncan pointed me here and asked to share a bit; note I deal with Python, not Java, though)

If I understand correctly, the basic thing you're trying to do is something like a plugin model, right? Your rule provides the base framework ("invoker" attribute), users provide the plugin ("tool" attribute), and then when your tool runs, it loads the plugin code as part of its execution?

Here's a few ideas on how to make this work: define a new binary target with the additional dependencies, or use whatever Java-specific mechanism there is to tell a binary "here's extra stuff", or (maybe, not 100% sure) manually merge and creating your own runfiles tree.

1. Defining a new binary with extra deps. This is how you get the library into the binary's runfiles. This is the most sound solution. The drawback is defining a new binary for every usage doesn't scale as well (each usage is a unique binary, even if some are conceptually identical, so the aggregate cost of building adds up when it gets wide usage). To do this, during the macro phase, just define a java_binary and concat the extra dep on:

def invoke(name, tool, ...):

  java_binary(name=name + "_invoker", deps = [...] + [tool])

  _invoker(..., invoker=name + "_invoker")

This is the most sound because it avoids dynamically loading code. Dynamically loading code isn't wrong/bad, it just tends to have more edge cases that might fail or require more special care. Binaries are happiest when they know all the dependencies they need at build time.

This is also the only way to get "tool" directly into invoker's runfiles.

This is also the easiest since your get a Target in your rule, and can then just pass files_to_run along, and don't have to deal with any of the runfiles materializing the rest of this email talks about.

1b. Use a custom rule to merge the runfiles and symlink the executable.

I just thought of this one; I'm not entirely sure if this'll work, but you can try writing a rule to merge the runfiles and symlink the executable. Something like this:

def _merged_binary_impl(ctx):

  invoker = ctx.attr._invoker

  executable = ctx.actions.declare_file(ctx.label.name)

  ctx.actions.symlink(output=executable, target_file=invoker[DefaultInfo].files_to_run.executable)

  runfiles = invoker[DI].default_runfiles.merge(ctx.attr.tool[DI].default_runfiles)

  return [DefaultInfo(executable=executable, runfiles=runfiles)]

_merged_binary = rule(executable = True, ...)

def invoke(..., tool):

  _merged_binary(name=name + "_invoker", tool=tool)

  _invoker(invoker=name + "_invoker")

2. Saying "here's extra stuff" to a binary. How and if this can work is particular to how the Java binary and library rules work. But the basic idea is to tell the binary the location of the extra stuff using e.g. a flag or environment variable. If that extra stuff is just a file (e.g. a .jar), this should be pretty simple, e.g. just pass a flag: `args.add("--plugin", ctx.file.tool)`, or equiv. Were this Python, the gist would be something that ultimately modifies sys.path; idk what the Java equivalent is.

If it has to be a directory of all the runfiles as would be normally seen, then see (3)

3. Manually merging and materializing runfiles

You can basically mimick what Bazel does under the hood had (1) been done. So e.g. write starlark code to traverse the runfiles, declare_file and symlink for each file, put it all in a depset, calculate the logical runfiles root of your files, then pass that along to the action and tool. Using a fileset might be possible, too; not sure. Obviously, this means you have to flatten the depsets of the runfiles, so those usual caveats apply. I had to this once, and it's fairly straight forward, but kind of tedious/a pain; I was just packaging things into a zip file, though, I never tried to run anything out of it. You might have to fiddle more to make that work, idk.

Issue 15486 is sort of related to this. When I filed that, I wasn't thinking of merging arbitrary runfiles to create a new unified runfiles tree. I was more interested in just being able to pass runfiles directly into actions (my case wouldn't want to merge the runfiles). The relevant part of 15486 to this situation is the idea of being able to manually construct a FilesToRun-like object that has the triple of (executable, runfiles, runfiles_directory) (or similar), where you've specified, at the least, the executable and runfiles.

HTH!

[T H]

indeed the plugin model is the desired outcome

the current implementation I have is actually what you described in option 1, and for the exact reasons you mentioned regarding scaling is why I went down this route. The preference for runfiles however is mostly due to some "tools" already using the bazel_tools runfiles library. 

I've just taken your suggestions in 1b though and managed to get the example I posted to work ! 

Thank you very much!