Runfiles library spec documentation

[Matt Stark]

I recently ran into an issue where runfiles stopped working in bazel 6, and it turned out the issue was because how runfiles works changed in this CL, so now it's stored under the directory _main instead of <workspace name>.

Xudong informed me that apparently Java and C++ have updated their runfiles libraries to support this, and that it should work if I use the new runfiles library constructor. However, the problem is that I was using rust, which never got the memo to update their runfiles library. Upon further discussion, he mentioned that Fabian was working on a runfiles library spec a few months ago.

In my opinion, there are several problems with the way we currently do runfiles:

• There is no clear specification for how runfiles should work. I've been using this, but it isn't up to date, is hard to find, and it doesn't look like official documentation. Because of this, we end up with issues like:
• rules_go: Complete rewrite of go runfiles library because it's inconsistent with other libraries
• rules_rust: Didn't support the RUNFILES_DIR environment variable. When I sent the PR in, the reviewers had no clue where the documentation was.
• Runfiles library authors are unaware of updates to the specification
• Runfiles library users are unaware of what the specification they're using is

I propose we solve this with an official, versioned specification for runfiles libraries (we can probably start based off what Fabian was working on). Here's how it would work, using the example CL above that broke me:

1. When the CL above  is created, there would be a change to the version specification (eg. v1.0 -> v2.0)
2. When a change occurs to the version specification, an email would go out to a mailing list for any runfiles authors (or maybe you could subscribe to have a bug automatically filed or something).
3. If I was the rust runfiles maintainer, I would see that email and update the runfiles library, and change the documentation for the rust runfiles library to say that it supported runfiles spec v2

--

Thanks, Matt.

[Fabian Meumertzheim]

I agree that this level of sophistication is probably worth it given the number of runfiles libraries that have been written across languages.

I am willing to contribute the original content of the spec, but I don't know where it should live and how exactly it should be versioned. I would welcome your opinions on that.

Fabian

[Matt Stark]

These questions sound like they should be deferred to the bazel team. @Xudong Yang, would you be able to pull in the appropriate contact on the team for these discussions?

--

Thanks, Matt.

[Xudong Yang]

I think the spec should live with the Bazel codebase. It's really just part of the documentation. So depending on what the spec looks like, it should either just live somewhere in the Bazel source tree (if it's in the form of, say, Starlark code), or live on the bazel.build website next to the other documentation (if it's in the form of prose).

"  YÁNG Xùdōng (杨旭东)

#  SURNAME Givenname

=  Software Engineer

@  Google Munich

[Fabian Meumertzheim]

I have been thinking about providing a commented reference Implementation in Starlark instead of a prose spec. Assuming that all ruleset authors should be reasonably familiar with Starlark anyway, that may do a better job at avoiding ambiguities in the spec. 

An additional advantage would be that the spec could be accompanied by unit tests that runfiles library authors could adapt and that the spec could run against in CI.

Since I'm not set sure whether that's a good idea, I am very much looking for opinions on "prose vs. reference implemention".

Fabian 

[Matt Stark]

I agree that a reference implementation should avoid ambiguity. Could you clarify what you mean by starlark though?

If you're referring to raw starlark (without bazel), then I'm not sure it can access environment variables (never used it, but couldn't find it in the spec)? Also, is there a particular reason to use starlark instead of python as the default language then? If writing in python, you can just choose to write without the features of python that might make it hard for a reader to understand.

If you're referring to doing it as a bazel rule, then I don't understand the point of it? IIUC, runfiles don't really make much sense for rules. Could you provide an example of what the outer layer might look like.

+1 to the point about unit tests.

--

Thanks, Matt.

[Fabian Meumertzheim]

I was thinking of raw Starlark with the (few) parts of I/O it can't do injected in other ways or stubbed, e.g.:

def create_runfiles(

        env, 

        # Stubbed by tests.

        is_file = _is_file):

    runfiles_manifest = env.get("RUNFILES_MANIFEST_FILE")

    if is_file(runfiles_manifest):

        ...

    ...

def _is_file(path):

    """Returns True if the OS-specific path points to an existing file."""

    fail("To be implemented by a concrete implementation")

There are a couple reasons why we might want to prefer Starlark:

1. Ruleset authors may not be familiar with the behavior of Python's OS-specific functions (e.g. which path separators are supported by which function on which platform?), leading to potential edge case bugs.

2. The Python runfiles library would subtly differ from the Python runfiles library reference implementation (the former doesn't need to locate runfiles relative to argv[0]), which could lead to confusion.

3. If we restrict ourselves to a subset of Python that we truly expect everyone to be familiar with, we are probably close to writing Starlark anyway ;-)

That said, I can also see arguments in favor of Python, so I'm open to both as long as we have properly weighed the pros and cons.

[Matt Stark]

I think IMO, I'd be in favor of python over starlark. As python is basically a superset of starlark, anything we can do in starlark, we can do in python just as easily. I also prefer the idea of using a language that's more similar to what other languages will be used to. For example, unit tests in python should be easy to replicate in other languages, since we can literally just write:

py_test(

 ...,

  data = glob(["testdata/**"])

)

Then we can just use the library as normal, instead of having to stub things out in the tests.

1. Ruleset authors may not be familiar with the behavior of Python's OS-specific functions (e.g. which path separators are supported by which function on which platform?), leading to potential edge case bugs.

IMO this can be solved relatively easily with either good comments for the code, or by appropriate use of the standard library (eg. os.path.sep, os.path.join, etc should help make it platform-independent).

And even if this was a problem, we can just choose not to use python's OS functions. I understand that at that point, we'd be mostly writing starlark, but as I mentioned in response to your point 3, I don't think there's actually an issue with that?

2. The Python runfiles library would subtly differ from the Python runfiles library reference implementation (the former doesn't need to locate runfiles relative to argv[0]), which could lead to confusion.

I'm a little confused here - why doesn't the python runfiles library need to locate runfiles relative to argv[0]? If they differ, isn't that, by definition, a bug in the implementation? And adding that feature to the python runfiles library should be backwards compatible, right?

 

3. If we restrict ourselves to a subset of Python that we truly expect everyone to be familiar with, we are probably close to writing Starlark anyway ;-)

I don't disagree, but this doesn't sound like a reason to write it in starlark - I don't think there's anything wrong with writing simple python code 

--

Thanks, Matt.

[Fabian Meumertzheim]

It's true that this kind of stubbing may seem foreign to developers. 

The Python runfiles library doesn't have to locate runfiles relative to argv[0] because Python scripts are executed by a dedicated launcher that sets the RUNFILES_DIR/RUNFILES_MANIFEST_FILE environment variables. This part is implemented in python_bootstrap_template.txt (for Unix) and python_launcher.cc (for Windows) in a way that's actually somewhat scary (it seems to follow symlinks until it finds itself under any runfiles directory). 

If it weren't for that special handling, we could just elevate the existing runfiles library to be the reference implementation. But maybe we can just copy it and add the proper argv[0] handling? It existed as dead code in a previous version.

[Matt Stark]

Adding the proper argv[0] handling sounds good to me.

--

Thanks, Matt.

[Fabian Meumertzheim]

Since I don't want to block anyone but realistically won't get to writing a proper spec soon, I'm going to collect some references that should help folks write their own compliant runfiles libraries.

Depending on the language your are most familiar with, I would recommend studying either of these existing runfiles libraries, all of which are compatible with Bzlmod:

- Go (https://github.com/bazelbuild/rules_go/blob/master/go/runfiles/runfiles.go)

- Python (https://github.com/bazelbuild/rules_python/blob/main/python/runfiles/runfiles.py)

- Java (https://github.com/bazelbuild/bazel/blob/master/tools/java/runfiles/Runfiles.java)

All of these provide the Rlocation and EnvVars functions that comprise the core API of a runfiles library as described in https://docs.google.com/document/d/e/2PACX-1vSDIrFnFvEYhKsCMdGdD40wZRBX3m3aZ5HhVj4CtHPmiXKDCxioTUbYsDydjKtFDAzER5eg7OjJWs3V/pub.

Since Java and Python can rely on launchers setting certain environment variables, their runfiles don't contain the logic that derives the location of the runfiles directory and manifest from argv[0]. A Python implementation of this part is available as the code deleted in this commit: https://github.com/bazelbuild/rules_python/commit/86b01a3a1b30b2244b86c80181f42492abaa09a1

The tricky part of making a runfiles library compatible with Bzlmod is that runfiles lookups require the canonical repository name of the Bazel target containing the source file that calls Rlocation as context. This information either has to be injected at compile-time (for compiled languages) or determined at runtime by parsing source file paths and is used to resolve apparent to canonical repository names (e.g. turn rules_go into rules_go~0.38.1). The languages with Bzlmod support all adopted their own approach to implement this CurrentRepository function:

* The C++ rules add a BAZEL_CURRENT_REPOSITORY define containing the repository name to all compilation actions. Users are expected to pass this value in when creating a Runfiles instance (or alternatively to Rlocation).

* Java offers the AutoBazelRepository annotation that causes a class to be generated that has the repository name in a static constant. Users are expected to pass this value in when creating a Runfiles instance.

* Go relies on runtime.Caller to get the execpath of a source file and parses the repository name out of it, using that the canonical repository name of the main repository is always the empty string. Users don't have to manually pass in the name.

* Python relies on sys._getframe and inspect.getfile and parses the repository name out of the runfiles path, using that with Bzlmod the name of the runfiles directory of the main repository is always _main. Users don't have to manually pass in the name.

If you have any questions, feel free to reach out to me. I am still planning to produce a comprehensive spec at some point.

Fabian