navigating the Bazel "path" morass

[Matthew Moore]

Basically, I am trying to make sense of File's path / short_path / root.

According to the live docs, ".path" is defined to be:

The execution path of this file, relative to the workspace's execution directory. It consists of two parts, an optional first part called the root (see also the root module), and the second part which is the short_path. The root may be empty, which it usually is for non-generated files. For generated files it usually contains a configuration-specific path fragment that encodes things like the target CPU architecture that was used while building said file. Use the short_path for the path under which the file is mapped if it's in the runfiles of a binary.

(emphasis mine), which I take to mean: path == os.path.join(root.path, short_path)...?

However, this is not true in general.  Leveraging Doug Greiman's brilliant @subpar//:debug.bzl I am able to see:

    ...

    path[string]: external/com_google_guava_guava/jar/guava-18.0.jar

    root[root]:

        path[string]:

    short_path[string]: ../com_google_guava_guava/jar/guava-18.0.jar

Basically what I'm trying to do is lay out the dependencies of a particular binary target in the context of a filesystem of an appropriate Docker base image.  I can get some basic things working, but as soon as the dependencies get mildly complex (e.g. here using guava via maven_jar), my attempts at this fall over, likely due to bad assumptions about the behavior of these properties.

This code is possibly also at some fault, but I think that it too suffers from managing the complexity of these properties.

Is there any better documentation floating around?

thanks,

-M

--

Matthew Moore

DI/Docker

Developer Infrastructure @ Google

[Matthew Moore]

I opened this to track a clear bug in the logic I linked above.

I've assigned it to Damien for now because IIRC he added the data_path logic (ages ago), and I'm not sure how to fix this given the problems highlighted above.

Again, if there are better docs for this stuff, I'd love a pointer.

-M

[Kristina Chodorow]

The docs on this are supremely confusing.  They are only correct if you're using the main repository.

For external repositories, the path/short path are as you've outlined.  I'd assume that someone used short_path in the Docker rules because they didn't want the bazel-out/... substring, but short_path for external repos assumes that you're running a binary from the bin.runfiles/main_ws directory and external repositories need a ../repo-name to escape that.  It would be a lot handier if Skylark had more of a standard library for path operations, but the Docker rules probably need to implement a bit more complicated logic for external repo paths.  I'm not sure what would be sensible, but maybe putting everything under its repository name.

On Mon, Aug 7, 2017 at 6:54 PM, 'Matthew Moore' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:

I opened this to track a clear bug in the logic I linked above.

I've assigned it to Damien for now because IIRC he added the data_path logic (ages ago), and I'm not sure how to fix this given the problems highlighted above.

Again, if there are better docs for this stuff, I'd love a pointer.

-M

On Mon, Aug 7, 2017 at 3:04 PM, Matthew Moore <matt...@google.com> wrote:

Basically, I am trying to make sense of File's path / short_path / root.

According to the live docs, ".path" is defined to be:

The execution path of this file, relative to the workspace's execution directory. It consists of two parts, an optional first part called the root (see also the root module), and the second part which is the short_path. The root may be empty, which it usually is for non-generated files. For generated files it usually contains a configuration-specific path fragment that encodes things like the target CPU architecture that was used while building said file. Use the short_path for the path under which the file is mapped if it's in the runfiles of a binary.

(emphasis mine), which I take to mean: path == os.path.join(root.path, short_path)...?

This is true for local targets.

 

However, this is not true in general.  Leveraging Doug Greiman's brilliant @subpar//:debug.bzl I am able to see:

    ...

    path[string]: external/com_google_guava_guava/jar/guava-18.0.jar

    root[root]:

        path[string]:

    short_path[string]: ../com_google_guava_guava/jar/guava-18.0.jar

Basically what I'm trying to do is lay out the dependencies of a particular binary target in the context of a filesystem of an appropriate Docker base image.  I can get some basic things working, but as soon as the dependencies get mildly complex (e.g. here using guava via maven_jar), my attempts at this fall over, likely due to bad assumptions about the behavior of these properties.

This code is possibly also at some fault, but I think that it too suffers from managing the complexity of these properties.

Is there any better documentation floating around?

thanks,

-M

--

Matthew Moore

DI/Docker

Developer Infrastructure @ Google

--

Matthew Moore

DI/Docker

Developer Infrastructure @ Google

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACW46hfgb8vSVELjUKxByguTSwU7pwdxA9ZUx3bezLV5eEOjLQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Matthew Moore]

It would be a lot handier if Skylark had more of a standard library for path operations, but the Docker rules probably need to implement a bit more complicated logic for external repo paths

+1000, and I don't think this is limited to paths :-D

I suspect we're going to need to support a "files" dict to allow users to specify the path to which each file should map, which could be a nasty change...

-M

[Matthew Woehlke]

On a related note... something else that would be useful is to be able
to get the path relative to the current BUILD file. Drake needs this in
order to implement installs that preserve partial paths. See output_path
in https://github.com/RobotLocomotion/drake/blob/master/tools/pathutils.bzl.

(It would also be nice to see some of the basics of os.path available.
I'm thinking about join, dirname, basename and splitext, in particular.
Drake would also benefit from the ability to remove a prefix from a
path, where the prefix may contain globs. We have an implementation of
this in skylark, but it has some limitations.)

Matthew

On 2017-08-09 15:33, 'Kristina Chodorow' via bazel-discuss wrote:
> The docs on this are supremely confusing. They are only correct if you're
> using the main repository.
>
> For external repositories, the path/short path are as you've outlined. I'd
> assume that someone used short_path in the Docker rules because they didn't
> want the bazel-out/... substring, but short_path for external repos assumes
> that you're running a binary from the bin.runfiles/main_ws directory and
> external repositories need a ../repo-name to escape that. It would be a
> lot handier if Skylark had more of a standard library for path operations,
> but the Docker rules probably need to implement a bit more complicated
> logic for external repo paths. I'm not sure what would be sensible, but
> maybe putting everything under its repository name.
>
>
>
> On Mon, Aug 7, 2017 at 6:54 PM, 'Matthew Moore' via bazel-discuss <

> bazel-...@googlegroups.com> wrote:
>
>> I opened this <https://github.com/bazelbuild/rules_docker/issues/106> to

>> track a clear bug in the logic I linked above.
>>
>> I've assigned it to Damien for now because IIRC he added the data_path
>> logic (ages ago), and I'm not sure how to fix this given the problems
>> highlighted above.
>>
>>
>> Again, if there are better docs for this stuff, I'd love a pointer.
>> -M
>>
>> On Mon, Aug 7, 2017 at 3:04 PM, Matthew Moore <matt...@google.com> wrote:
>>
>>> Basically, I am trying to make sense of File's path / short_path / root.
>>>
>>>
>>> According to the live docs

>>> <https://docs.bazel.build/versions/master/skylark/lib/File.html#path>,

>>> ".path" is defined to be:
>>>
>>>> The execution path of this file, relative to the workspace's execution

>>>> directory. *It consists of two parts, an optional first part called the

>>>> root (see also the root module), and the second part which is the

>>>> short_path*. The root may be empty, which it usually is for

>>>> non-generated files. For generated files it usually contains a
>>>> configuration-specific path fragment that encodes things like the target
>>>> CPU architecture that was used while building said file. Use the short_path
>>>> for the path under which the file is mapped if it's in the runfiles of a
>>>> binary.
>>>>
>>>

>>> (*emphasis* mine), which I take to mean: path == os.path.join(root.path,

>>> short_path)...?
>>>
>>
> This is true for local targets.
>
>
>>
>>>
>>> However, this is not true in general. Leveraging Doug Greiman's
>>> brilliant @subpar//:debug.bzl I am able to see:
>>> ...
>>> path[string]: external/com_google_guava_guava/jar/guava-18.0.jar
>>> root[root]:
>>> path[string]:
>>> short_path[string]: ../com_google_guava_guava/jar/guava-18.0.jar
>>>
>>>
>>>
>>>
>>> Basically what I'm trying to do is lay out the dependencies of a
>>> particular binary target in the context of a filesystem of an appropriate
>>> Docker base image. I can get some basic things working, but as soon as the
>>> dependencies get mildly complex (e.g. here using guava via maven_jar), my
>>> attempts at this fall over, likely due to bad assumptions about the
>>> behavior of these properties.
>>>
>>> This

>>> <https://github.com/bazelbuild/rules_docker/blob/master/docker/build.bzl#L65-L75>

>>> code is possibly also at some fault, but I think that it too suffers from
>>> managing the complexity of these properties.
>>>
>>>
>>> Is there any better documentation floating around?
>>>
>>>
>>> thanks,
>>> -M
>>>
>>>
>>>
>>> --
>>> Matthew Moore
>>> DI/Docker
>>> Developer Infrastructure @ Google
>>>
>>
>>
>>
>> --
>> Matthew Moore
>> DI/Docker
>> Developer Infrastructure @ Google
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "bazel-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to bazel-discus...@googlegroups.com.

>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/bazel-discuss/CACW46hfgb8vSVELjUKxByguTSwU7pwdxA9ZUx3bez
>> LV5eEOjLQ%40mail.gmail.com

>> <https://groups.google.com/d/msgid/bazel-discuss/CACW46hfgb8vSVELjUKxByguTSwU7pwdxA9ZUx3bezLV5eEOjLQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .