Status of recursive WORKSPACEs?

[Jay Conrod]

What is the status of the Recursive WORKSPACE file parsing? I know Kristina was working on this a few months ago. Is it still going to be implemented?

We are having a problem in rules_go this would solve. We advise our users to add some boilerplate below to their WORKSPACE file:

git_repository(
    name = "io_bazel_rules_go",
    remote = "https://github.com/bazelbuild/rules_go.git",
    tag = "0.5.4",
)
load("@io_bazel_rules_go//go:def.bzl", "go_repositories")
go_repositories()

The go_repositories function adds dependencies on Go SDKs, our toolchains, and some other tools and utilities. Most of these dependencies are implemented in private repository rules, but some use public builtin rules like com_github_bazelbuild_buildtools, which uses http_archive. Our users would like to be able to override the public dependencies, but because of the private rules, we don't want them to inline and modify go_repositories. 

It doesn't seem like there's currently a good way to do this. If you declare a dependency on buildtools before the load statement, it gets overridden by the declaration in go_repositories. If you declare the dependency afterward, Bazel reports an error. A workaround is to add --override_repository in tools/bazel.rc, but this is ugly.

Recursive WORKSPACE file parsing would solve this by allowing parent WORKSPACE files to override dependencies declared in children. If this is still being developed, when will this be available? If not, is there a better way to override dependencies?

-Jay

[Josh Powell]

I've encountered related issues in my personal use of Bazel and rules_go as well.  I've thought that if a native rule 'existing_repository' in the same vein as existing_rule, it could be used to solve a lot of these problems.  A weaker version might even suffice, which only indicates if a repository with the given name exists, rather than returning its list of attributes.  The example of existing_rule for conditional instantiation uses this rule for a simple existence check, too.

In rules_go, you could imagine something like the following in repositories.bzl:

if native.existing_repository("com_github_bazelbuild_buildtools") == None:

    native.http_archive(

        name = "com_github_bazelbuild_buildtools",

        [...],

    )

[Damien Martin-Guillerez]

Hi Jay,

Unfortunately, as you might have heard, Kristina has recently left Google and the Bazel team for a new venture and she was leading that work.

Last time she tried to implement the recursive file parsing she hit a non trivial issue regarding the creation on the fly of external repositories. We probably need a more in-depth design before we can move forward. At the moment, unfortunately, nobody is working on it but it is definitely on our roadmap before 1.0

On Tue, Aug 29, 2017 at 12:54 AM Josh Powell <jwpo...@gmail.com> wrote:

I've encountered related issues in my personal use of Bazel and rules_go as well.  I've thought that if a native rule 'existing_repository' in the same vein as existing_rule, it could be used to solve a lot of these problems.  A weaker version might even suffice, which only indicates if a repository with the given name exists, rather than returning its list of attributes.  The example of existing_rule for conditional instantiation uses this rule for a simple existence check, too.

In rules_go, you could imagine something like the following in repositories.bzl:

if native.existing_repository("com_github_bazelbuild_buildtools") == None:

    native.http_archive(

        name = "com_github_bazelbuild_buildtools",

        [...],

    )

you can use native.existing_rule("com_github_bazelbuild_buildtools") already 

On Monday, August 28, 2017 at 11:35:24 AM UTC-6, Jay Conrod wrote:

What is the status of the Recursive WORKSPACE file parsing? I know Kristina was working on this a few months ago. Is it still going to be implemented?

We are having a problem in rules_go this would solve. We advise our users to add some boilerplate below to their WORKSPACE file:

git_repository(
    name = "io_bazel_rules_go",
    remote = "https://github.com/bazelbuild/rules_go.git",
    tag = "0.5.4",
)
load("@io_bazel_rules_go//go:def.bzl", "go_repositories")
go_repositories()

The go_repositories function adds dependencies on Go SDKs, our toolchains, and some other tools and utilities. Most of these dependencies are implemented in private repository rules, but some use public builtin rules like com_github_bazelbuild_buildtools, which uses http_archive. Our users would like to be able to override the public dependencies, but because of the private rules, we don't want them to inline and modify go_repositories. 

It doesn't seem like there's currently a good way to do this. If you declare a dependency on buildtools before the load statement, it gets overridden by the declaration in go_repositories. If you declare the dependency afterward, Bazel reports an error. A workaround is to add --override_repository in tools/bazel.rc, but this is ugly.

Recursive WORKSPACE file parsing would solve this by allowing parent WORKSPACE files to override dependencies declared in children. If this is still being developed, when will this be available? If not, is there a better way to override dependencies?

-Jay

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/55f188a0-d57b-4b55-96eb-d01795b682d7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[rodr...@google.com]

Hey Damien,

Thanks for the update. Do you have a short-term recommendation for what projects should be doing? eg checking existing_rule() or omit_com_... parameters like rules_closure does?

Best,
Rodrigo

rules_closure for reference: https://github.com/bazelbuild/rules_closure/blob/master/closure/repositories.bzl

[Jay Conrod]

Thanks Damien! I didn't know about native.existing_rule() and existing_rules(). They look like a good solution for our problem, and they may solve some other problems for us as well.

We will do something like this to allow some repositories to be overridden.

def go_repositories():

  if "com_github_bazelbuild_buildtools" not in native.existing_rules():

    native.http_archive(

        name = "com_github_bazelbuild_buildtools",

        ...,

    )

  ...

Rodrigo, I think this is an improvement over what rules_closure is doing, since users don't need to pass any arguments, and the interface doesn't need to change when a new dependency is added.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/7c2aeb5f-1e1e-4364-9d76-15683980d880%40googlegroups.com.

[Josh Powell]

Thanks, Damien.  I probably should have guessed, if for no other reason than the wording in https://docs.bazel.build/versions/master/skylark/repository_rules.html, that existing_rule(s) would work with repositories too.

[Rodrigo Queiro]

I think this is an improvement over what rules_closure is doing

I'm not sure which I prefer as a short-term solution. On the one hand, using existing_rules() is less boilerplate. On the other hand, it's less explicit so could lead to some weird cases. If a user calls two xyz_repositories() functions, which both need the same repo, whichever is called first will decide the version. The build then depends on the order of the calls.

On Tue, Aug 29, 2017 at 4:22 PM, Jay Conrod <jayc...@google.com> wrote:

Thanks Damien! I didn't know about native.existing_rule() and existing_rules(). They look like a good solution for our problem, and they may solve some other problems for us as well.

We will do something like this to allow some repositories to be overridden.

def go_repositories():

  if "com_github_bazelbuild_buildtools" not in native.existing_rules():

    native.http_archive(

        name = "com_github_bazelbuild_buildtools",

        ...,

    )

  ...

Rodrigo, I think this is an improvement over what rules_closure is doing, since users don't need to pass any arguments, and the interface doesn't need to change when a new dependency is added.

-- 

Google Germany GmbH | Erika-Mann-Strasse | 80636 Muenchen | Germany

AG Hamburg, HRB 86891 | Sitz der Gesellschaft: Hamburg | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[Damien Martin-Guillerez]

Rodrigo: I suggest going with existing_rules unless you are worried about version skew. Generally I prefer the one version policy.

On Tue, Aug 29, 2017 at 5:06 PM Rodrigo Queiro <rodr...@google.com> wrote:

I think this is an improvement over what rules_closure is doing

I'm not sure which I prefer as a short-term solution. On the one hand, using existing_rules() is less boilerplate. On the other hand, it's less explicit so could lead to some weird cases. If a user calls two xyz_repositories() functions, which both need the same repo, whichever is called first will decide the version. The build then depends on the order of the calls.

On Tue, Aug 29, 2017 at 4:22 PM, Jay Conrod <jayc...@google.com> wrote:

Thanks Damien! I didn't know about native.existing_rule() and existing_rules(). They look like a good solution for our problem, and they may solve some other problems for us as well.

We will do something like this to allow some repositories to be overridden.

def go_repositories():

  if "com_github_bazelbuild_buildtools" not in native.existing_rules():

    native.http_archive(

        name = "com_github_bazelbuild_buildtools",

        ...,

    )

  ...

Rodrigo, I think this is an improvement over what rules_closure is doing, since users don't need to pass any arguments, and the interface doesn't need to change when a new dependency is added.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/7c2aeb5f-1e1e-4364-9d76-15683980d880%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Rodrigo Queiro]

OK, thanks! Jay, shall I adjust my pull request to use existing_rules?

On Tue, Aug 29, 2017 at 5:53 PM, Damien Martin-Guillerez <dmar...@google.com> wrote:

Rodrigo: I suggest going with existing_rules unless you are worried about version skew. Generally I prefer the one version policy.

On Tue, Aug 29, 2017 at 5:06 PM Rodrigo Queiro <rodr...@google.com> wrote:

I think this is an improvement over what rules_closure is doing

I'm not sure which I prefer as a short-term solution. On the one hand, using existing_rules() is less boilerplate. On the other hand, it's less explicit so could lead to some weird cases. If a user calls two xyz_repositories() functions, which both need the same repo, whichever is called first will decide the version. The build then depends on the order of the calls.

On Tue, Aug 29, 2017 at 4:22 PM, Jay Conrod <jayc...@google.com> wrote:

Thanks Damien! I didn't know about native.existing_rule() and existing_rules(). They look like a good solution for our problem, and they may solve some other problems for us as well.

We will do something like this to allow some repositories to be overridden.

def go_repositories():

  if "com_github_bazelbuild_buildtools" not in native.existing_rules():

    native.http_archive(

        name = "com_github_bazelbuild_buildtools",

        ...,

    )

  ...

Rodrigo, I think this is an improvement over what rules_closure is doing, since users don't need to pass any arguments, and the interface doesn't need to change when a new dependency is added.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/7c2aeb5f-1e1e-4364-9d76-15683980d880%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 

Google Germany GmbH | Erika-Mann-Strasse | 80636 Muenchen | Germany

AG Hamburg, HRB 86891 | Sitz der Gesellschaft: Hamburg | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[Jay Conrod]

Yes, I'll follow up with you on that PR.