Re: Musings about external repository support

[Tony Aiuto]

Hi Mark.

I just wanted to close the loop. Others answered before I got to my mail, so you see we did start the discussion group.

And, as Xudong said, we devoted some people to work on this area specifically.

You mentioned "how this might fit into the evolution of Bazel dependency management". I think we are at a point

where the evolution is whatever the community decides. We need to collect user journeys and specific needs

so we can get a comprehensive view of what people need.  Posting them to the list for discussion is one way

to start.

Tony

On Thu, Aug 20, 2020 at 8:19 PM Joe Hicks <joeh...@google.com> wrote:

+extern...@bazel.build

On Thu, Aug 20, 2020 at 5:15 PM Xudong Yang <w...@google.com> wrote:

Hey Mark,

We now have a three-person strike team working on the external deps project (Philipp, Yun, and myself). We're currently still in the phase of gathering information and exploring the solution space -- there are a lot of questions to answer :) Once things settle down a bit more, we'll start doing interviews with community members and potentially send out some questionnaires.

And yes, I think it would be a good idea to use this year's BazelCon to share some of the findings and early ideas that we have, and to get the community involved. Hopefully by October/November we'll have more concrete stuff to share.

Xudong

On Fri, Aug 21, 2020 at 9:52 AM 'Joe Hicks' via bazel-dev <baze...@googlegroups.com> wrote:

Hi Mark,

This may be what you are looking for: https://groups.google.com/a/bazel.build/g/external-deps .

Also, feel free to join the Slack channel #external-deps in Bazel's slack.

Let me know if you need more info.

Thanks,

Joe

On Thu, Aug 20, 2020 at 2:35 PM Mark Zeren <mze...@vmware.com> wrote:

Tony,

 

Was this mailing list ever created?

 

Philipp,

 

Have you had a chance to draft the document that you suggested?

 

At VMware Rehana Tabassum and I have been investigating "rules_license"-like support for one of our internal builds. We are interested in how that might fit into the evolution of Bazel dependency management.

 

Can we use the upcoming BazelCon (CFP ends a week from today!) as a forcing function to get this discussion started?

 

Mark

 

 

From: Tony Aiuto <ai...@google.com>
Date: Wednesday, June 10, 2020 at 9:52 PM
To: Philipp Wollermann <phi...@google.com>
Cc: Joe Hicks <joeh...@google.com>, Jakob Buchgraber <buc...@google.com>, "Lukács T. Berki" <lbe...@google.com>, Steve Billings <Steve.B...@synopsys.com>, Mark Zeren <mze...@vmware.com>, László Csomor <las...@engflow.com>, bazel-dev <baze...@googlegroups.com>, Julio Merino <jm...@google.com>, Sven Tiffe <sti...@google.com>, Janak Ramakrishnan <jan...@google.com>, Tobias Werth <twe...@google.com>
Subject: Re: Musings about external repository support

 

SGTM.  My only strong suggestion right now is to make a new mailing list rather than bazel-discuss+X.  That makes it much easier for people to filter away bazel-discuss but be actively involved in this discussion.

 

On Wed, Jun 3, 2020 at 6:30 AM Philipp Wollermann <phi...@google.com> wrote:

Hello,

 

Thank you all for your input. I agree that we need to listen to and gather input from users and stakeholders.

I'd like to propose a different approach that should achieve this in a simpler way.

 

First, I will write a public design doc that proposes a solution for Bazel's handling of external dependencies.

It will be based on my understanding of the requirements and all the docs and stories that have been shared with me in the past months.

We can then discuss this design on the document itself and in the newly created Slack channel #external-deps in Bazel's slack.

Feel free to already join there and let me know about your thoughts and requirements, so that I can address them as early as possible.

I'll make sure that I'm online and available in Slack during working hours.

 

I would like to use asynchronous communication (email, chat, comments on docs) rather than meetings / video conferencing to work on this project, as it will be more accessible for people who want to follow our progress and leaves a better trail of thoughts and work done.

 

When the design has been discussed appropriately, we will build the solution for this as an open-source tool on GitHub in the bazelbuild organization.

If you would like to participate in the development there, please let me know and I'll see how we can organize this when we get to that phase.

I'm also happy to take issues and PRs that fit the design constraints there.

 

Cheers,

Philipp

 

On Wed, Jun 3, 2020 at 9:09 AM Joe Hicks <joeh...@google.com> wrote:

Thanks Tony for reframing the conversation. I support creating a working group for this. Who do you see as being the chair and when would you like to start this working group?

 

Thanks again,

 

Joe

 

On Fri, May 29, 2020 at 11:53 AM Tony Aiuto <ai...@google.com> wrote:

+Steve Billings, +Mark Zeren +Joe Hicks 

 

We are already lost in the weeds of "this technique won't work because of that".  Let's restart the discussion.

 

I agree with Lukács that our current implementation has problems. I think we all do. I'll go a step further and state that I think dependency management is probably the single biggest usability improvement we can make to Bazel to improve adoption.

 

I also think that many of us believe we can go and write a better design or evaluate someone else's design - right now.  I don't believe that. After multiple discussions with users and having read many open issues I have come to the conclusion that a few simple changes won't improve the product measurably. This is an initiative that demands input and effort from beyond the Bazel team.

• we need to gather real world use cases and their relative importance
• we need to synthesize that up to requirments for a replacement
• we need to plan and deliver implementations in a way that the right teams, including users, can contribute to each part.

One reason I explicitly added some non-team users to this thread is because I do not believe the Bazel team (which is mostly Googlers) should attempt this alone is that, as Lukács hinted at, we are a monorepo shop and our requirements gathering process is tainted by that. Nor do I believe that the Google Bazel team has enough engineering slack to pre-commit to taking this all on. Ultimately, actually getting things done involves finding the people to do the work. We need all the stakeholders to share the work.

 

I propose that we form a working group for this effort to shepherd it from requirements gathering through implementation and finally to maintaining the ongoing future maintenance.  

• formsl WG structure: chairperson, distinct mailing list for all communications, regular meetings (can do public VC even)

• actively solicit membership in the WG to users large enough to have interesting needs and/or capable of contributing to the work (joehicks@ can certainly help there)

• expected deliverables (3 month time frame - not to linger for a year)

1. requirements from stakeholders
2. develop and solicit design proposals
3. manage design reviews (note: not quite the same as *do* design reviews, that may be delegated)
4. selection of designs against resource availability to implement. It does us no good to design something that we don't have the engineers to build

• management of the work

• Negotiating who gets to build each piece,
• Sitting on the schedule
• Coordinate feature drops into Bazel releases with promise of active evaluation/testing by user stakeholders

I do believe that, between Sven and I, we can afford to make someone available to chair(1) the group and lead it through at least the requirements gathering phase. What we build and who builds it is another round of negotiations. We can face those later.  If the group produces good designs and we all believe they can be staffed, great - keep going. If it turns into bike-shedding and political infighting for features, we shut it down and design and build from scratch.

 

1) To answer the inevitable question about why the chair has to be from Google - I say that because I think that ultimately Google will be doing a lot of the work, or stuck with maintaining it, so I want to head off a group creating a wishlist of features that we don't have the resources to build.

 

 

 

On Fri, May 29, 2020 at 7:52 AM 'Jakob Buchgraber' via bazel-dev <baze...@googlegroups.com> wrote:

I would argue that before thinking about versioning and dependency resolution the Bazel team should think about the interfaces and mechanisms

of Bazel to pull in dependencies and inspect its build environment. That's strictly separate from versioning and would already be a huge win.

 

I think it certainly makes sense for a tool that pulls in Go packages to use MVS. However, it's easy to see that this won't generally work because

MVS assumes semantic versioning. Many ecosystems don't use semantic versioning. So you will likely need to have ecosystem specific logic to

do dependency resolution and having to define contracts between the ecosystems in order to make multi language builds work. All this is probably

best done in a tool separate from Bazel.

 

Jakob Buchgraber

Software Engineer

 

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

 

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

 

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

 

 

On Wed, May 13, 2020 at 12:20 PM 'Philipp Wollermann' via bazel-dev <baze...@googlegroups.com> wrote:

By the way, we discussed our problems with dependency management and thoughts with the people who built the Go versioning design and package manager.

 

Our conclusion was that we should probably just profit from their research into this complex topic and adopt their design principles like using the Minimal Version Selection algorithm instead of the classic "SAT solver and Lockfiles" approach for our own tooling as well. However, we didn't have time to actually build something yet.

 

On Wed, May 13, 2020 at 11:54 AM Lukács T. Berki <lbe...@google.com> wrote:

 

 

On Wed, May 13, 2020 at 9:30 AM László Csomor <las...@engflow.com> wrote:

That's a great summary Lukács.

 

This topic is dear to me. I wrote an internal doc about it, but I no longer work at Google so I lost access. Could that be published?

Here it is:

 

https://docs.google.com/document/d/1ldbIa5n1goxcRTbrnLklJJuy7aIO_Qc3vPFmZPc8tDU/edit#

 

 

If we started over, I'd delegate package management to an external tool. Divide and conquer. The silver lining is, it's still possible to go that way.

 

--

László Csomor | las...@engflow.com | EngFlow GmbH

Fischerweg 51, 82194 Gröbenzell, Germany | Geschäftsführer (CEO): Ulf Adams

https://www.engflow.com

 

 

 

On Wed, May 13, 2020 at 9:17 AM 'Lukács T. Berki' via bazel-dev <baze...@googlegroups.com> wrote:

Hey there,

 

In the wake of a virtual water cooler discussion about external repository support, I think it's worth summarizing the decisions that went into supporting them in Bazel.

 

The decision was mostly taken because Bazel has its roots in a monorepo world, but most of the world does not work that way and a large majority of projects do have dependencies that need to be fetched. We thought that the large chunk of complexity it adds is a good price for being able to start using Bazel in a non-monorepo project.

 

There were two alternatives we seriously thought about:

1. Declaring that Bazel is a monorepo tool. This would have resulted in a barrier of entry that's too high or, even worse, people working around Bazel not being able to access source code outside the main source tree by various custom ways.
2. Delegating fetching external repositories to a separate tool. We thought that this would result in a large amount of friction since one would have to manually call it before calling Bazel and that it went against the "correct" part of our tagline, since then Bazel would not have recognized changes to e.g. the WORKSPACE file. 

Then, in the last five years, the API surface for external repositories became very complex and they started to be used for things we did not think about when we started out; they have become the default way to inject non-hermetic information into the build.

 

Most notably, in some cases, we started doing transitive dependency resolution, thereby turning Bazel into an accidental multi-language package manager, which is a hard problem even when we constrain ourselves to a single programming language.

 

I was about to add an "if I had to start again" paragraph, but then I realized that I don't know what I would do differently, even with the knowledge I have today (other than not putting external repositories at the $EXECROOT/external, which is prone to conflicts with the source tree, but that's only a small slice of the problem)

 

Appendix: the initial design doc is here (it also has a version from 2014, which we wrote in preparation for the initial code drop)

 

 

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAOu%2B0LUPHx7ofrY%3DoHmKNjn2UpU3fp9rgeQ%2Bkez%2Bbw_TTvm7yg%40mail.gmail.com.

 

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

 

--

Philipp Wollermann | Software Engineer | phi...@google.com
Google Germany GmbH | Erika-Mann-Straße 33 | 80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CA%2BAhZoi0cvLHXg%3DgZQ%2B6aNNu8FpQbZ7mAzZbLsteL421TTe%2Bjg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAGQ4vn1VgVkWUdg7fB9FAqhj-AOzXqYjinfKRGnzfu0XCOaPDw%40mail.gmail.com.

 

--

Joe Hicks
Product Manager, Google Core Developer

 

--

Philipp Wollermann | Software Engineer | phi...@google.com
Google Germany GmbH | Erika-Mann-Straße 33 | 80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

--

Joe Hicks
Product Manager, Google Core Developer

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAMDFopgxcEi72OBTUP1h_yqqs3gbkmStGiO%2B3%2BfiexRVd%2B5Fvw%40mail.gmail.com.

--

Joe Hicks
Product Manager, Google Core Developer

[Tony Aiuto]

Yes. Examples of how to do things with the current tools are important.

Equally important are the things you want to do, but can't, along with an example of the way you might want to declare it.

For the most part, rules should expect to always wrap http_load with a "maybe" (catch2 does explicitly what the maybe rule does).

Perhaps the "maybe" part should be incorporated into *_archive. Something like

• if a second instance loads the same version, that is a silent success
• if a second instance would load a different version
• and both instances were in deps() functions of other repositories, fail and tell me the call stack to each
• the resolution is to add an explicit *_archive to your project WORKSPACE
  
• you get to pick the version you want, not the rules you are using
• when the dep to a different version appears, log it as "rules_x asks for rules_y@2.7 but we substituted rules_y@3.0"
• If rules_x MUST have rules_y@2.7, it should load it under a different name. E.g.  "rules_x_pinned_rules_y" and use that throughout it's BUILD tree.  The main WORKSPACE could still override that if it really wants.
• You should be able to run Bazel in a way that logs all the dep graph such that it is easy to find them and explicitly make them part of your WORKSPACE, eliminating the need to call any deps functions.
• Registering toolchains is a different problem.
  

But that is just one idea.  We could also just go full recursive workspace.  Then you would never call transitive deps from your own workspace, you would only load your direct deps.  You still have the diamond dependency problems of above. They might be solved in a similar way. 

And there are many other dimensions.

• Some users need to vendor in all the dependencies
• some need to untar them too
• others are willing to just use a tar file
• Some users publish their own internal packages to their own package servers
• sometimes they need strange authentication to download packages
• sometimes they copy packages from the canonical repository, let's say maven central to their own. They want to do generic rename like https://foo.bar.com/some/path/to/package.tgz  to https://archive.mycorp.com/packages/package.tgz.
• should we make blob download external to bazel so these issues are easy to work on without mucking around in the core of bazel?

[Tony Aiuto]

That sounds like interesting work. Is there anything to share?  Is it based on rules_license and/or rules_jvm_external or did you take a different approach?

Regardless of those questions, my point was that we need to define Bazel providers for package information that are upstream repository agnostic.  We can not end up with MavanPackageInfo, and PyPyPackageInfo, ...  You want to make your scheme work for C++, so you are probably now seeing that pom files as the source of truth won't fly in that community.

On Thu, Sep 3, 2020 at 9:02 PM Rehana Tabassum <rtab...@vmware.com> wrote:

Hi Tony,

 

As Mark mentioned already, we (Daniel Machlab also joined recently) are working on this scope. Our target is to come up with a license compliance mechanism that can be used at Bazel build time to produce an OSS manifest accurately.

 

The prototype we are working on is for Java components. It generates an OSS manifest file for rpm packaging during a Bazel build run. In VMware, to manage maven external dependencies for java projects, we use the ruleset rules_jvm_external. And the maven artifacts are downloaded from our internal artifactory.

 

We defined a rule that takes any pkg_rpm_ex target of a java project and generates an OSS manifest file with the aggregated external maven dependency jars and license info.

The rule uses an aspect, that we defined, to analyze the dependencies transitively for the particular target rpm and gather maven coordinates with artifact details. Since the maven artifacts are downloaded from our internal artifactory, we expect to store the license metadata info with the artifacts. Currently, the prototype extracts the license info from pom.xml next to the jars. We plan to expand this prototype to C++ and are also looking at how this work fits into our internal Open Source License approval process.

 

Thanks,

Rehana