Musings about external repository support

[Lukács T. Berki]

Hey there,

In the wake of a virtual water cooler discussion about external repository support, I think it's worth summarizing the decisions that went into supporting them in Bazel.

The decision was mostly taken because Bazel has its roots in a monorepo world, but most of the world does not work that way and a large majority of projects do have dependencies that need to be fetched. We thought that the large chunk of complexity it adds is a good price for being able to start using Bazel in a non-monorepo project.

There were two alternatives we seriously thought about:

1. Declaring that Bazel is a monorepo tool. This would have resulted in a barrier of entry that's too high or, even worse, people working around Bazel not being able to access source code outside the main source tree by various custom ways.
   
   
2. Delegating fetching external repositories to a separate tool. We thought that this would result in a large amount of friction since one would have to manually call it before calling Bazel and that it went against the "correct" part of our tagline, since then Bazel would not have recognized changes to e.g. the WORKSPACE file. 

Then, in the last five years, the API surface for external repositories became very complex and they started to be used for things we did not think about when we started out; they have become the default way to inject non-hermetic information into the build.

Most notably, in some cases, we started doing transitive dependency resolution, thereby turning Bazel into an accidental multi-language package manager, which is a hard problem even when we constrain ourselves to a single programming language.

I was about to add an "if I had to start again" paragraph, but then I realized that I don't know what I would do differently, even with the knowledge I have today (other than not putting external repositories at the $EXECROOT/external, which is prone to conflicts with the source tree, but that's only a small slice of the problem)

Appendix: the initial design doc is here (it also has a version from 2014, which we wrote in preparation for the initial code drop)

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[László Csomor]

That's a great summary Lukács.

This topic is dear to me. I wrote an internal doc about it, but I no longer work at Google so I lost access. Could that be published?

If we started over, I'd delegate package management to an external tool. Divide and conquer. The silver lining is, it's still possible to go that way.

--

László Csomor | las...@engflow.com | EngFlow GmbH

Fischerweg 51, 82194 Gröbenzell, Germany | Geschäftsführer (CEO): Ulf Adams

https://www.engflow.com

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAOu%2B0LUPHx7ofrY%3DoHmKNjn2UpU3fp9rgeQ%2Bkez%2Bbw_TTvm7yg%40mail.gmail.com.

[Tobias Werth]

I fully agree with Laszlo. Splitting up the task of (third party) dependency management from actual building is a great simplification.

If we go down that route, we have to be careful though to think how existing use cases can be migrated.

[Philipp Wollermann]

I'm also on board with the external package manager / virtual monorepo manager tool idea. It would be a useful thing independently of Bazel, too.

Maybe it's time to write Repozelisk? ^^

Cheers,

Philipp

--

Philipp Wollermann | Software Engineer | phi...@google.com
Google Germany GmbH | Erika-Mann-Straße 33 | 80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

[Lukács T. Berki]

On Wed, May 13, 2020 at 9:30 AM László Csomor <las...@engflow.com> wrote:

That's a great summary Lukács.

This topic is dear to me. I wrote an internal doc about it, but I no longer work at Google so I lost access. Could that be published?

Here it is:

https://docs.google.com/document/d/1ldbIa5n1goxcRTbrnLklJJuy7aIO_Qc3vPFmZPc8tDU/edit#

[Philipp Wollermann]

By the way, we discussed our problems with dependency management and thoughts with the people who built the Go versioning design and package manager.

Our conclusion was that we should probably just profit from their research into this complex topic and adopt their design principles like using the Minimal Version Selection algorithm instead of the classic "SAT solver and Lockfiles" approach for our own tooling as well. However, we didn't have time to actually build something yet.

--

Philipp Wollermann | Software Engineer | phi...@google.com
Google Germany GmbH | Erika-Mann-Straße 33 | 80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

[Irina Chernushina]

>Our conclusion was that we should probably just profit from their research into this complex topic and adopt their design principles like using the Minimal Version Selection algorithm instead of the classic "SAT solver and Lockfiles" approach for our own tooling as well. However, we didn't have time to actually build something yet.

Philipp, I would suggest that we continue to work on this variant and we may identify the parts that the external community could help us with (Alex Eagle?). Using Go team experience sounds great.

Linking my document and slides: go/bazel-pm-slides, Bazel Package Manager

The external package manager solution is also possible, but for that we still need to solve the same problem of enumerating existing rule sets and finding the combinations of mutually consistent versions.

The monorepo solution requires... the same:) finding the combinations of mutually consistent versions of existing software. Also, I am a believer that mono repo is impossible for Bazel world with so many different participants from different organizations. It is not possible to have a strict policy and move fast with development - at the scale of GitHub individual players must be as independent as possible.

Cheers, Irina

Irina Chernushina |

Software Developer |

ich...@google.com |

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CA%2BAhZoi0cvLHXg%3DgZQ%2B6aNNu8FpQbZ7mAzZbLsteL421TTe%2Bjg%40mail.gmail.com.

[Jakob Buchgraber]

Heya Lukacs,

It's important to distinguish between repository_rule, external repositories and the management of external repositories.

repository_rule

It's a good idea to use repository_rule as the Bazel equivalent of autoconf's ./configure. It allows Bazel to inspect its environment

and put a hash on it. It's also easier to write correct and portable Starlark than Shell. The API does have some shortcomings

though:

• repository_ctx.execute(...) allows non-zero exit codes.
• Referencing a label causes the rule to restart.
• repository_ctx.execute(...) does not allow declaring inputs and outputs. This would allow
  incrementality during restarts and make it easier to support remote execution.
• For remote builds Bazel needs to be able to inspect the remote environment just like the local environment for local builds.
• repository_ctx.path is difficult to make work remotely.
• repository_ctx.download should be separate.
  
• exec_result.return_code is misnamed. It's really annoying.

External Repositories

I'd argue that Bazel should continue to be responsible for fetching external repositories. A build typically only needs a subset

of its dependencies. It would be a waste to fetch all of them eagerly. A remote build likely does not need to stage any dependencies

on the host.

Go's approach with .mod and .sum files seems like it would fit Bazel very well. Klaus had started work on a conceptually similar solution

with WORKSPACE.resolved.

Management of External Repositories

I understand that's where most of the pain comes from. I agree with the sentiment on this thread that separating this from Bazel

is likely to be a winning strategy.

Best,

Jakob

Jakob Buchgraber

Software Engineer

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

On Wed, May 13, 2020 at 9:17 AM 'Lukács T. Berki' via bazel-dev <baze...@googlegroups.com> wrote:

--

[László Csomor]

> I'd argue that Bazel should continue to be responsible for fetching external repositories. A build typically only needs a subset

> of its dependencies. It would be a waste to fetch all of them eagerly.

I agree with the second half, but not the first. I could envision a lazy filesystem that does the fetching.

> A remote build likely does not need to stage any dependencies on the host.

Yes, that's an important point. The remote execution service could be connected to the same fetching service as Bazel. And Bazel wouldn't need to stage the dependencies locally, because CAS.FindMissingBlobs would respond that all blobs are available.

--

László Csomor | las...@engflow.com | EngFlow GmbH

Fischerweg 51, 82194 Gröbenzell, Germany | Geschäftsführer (CEO): Ulf Adams

https://www.engflow.com

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAGQ4vn31hO-Ev5LwPshnQfOQEZw3ZthPx6E5C3OQFoxGtw6kkA%40mail.gmail.com.

[Jon Brandvein]

Doesn't Go's packaging system very strongly assume backwards compatibility guarantees that we cannot easily require in the Bazel community?

In any case, using a new version resolution algorithm is attempting to solve the problem of how to specify and resolve dependencies. I agree it'd be better if Bazel weren't in the business of trying to do that. It's fine for us to have labels like @foo//pkg:bar, but I wish we could just ask someone else to fetch @foo.

To unsubscribe from this group and stop receiving emails from it, send an email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAOu%2B0LUPHx7ofrY%3DoHmKNjn2UpU3fp9rgeQ%2Bkez%2Bbw_TTvm7yg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to baze...@googlegroups.com.

[Ulf Adams]

You might also want to consider this critique of the go versioning system:

https://sdboyer.io/vgo/intro/

Personally, I think Bazel should continue to pull the dependencies - this allows tighter integration in a way that allows holding some desirable properties. Lazy fetching is one of those. It might be possible to build an integration protocol with another tool that allows lazy fetches, but it seems shaky to me. A lazy file system is probably out of the question, as that would not work for a significant subset of users due to security policy reasons.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/c94a93d2-ba33-4c6e-8184-ce400480108e%40googlegroups.com.

[Jakob Buchgraber]

I would argue that before thinking about versioning and dependency resolution the Bazel team should think about the interfaces and mechanisms

of Bazel to pull in dependencies and inspect its build environment. That's strictly separate from versioning and would already be a huge win.

I think it certainly makes sense for a tool that pulls in Go packages to use MVS. However, it's easy to see that this won't generally work because

MVS assumes semantic versioning. Many ecosystems don't use semantic versioning. So you will likely need to have ecosystem specific logic to

do dependency resolution and having to define contracts between the ecosystems in order to make multi language builds work. All this is probably

best done in a tool separate from Bazel.

 

Jakob Buchgraber

Software Engineer

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

On Wed, May 13, 2020 at 12:20 PM 'Philipp Wollermann' via bazel-dev <baze...@googlegroups.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CA%2BAhZoi0cvLHXg%3DgZQ%2B6aNNu8FpQbZ7mAzZbLsteL421TTe%2Bjg%40mail.gmail.com.

[Tony Aiuto]

+Steve Billings, +Mark Zeren +Joe Hicks 

We are already lost in the weeds of "this technique won't work because of that".  Let's restart the discussion.

I agree with Lukács that our current implementation has problems. I think we all do. I'll go a step further and state that I think dependency management is probably the single biggest usability improvement we can make to Bazel to improve adoption.

I also think that many of us believe we can go and write a better design or evaluate someone else's design - right now.  I don't believe that. After multiple discussions with users and having read many open issues I have come to the conclusion that a few simple changes won't improve the product measurably. This is an initiative that demands input and effort from beyond the Bazel team.

• we need to gather real world use cases and their relative importance
• we need to synthesize that up to requirments for a replacement
• we need to plan and deliver implementations in a way that the right teams, including users, can contribute to each part.

One reason I explicitly added some non-team users to this thread is because I do not believe the Bazel team (which is mostly Googlers) should attempt this alone is that, as Lukács hinted at, we are a monorepo shop and our requirements gathering process is tainted by that. Nor do I believe that the Google Bazel team has enough engineering slack to pre-commit to taking this all on. Ultimately, actually getting things done involves finding the people to do the work. We need all the stakeholders to share the work.

I propose that we form a working group for this effort to shepherd it from requirements gathering through implementation and finally to maintaining the ongoing future maintenance.  

• formsl WG structure: chairperson, distinct mailing list for all communications, regular meetings (can do public VC even)
• actively solicit membership in the WG to users large enough to have interesting needs and/or capable of contributing to the work (joehicks@ can certainly help there)
• expected deliverables (3 month time frame - not to linger for a year)
1. requirements from stakeholders
2. develop and solicit design proposals
3. manage design reviews (note: not quite the same as *do* design reviews, that may be delegated)
4. selection of designs against resource availability to implement. It does us no good to design something that we don't have the engineers to build
• management of the work
• Negotiating who gets to build each piece,
• Sitting on the schedule
• Coordinate feature drops into Bazel releases with promise of active evaluation/testing by user stakeholders

I do believe that, between Sven and I, we can afford to make someone available to chair(1) the group and lead it through at least the requirements gathering phase. What we build and who builds it is another round of negotiations. We can face those later.  If the group produces good designs and we all believe they can be staffed, great - keep going. If it turns into bike-shedding and political infighting for features, we shut it down and design and build from scratch.

1) To answer the inevitable question about why the chair has to be from Google - I say that because I think that ultimately Google will be doing a lot of the work, or stuck with maintaining it, so I want to head off a group creating a wishlist of features that we don't have the resources to build.

On Fri, May 29, 2020 at 7:52 AM 'Jakob Buchgraber' via bazel-dev <baze...@googlegroups.com> wrote:

I would argue that before thinking about versioning and dependency resolution the Bazel team should think about the interfaces and mechanisms

of Bazel to pull in dependencies and inspect its build environment. That's strictly separate from versioning and would already be a huge win.

I think it certainly makes sense for a tool that pulls in Go packages to use MVS. However, it's easy to see that this won't generally work because

MVS assumes semantic versioning. Many ecosystems don't use semantic versioning. So you will likely need to have ecosystem specific logic to

do dependency resolution and having to define contracts between the ecosystems in order to make multi language builds work. All this is probably

best done in a tool separate from Bazel.

 

Jakob Buchgraber

Software Engineer

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAGQ4vn1VgVkWUdg7fB9FAqhj-AOzXqYjinfKRGnzfu0XCOaPDw%40mail.gmail.com.

[Joe Hicks]

Thanks Tony for reframing the conversation. I support creating a working group for this. Who do you see as being the chair and when would you like to start this working group?

Thanks again,

Joe

--

Joe Hicks
Product Manager, Google Core Developer

[Philipp Wollermann]

Hello,

Thank you all for your input. I agree that we need to listen to and gather input from users and stakeholders.

I'd like to propose a different approach that should achieve this in a simpler way.

First, I will write a public design doc that proposes a solution for Bazel's handling of external dependencies.

It will be based on my understanding of the requirements and all the docs and stories that have been shared with me in the past months.

We can then discuss this design on the document itself and in the newly created Slack channel #external-deps in Bazel's slack.

Feel free to already join there and let me know about your thoughts and requirements, so that I can address them as early as possible.

I'll make sure that I'm online and available in Slack during working hours.

I would like to use asynchronous communication (email, chat, comments on docs) rather than meetings / video conferencing to work on this project, as it will be more accessible for people who want to follow our progress and leaves a better trail of thoughts and work done.

When the design has been discussed appropriately, we will build the solution for this as an open-source tool on GitHub in the bazelbuild organization.

If you would like to participate in the development there, please let me know and I'll see how we can organize this when we get to that phase.

I'm also happy to take issues and PRs that fit the design constraints there.

Cheers,

Philipp

[Tony Aiuto]

SGTM.  My only strong suggestion right now is to make a new mailing list rather than bazel-discuss+X.  That makes it much easier for people to filter away bazel-discuss but be actively involved in this discussion.

[Mark Zeren]

Document based in a separate mailing list SGTM. I would appreciate having an interactive chat forum. If we are going to use #external-deps, what are the plans for paying for or otherwise unlocking history? 

To unsubscribe from this group and stop receiving emails from it, send an email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAOu%2B0LUPHx7ofrY%3DoHmKNjn2UpU3fp9rgeQ%2Bkez%2Bbw_TTvm7yg%40mail.gmail.com.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--

Philipp Wollermann | Software Engineer | phi...@google.com
Google Germany GmbH | Erika-Mann-Straße 33 | 80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CA%2BAhZoi0cvLHXg%3DgZQ%2B6aNNu8FpQbZ7mAzZbLsteL421TTe%2Bjg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAGQ4vn1VgVkWUdg7fB9FAqhj-AOzXqYjinfKRGnzfu0XCOaPDw%40mail.gmail.com.

--

Joe Hicks
Product Manager, Google Core Developer

[Mark Zeren]

Tony,

 

Was this mailing list ever created?

 

Philipp,

 

Have you had a chance to draft the document that you suggested?

 

At VMware Rehana Tabassum and I have been investigating "rules_license"-like support for one of our internal builds. We are interested in how that might fit into the evolution of Bazel dependency management.

 

Can we use the upcoming BazelCon (CFP ends a week from today!) as a forcing function to get this discussion started?

 

Mark

 

Jakob Buchgraber

Software Engineer

[Joe Hicks]

Hi Mark,

This may be what you are looking for: https://groups.google.com/a/bazel.build/g/external-deps .

Also, feel free to join the Slack channel #external-deps in Bazel's slack.

Let me know if you need more info.

Thanks,

Joe

[Xudong Yang]

Hey Mark,

We now have a three-person strike team working on the external deps project (Philipp, Yun, and myself). We're currently still in the phase of gathering information and exploring the solution space -- there are a lot of questions to answer :) Once things settle down a bit more, we'll start doing interviews with community members and potentially send out some questionnaires.

And yes, I think it would be a good idea to use this year's BazelCon to share some of the findings and early ideas that we have, and to get the community involved. Hopefully by October/November we'll have more concrete stuff to share.

Xudong

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAMDFopgxcEi72OBTUP1h_yqqs3gbkmStGiO%2B3%2BfiexRVd%2B5Fvw%40mail.gmail.com.

[Joe Hicks]

+extern...@bazel.build

[Tony Aiuto]

Hi Mark.

I just wanted to close the loop. Others answered before I got to my mail, so you see we did start the discussion group.

And, as Xudong said, we devoted some people to work on this area specifically.

You mentioned "how this might fit into the evolution of Bazel dependency management". I think we are at a point

where the evolution is whatever the community decides. We need to collect user journeys and specific needs

so we can get a comprehensive view of what people need.  Posting them to the list for discussion is one way

to start.

Tony

[Xavier Bonaventura]

I just wanted to write that I am also interested in the topic and indeed working on it. I created a small example trying to show how people should deal with third party dependencies in bazel, the goal is to be a small example but scalable for large projects.

https://github.com/limdor/bazel-examples/tree/master/third_party_dependencies#bazel-with-third-party-dependencies

Here you can see an example of a decision that to me (and I believe for more bazel users) it is not clear what is the best practice:
https://github.com/limdor/bazel-examples/blob/master/third_party_dependencies/third_party/catch2/catch2.bzl#L8

And here another thought regarding loading transitive dependencies:

https://github.com/limdor/bazel-examples/blob/master/third_party_dependencies/third_party/extra_dependencies.bzl#L4

I started this example because I think it would be very beneficial to have guidelines together with examples. I did not find anything similar but of course I could have miss it. I am still working on it but I plan to have something in good shape in the following months.

Let's keep in touch

Missatge de 'Tony Aiuto' via bazel-dev <baze...@googlegroups.com> del dia dv., 21 d’ag. 2020 a les 6:17:

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CACpMnutv61J24Wn-xLZ%2BxeakmMtPupoEZhWnu2uWA5JULiSxGQ%40mail.gmail.com.

[Tony Aiuto]

Yes. Examples of how to do things with the current tools are important.

Equally important are the things you want to do, but can't, along with an example of the way you might want to declare it.

For the most part, rules should expect to always wrap http_load with a "maybe" (catch2 does explicitly what the maybe rule does).

Perhaps the "maybe" part should be incorporated into *_archive. Something like

• if a second instance loads the same version, that is a silent success
• if a second instance would load a different version
• and both instances were in deps() functions of other repositories, fail and tell me the call stack to each
• the resolution is to add an explicit *_archive to your project WORKSPACE
  
• you get to pick the version you want, not the rules you are using
• when the dep to a different version appears, log it as "rules_x asks for rules_y@2.7 but we substituted rules_y@3.0"
• If rules_x MUST have rules_y@2.7, it should load it under a different name. E.g.  "rules_x_pinned_rules_y" and use that throughout it's BUILD tree.  The main WORKSPACE could still override that if it really wants.
• You should be able to run Bazel in a way that logs all the dep graph such that it is easy to find them and explicitly make them part of your WORKSPACE, eliminating the need to call any deps functions.
• Registering toolchains is a different problem.
  

But that is just one idea.  We could also just go full recursive workspace.  Then you would never call transitive deps from your own workspace, you would only load your direct deps.  You still have the diamond dependency problems of above. They might be solved in a similar way. 

And there are many other dimensions.

• Some users need to vendor in all the dependencies
• some need to untar them too
• others are willing to just use a tar file
• Some users publish their own internal packages to their own package servers
• sometimes they need strange authentication to download packages
• sometimes they copy packages from the canonical repository, let's say maven central to their own. They want to do generic rename like https://foo.bar.com/some/path/to/package.tgz  to https://archive.mycorp.com/packages/package.tgz.
• should we make blob download external to bazel so these issues are easy to work on without mucking around in the core of bazel?

[Rehana Tabassum]

Hi Tony,

 

As Mark mentioned already, we (Daniel Machlab also joined recently) are working on this scope. Our target is to come up with a license compliance mechanism that can be used at Bazel build time to produce an OSS manifest accurately.

 

The prototype we are working on is for Java components. It generates an OSS manifest file for rpm packaging during a Bazel build run. In VMware, to manage maven external dependencies for java projects, we use the ruleset rules_jvm_external. And the maven artifacts are downloaded from our internal artifactory.

 

We defined a rule that takes any pkg_rpm_ex target of a java project and generates an OSS manifest file with the aggregated external maven dependency jars and license info.

The rule uses an aspect, that we defined, to analyze the dependencies transitively for the particular target rpm and gather maven coordinates with artifact details. Since the maven artifacts are downloaded from our internal artifactory, we expect to store the license metadata info with the artifacts. Currently, the prototype extracts the license info from pom.xml next to the jars. We plan to expand this prototype to C++ and are also looking at how this work fits into our internal Open Source License approval process.

 

Thanks,

Rehana

[Tony Aiuto]

That sounds like interesting work. Is there anything to share?  Is it based on rules_license and/or rules_jvm_external or did you take a different approach?

Regardless of those questions, my point was that we need to define Bazel providers for package information that are upstream repository agnostic.  We can not end up with MavanPackageInfo, and PyPyPackageInfo, ...  You want to make your scheme work for C++, so you are probably now seeing that pom files as the source of truth won't fly in that community.

[Joe Hicks]

Hi everyone,

I just wanted to let you all know that we are having a Birds of a Feather session at BazelCon on external dependencies. Please sign up for BazelCon and select that BoF session.

"Join us virtually on November 12-13 for BazelCon 2020!  Register → http://goo.gle/bazelcon "

Thanks,

Joe