Is there an official repo for 3rd-party bazel rules?

[bre...@gmail.com]

Hi,

As a bazel user for several months, I really appreciate the work of the bazel team. Would like to express my thanks here first!

Based on my experience, the most difficult part of getting bazel started on your project is about importing those third party dependencies, to be specific - writing the bazel rules for them. Most of the 3rd-party libraries do not use bazel as its build system. To name a few: glog, opencv, gtest ...

I understand that ideally, the best solution to this would be having the libraries themselves to provide bazel build as an alternative to, say, CMake. However, this cannot be achieved in a short period of time. I am wondering whether instead there is already a repo that contains the bazel build rules for well-known libraries. It might be maintained by the community, with a WORKSPACE file, BUILD files for the libraries, and even some test rules. I think this can help more people adopt bazel quickly :)

Just my 2 cents. Thanks!

[Christian Rivasseau]

IMHO It would be a fantastic idea to start such a thing.

I found this once: it doesn't have that much but better than nothing https://github.com/mzhaom/trunk

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/64b6e291-a60e-4263-a398-72fd108c3bd4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christian Rivasseau
Co-founder and CTO @ Lefty
+33 6 67 35 26 74

[Thiago Farina]

Ming has one. Take a look at https://github.com/mzhaom/trunk if haven't yet.

Bazel has some rules for third_party (like gtest) as well.

https://github.com/bazelbuild/bazel/blob/master/third_party/BUILD

-- 

Thiago Farina

[bre...@gmail.com]

Thanks for the pointer to Ming's trunk repo. That is very helpful :)

One downside of not having the bazel build in the original libraries is that the build file can get stale without some one updating them together with the updates on the libraries themselves. An official repo hosting those build rules might attract enough people from the community to keep it up-to-date ~

[Markus Teufelberger]

I tried to start with something similar recently, however I'm not 100% sure how to properly implement this in practice.

There might be several rules needed for the same library if different versions should be supported (e.g. boost 1.60, boost 1.61 etc.). It seems like this would turn into something close to a package repository just for build rules, not for binaries (not unlike Arch Linux' AUR). If only one (the latest?) version of a library is to be supported, it might be of limited use to users.

I however still do agree that it would be great to have a central place where one can look for existing BUILD files to be used in new_git_repository() rules.
One open question is if it should also include custom Skylark rules (e.g. the "boost_library()" shorthand rule instead of the more verbose cc_library()) or rather depend only on native rules (or at most rules within the bazelbuild organization)?

I would propose a layout similar to the GOPATH/src/... layout with BUILD files which are named after a commit or tag in the respective repository. That way it would be easier to automatically test them as well as staying as flexible as possible (multiple versions supported and easy to contribute new files while it is always clear which commit is referred if an existing one needs to be changed). It would also always produce canonical locations for files so there is no discussion if OpenSSL belongs into the "crypto" or "security" category folder. Personally I would strongly prefer using no .bzl files unless they are really necessary. Also no git submodules of the actual source repositories.

About test rules:
Some Boost libraries for example contain test cases that are expected to fail with a certain error at compile time or run time. As far as I understand the cc_test() rule though, it is not really possible to tell Bazel about this. This really sucks when (semi-)automatically generating rules for all test files... so these might need some cc_test_run_fail() and cc_test_compile_fail() custom rules, be somehow manually excluded or wrapped in a genrule() of some sort (which really would hurt portability I'm afraid).
It should be possible to write a program to visualize which files in a repository are and which aren't used by rules within a BUILD file. This might be a useful helper tool to see if everything relevant is used and if for example tests are not exposed (yet) to Bazel.

[bre...@gmail.com]

I have similar concerns. Some of them, i.e. custom skylark rules, may be resolved by having another centralized repo - there probably already is one, but I am not aware of.

[Steren Giannini]

Hi,

Many Bazel users are indeed writing BUILD files for third party OSS libraries. 

Having an open repository to share them is a topic that came up a few times internally, and a few googlers are motivated to work on it, but nothing has started yet.

Not having to upstream the BUILD files would allow us to de-couple being able to use third party OSS libraries from Bazel adoption among those libraries. CocoaPods and TypeScript have used the same strategy to bootstrap their community: See the repo of pods and repo of type definitions.

Technically I share the opinion that this could be based on a git repo, at least to start with until we see the limits of such approach. But of course, we should agree on an implementation.

I'd like to schedule a meeting next week on this topic, please let me know if you'd like to to be part of it. I will in any case update this thread with the notes.

Steren

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4ed222b6-3162-48f2-b2cd-445636d23ce6%40googlegroups.com.

[Christian Rivasseau]

IMHO I had been thinking of starting such a thing, I believe there are 3 legs to this project:

First: a dedicated rules that mimics maven_jar and takes care of everything for me:

cpp_third_party_library("folly", "commit_hash") and then be able to depend cc_library on

deps = [

  "@folly"

]

It's not enough to have a git repo where bazel users are supposed to go copy and paste inspiration.

PS: 

When the open-source library has decided to provide BUILD files, the macro would just link to those.

When they didn't, it must be a .BUILD file that's contributed directly by us.

Second: a website that allows browsing of all existing projects (it may even be only a static website served out of git) and calls for 

Third: tons of contributed libraries :) It shall be easy to bootstrap the repo with all the existing rules from ming, tensorflow, etc...

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4ed222b6-3162-48f2-b2cd-445636d23ce6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CANHKq5Fkot-mVmfZ%2B%3DZKhxfOa5hscR5p8SLJXXhVqnQ%2BV8jPyg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[bre...@gmail.com]

Hi Steren,

Thank you for the update! Yeah it would be fantastic if there are googlers bootstrapping such effort! When I was at Google I talked with my colleagues about this as well, and I think there are already people doing this for internally-used third party libraries. Migrating those ones might be a good start :)

[bre...@gmail.com]

Really like the idea about having a site for searching for libraries. Something like creates.io (https://crates.io/crates/serde) is an example ~

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4ed222b6-3162-48f2-b2cd-445636d23ce6%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[David Chen]

Hello,

Having a website similar to Crates.io and CocoaPods that serves as a centralized place to publish and discover both BUILD files and Skylark rule sets, as well as the associated tooling for adding and managing such "packages" of third party libraries and rule sets to one's workspace, is exactly what we have in mind.

Looking forward to discussing more during next week's meeting.

Thanks,

David

[Steren Giannini]

Hi,

I am sharing an update:

We met a few weeks ago on this topic, you can read the meeting notes here. Feel free to add comments.

I created a repository that we will use to store Bazel configuration files and metadata for open source libraries: https://github.com/bazelbuild/bazel-packages 

David has started a design document outlining how to organize the repo, I will let him share it when it is ready for review.

Steren

[Doug Tangren]

Is the desired intent here to have a monolithic repository of source code or a registry of repositories which contain source code?

In the former case, that would be creating new problems that dont currently exist, required manual moderation to get new source code from external contributors in and making the repository that contains source code inherently more difficult to manage.

The latter class would be ideal ( eg crates.io) where publishing content and consuming content require no manual moderation and can evolve much more quickly as independent versioned components in the ecosystem.

I think what's actually needed in the 3rd party bazel ecosystem is discoverability. Is that the goal here?

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a201e520-c264-4282-9870-7f26c0152ca4%40googlegroups.com.

[Steren Giannini]

Hi,

The latter. You can find more details in the meeting notes.

The repository will only contain references to OSS libraries and their community-contributed Bazel configuration files.

Because the currently picked backend implementation is a git repo, it will require some sort minor moderation to accept a new package. Let's see if this becomes a bottleneck in the future, we can always change the backend.

Steren

On Mon, Jun 12, 2017 at 3:34 PM Doug Tangren <d.ta...@gmail.com> wrote:

Is the desired intent here to have a monolithic repository of source code or a registry of repositories which contain source code?

In the former case, that would be creating new problems that dont currently exist, required manual moderation to get new source code from external contributors in and making the repository that contains source code inherently more difficult to manage.

The latter class would be ideal ( eg crates.io) where publishing content and consuming content require no manual moderation and can evolve much more quickly as independent versioned components in the ecosystem.

I think what's actually needed in the 3rd party bazel ecosystem is discoverability. Is that the goal here?

On Jun 12, 2017 9:15 AM, "'Steren Giannini' via bazel-discuss" <bazel-...@googlegroups.com> wrote:

Hi,

I am sharing an update:

We met a few weeks ago on this topic, you can read the meeting notes here. Feel free to add comments.

I created a repository that we will use to store Bazel configuration files and metadata for open source libraries: https://github.com/bazelbuild/bazel-packages 

David has started a design document outlining how to organize the repo, I will let him share it when it is ready for review.

Steren

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a201e520-c264-4282-9870-7f26c0152ca4%40googlegroups.com.

[Markus Teufelberger]

Am Montag, 12. Juni 2017 15:34:53 UTC+2 schrieb Doug Tangren:

Is the desired intent here to have a monolithic repository of source code or a registry of repositories which contain source code?

As far as I am concerned, I'd rather have no source code beyond internal tools at all in there, only "recipes" to go from a certain published state of source code to a compiled binary.
Basically the answer to the question "What do I put in the 'build_file' section of new_http_archive or new_git_repository for version X of library Y?". Everything beyond that ("Does version X of library Y work with version foo of library bar or not?") is then up to the developer to decide and resolve. After all, developers might have a quite good understanding how their own code should be compiled, but probably don't want to investigate first for every single dependency they have how to compile that code.

What I also hope that this could help with is to give examples for how to write good BUILD files and how to test them.

[David Chen]

Hi,

To add to what Steren said, the plan is to ultimately have a service similar to Crates.io. The repository that Steren mentioned is not a monolithic repository of source code but more repository of "recipes." It contains metadata in addition to BUILD and WORKSPACE files for each package.

The doc that I will be sharing details the structure of this package repository and the schema for the metadata. We will also be having further discussions about the command line tools and build rules for adding packages as well as a web interface for browsing packages, similar to Crates.io and other services.

Thanks,

David

[Doug Tangren]

On Jun 12, 2017 3:59 PM, "'David Chen' via bazel-discuss" <bazel-...@googlegroups.com> wrote:

Hi,

To add to what Steren said, the plan is to ultimately have a service similar to Crates.io. The repository that Steren mentioned is not a monolithic repository of source code but more repository of "recipes." It contains metadata in addition to BUILD and WORKSPACE files for each package.

The doc that I will be sharing details the structure of this package repository and the schema for the metadata. We will also be having further discussions about the command line tools and build rules for adding packages as well as a web interface for browsing packages, similar to Crates.io and other services.

Awesome. Crates.io does an amazing job at both reducing friction for sharing work and discovering work. I look forward to a similar community in the bazel world. 

Thanks,

David

On Mon, Jun 12, 2017 at 12:48 PM Markus Teufelberger <markusteu...@gmail.com> wrote:

Am Montag, 12. Juni 2017 15:34:53 UTC+2 schrieb Doug Tangren:

Is the desired intent here to have a monolithic repository of source code or a registry of repositories which contain source code?

As far as I am concerned, I'd rather have no source code beyond internal tools at all in there, only "recipes" to go from a certain published state of source code to a compiled binary.
Basically the answer to the question "What do I put in the 'build_file' section of new_http_archive or new_git_repository for version X of library Y?". Everything beyond that ("Does version X of library Y work with version foo of library bar or not?") is then up to the developer to decide and resolve. After all, developers might have a quite good understanding how their own code should be compiled, but probably don't want to investigate first for every single dependency they have how to compile that code.

What I also hope that this could help with is to give examples for how to write good BUILD files and how to test them.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAHb-%3DNLHqx9u-R%3DcEvu%2BK9uNegzKoBNze%3D3Yj379TzTPt8Vrng%40mail.gmail.com.

[alpha....@gmail.com]

Hi all,

I think this is a great effort!

Regarding this document: https://docs.google.com/document/d/1VuoIsaqG1PwlncvMRY-fcW7e8lsY0SkEpaBpXZnZRIE/view#heading=h.t47ou3i3pl89 do we expect to have continuous builds to verify the build rules will continue to work?

Alpha

On Monday, June 12, 2017 at 7:13:36 PM UTC-4, Doug Tangren wrote:

On Jun 12, 2017 3:59 PM, "'David Chen' via bazel-discuss" <bazel-...@googlegroups.com> wrote:

Hi,

To add to what Steren said, the plan is to ultimately have a service similar to Crates.io. The repository that Steren mentioned is not a monolithic repository of source code but more repository of "recipes." It contains metadata in addition to BUILD and WORKSPACE files for each package.

The doc that I will be sharing details the structure of this package repository and the schema for the metadata. We will also be having further discussions about the command line tools and build rules for adding packages as well as a web interface for browsing packages, similar to Crates.io and other services.

Awesome. Crates.io does an amazing job at both reducing friction for sharing work and discovering work. I look forward to a similar community in the bazel world. 

Thanks,

David

On Mon, Jun 12, 2017 at 12:48 PM Markus Teufelberger <markusteu...@gmail.com> wrote:

Am Montag, 12. Juni 2017 15:34:53 UTC+2 schrieb Doug Tangren:

Is the desired intent here to have a monolithic repository of source code or a registry of repositories which contain source code?

As far as I am concerned, I'd rather have no source code beyond internal tools at all in there, only "recipes" to go from a certain published state of source code to a compiled binary.
Basically the answer to the question "What do I put in the 'build_file' section of new_http_archive or new_git_repository for version X of library Y?". Everything beyond that ("Does version X of library Y work with version foo of library bar or not?") is then up to the developer to decide and resolve. After all, developers might have a quite good understanding how their own code should be compiled, but probably don't want to investigate first for every single dependency they have how to compile that code.

What I also hope that this could help with is to give examples for how to write good BUILD files and how to test them.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[Damien Martin-Guillerez]

Yes we need to

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/875a46bc-8b72-4e51-abb0-8f6388ef389a%40googlegroups.com.

[bre...@gmail.com]

Feel very excited to see this happening. Having something similar to crates.io can hugely improve the efficiency of software development in Bazel community! Thank you :)

On Monday, June 12, 2017 at 6:34:53 AM UTC-7, Doug Tangren wrote:
> Is the desired intent here to have a monolithic repository of source code or a registry of repositories which contain source code?
>
>
> In the former case, that would be creating new problems that dont currently exist, required manual moderation to get new source code from external contributors in and making the repository that contains source code inherently more difficult to manage.
>
>
> The latter class would be ideal ( eg crates.io) where publishing content and consuming content require no manual moderation and can evolve much more quickly as independent versioned components in the ecosystem.
>
>
> I think what's actually needed in the 3rd party bazel ecosystem is discoverability. Is that the goal here?
>
>
>
>
> On Jun 12, 2017 9:15 AM, "'Steren Giannini' via bazel-discuss" <bazel-...@googlegroups.com> wrote:
>
>
> Hi,
> I am sharing an update:
>
>
>
> We met a few weeks ago on this topic, you can read the meeting notes here. Feel free to add comments.
>
>
> I created a repository that we will use to store Bazel configuration files and metadata for open source libraries: https://github.com/bazelbuild/bazel-packages 
> David has started a design document outlining how to organize the repo, I will let him share it when it is ready for review.
>
>
>
>
>
> Steren
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[Alpha Lam]

Totally agree. I'll be happy to contribute build rules for some c libraries I'm using too.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4eb2624e-7d58-42f9-9fa7-6be2f84cc32a%40googlegroups.com.

[steven....@airbnb.com]

> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4eb2624e-7d58-42f9-9fa7-6be2f84cc32a%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.

Are there any further updates on this collection for build files for open-source projects?

[Laurent Le Brun]

Thanks for bringing that up! I don't think there has been any update.
Some people who were involved are no longer active on Bazel.

I've just had a look at the meeting notes
(https://docs.google.com/document/d/1VuoIsaqG1PwlncvMRY-fcW7e8lsY0SkEpaBpXZnZRIE/view?pli=1).
It seems like the next step is to agree on the repository structure
and add an example.

> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/872a8ca5-e959-4c79-a0b8-cad6cb51cd62%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--

Laurent

[Laurent Le Brun]

+klaus

Any recommendation on how we should organize the repository?

[Klaus Aehlig]

On Mon, Sep 24, 2018 at 03:43:56PM +0200, Laurent Le Brun wrote:
> +klaus
>
> Any recommendation on how we should organize the repository?

I don't have any specific recommendations on how to organize the
repository, and, to be honest, I wonder if time is better invested in
improving (as in: "adding") bazel's support for building against
precompiled libraries (i.e., the "I have library together with its
header files already installed on my system; just build against that").
Why should every library be able to build with every build system anyway?

In general, when attempting to start such a collection of BUILD files,
I would look at existing such examples, like the FreeBSD ports tree
(source: https://svnweb.freebsd.org/ports/,
handbook: https://www.freebsd.org/doc/en/books/porters-handbook/).

However, for bazel specific, there are several things to keep in mind.

- Bazel requests that all fetch-like actions be declared (directly or
indirectly) in the WORKSPACE file; however, once such a collection grows,
you don't want even read all the definitions, and, instead only discover
what is actually needed. Recursive WORKSPACE files are on the agenda,
but are not implemented yet. There are ways to work around this for the
time being, but do we want such a pattern to establish?

- The problem that we might several versions of a library in parallel
was already mentioned; renaming of repositories should be good enough
to work now (as a library wants to refer to a different library as
@foo//, not as @foo_v1.5//). For the repository design that means that
somehow, we need to be able to have instructions for several versions
of the same libary available in parallel (version directories? switch
statement?)

- Sooner or later, you will run into the issue of fetch/extract/patch
dependencies (i.e., situations where you first have to build a tool
before you can even set up the sources of another library). By the
current design of bazel this is not allowed; our current approach is
to ignore the problem and pretend that all tools (the various version
control systems, find, sed, ...) are already installed on the ambient
system. What is our long-term story here?

- As far as I understand, the project is a big collection of receipies
for building libraries with bazel, regardless of the host system. But
not only languages need to discover their tool-chain, also individual
projectes/libaries need some configuration steps; that can probably
be part of the external repository rule, but we should keep it in mind.

I don't have good answers for these questions, but I think they should
be answered in design specifying how the repository is organized.

Regards,
Klaus

--
Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

[Laurent Le Brun]

Some projects are maintaining a collection of BUILD files for their dependencies, e.g.

  https://github.com/tensorflow/tensorflow/tree/master/third_party

  https://github.com/ApolloAuto/apollo/tree/master/third_party

Any solution that helps share the files (and make easy for other users to use them as dependencies) would be valuable.

--

Laurent

[Philipp Wollermann]

I discussed this topic with Klaus before he sent his mail and asked him to send it like that, because he makes some important points and also shows that there's an interesting split in how people want to treat dependencies in their builds.

Basically, there are two approaches to dependency management:

#1: Bazel should be "a good citizen" in your respective operating system / environment. You should be able to build against a libxml or Boost that your distribution ships and installs in /usr/lib and /usr/include. It should be able to use an installed OpenJDK or Android SDK on your system, a .NET Framework or Platform SDK on Windows, ... instead of requiring you to download these all in various versions via external repositories for every project that you want to build. Sometimes this might not even be possible due to licensing restrictions that prohibit automatic downloading and mirroring. Sometimes it would be just super annoying, because these things can be multiple gigabytes in size.

#2: Bazel should be able to build your software without relying on stuff being installed in the correct version in your system. It should be able to just build all your dependencies, if you use a mono-repo or a vendoring approach (e.g. have your own private copy of libxml with a defined version in your "third_party" folder or referenced via your WORKSPACE file). For this use case, it would be very helpful to have a community-maintained collection of recipes how to depend on popular third party libraries. I think that this is especially useful for C++, because there's still no established package manager that we could wrap for that language. Java has Maven (and we support that quite well via https://github.com/johnynek/bazel-deps), Python has PyPi (which we absolutely should support better and in a similar fashion to what bazel_deps offers for Java).

I believe both use cases are totally valid, should be supported by Bazel and pursued by people interested in them. :)

The bazel-packages repo would cater to use case #2. That's great, because due to its roots in Google's monorepo world, Bazel also tends to support #2 better at the moment. Laurent pointed out two good examples that we could follow - let's get started, accept PRs and see where this goes?

Cheers,

Philipp

--

Philipp Wollermann
Software Engineer
phi...@google.com

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.