External binary dependencies and multiple versions

[ittai zeidman]

Hi,

This is somewhat of a best practices questions.

Do you have one target for every external maven_jar dependency you use in your codebase?

If you migrated from maven/gradle how did you handle the conflicts? highest? manually? something else?

If you have one target how do you do upgrades of a library? Do you just trust the build?

@bazel team,

Does google just build everything from source? 

Thanks in advance,

Ittai

[Damien Martin-guillerez]

Mostly yes but we have distributed caching.

 

Thanks in advance,

Ittai

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a2cd666b-21ad-4f17-a81b-60f118364191%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ittai zeidman]

How does distributed caching relate to this? Sounds orthogonal to me but I'm probably missing something out

[Damien Martin-guillerez]

Well if you rebuild all artifact on all developer platform, it can be really expensive. We are building mostly everything from sources but each developer get cache hit when trying to rebuild it. This was just to say that if you don't have distributed caching, building from source might be prohibitive.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f0b12d4f-04a9-4097-9604-170e1a9c99f9%40googlegroups.com.

[ittai zeidman]

oh ok, got you.

I was actually referring to the usage of third party libraries.

Given "service-a" that depends on "lib1" and "lib2".

And "lib1" and "lib2" both depend on different versions of apache http-client, how do you handle it?

In maven we just had shortest paths resolution and it was implicit.

One solution is to decide you add the sources of apache http-client into your repo. Another solution is to use `maven_jar`.

Each has their pros and cons and I'm looking to hear what people use, how they manage the tradeoffs and the day to day (including upgrade of a library) 

[Damien Martin-guillerez]

In Google we have a one version policy so we use the http-client from our repository.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/5c762db2-170b-46f9-a077-0a391e471a9c%40googlegroups.com.

[ittai zeidman]

And if you want to upgrade that version to one which has breaking changes? Doesn't it take months? And does the build give you enough confidence? (Json parsing runtime behavior might be different and surface only on prod)

[Damien Martin-guillerez]

Yes test coverage get us enough confidence, however, updating the code of all dependent service might be tedious.

[ittai zeidman]

I can only imagine :) Thanks!
Any other points of view?
I'm currently leaning towards the one version policy but want to hear other opinions before deciding.

[Eric Zundel Ayers]

In Square's java monorepo we use a one version policy for external libraries, but it is not strict.   Also, we use Pants (not bazel), and pants integrates with Apache Ivy to pull in libraries from external repositories.  This is just informational on "how we solve this" in case this is a direction bazel might want to tackle managing transitive dependencies from external repos.

We explicitly manage the versions we depend on for the most part.  Pants has `managed_jar_libraries()` and `managed_jar_dependnecies()` targets that behave like `<dependencyManagement>` in maven.  that is, you specify the explicit version that you want and it overrides any versions specified in transitive dependencies.   Most targets want to use these pinned dependencies.  However, a target can opt-out of this and choose a different library buy depending on a `jar_library()` target explicitly, or a different set of `managed_jar_libraries()`.  If an external library isn't mentioned at all in `managed_jar_libraries()` or `managed_jar_dependencies()` then, it will fall back to the default Ivy conflict resolution conflict policy (for us, the highest version)

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOfK4wWXnmnOewXTX%3DokLw_QWDD7KHjceTF9wpcFzDFPS0%3DvJw%40mail.gmail.com.

[ittai zeidman]

Thanks but I really like the fact that Bazel doesn't bring Maven/Ivy dependency resolution, which I consider as part of their baggage.
Damien,
I don't remember but in Bazel as week I can exclude a transitive dependency a direct dependency of mine right?

[P. Oscar Boykin]

Ittai, 

I don't know if you have seen this:

https://github.com/johnynek/bazel-deps

You make a yaml file to declare your versions you want, then it builds a file of all the transitive versions and shas you need. It also creates BUILD files for all the third party dependencies.

But this solves the problem for us at Stripe. It has configurable policy for fixing versions (highest is default). So you can bump 1 version and find new transitive versions and their shas.

It's made it really nice to maintain our dependencies since we don't have to manually recompute all the targets and shas when we want to upgrade something.