Are there any quality examples of Java monorepo projects with Maven Jars using Bazel?

[Jason Banich]

I'm looking to create a Java monorepo-style project with Bazel as the build system and maven jars used throughout, but I'm having a hard time finding any good example projects.

I keep hitting questions like:

 * How many WORKSPACE files do I use? Just one or one under every project? (I think the answer is just one)

 * What is the best way to manage dependencies? If I need to use apache httpclient for example, that maven jar has 5 dependencies of which some have their own dependencies. Where do these go? A global workspace file? In separate workspace files? In the BUILD files? How do you manage these and their versions with the least amount of manual effort?

I feel like a solid example would answer a lot of these questions. 

Does anyone know of any good examples?

I'm thinking about creating a small monorepo with 2-3 simple java projects and asking for help with that. 

Thanks!

-Jason

[Kevin Gessner]

On Thu, Jun 28, 2018 at 7:07 PM, Jason Banich <jason....@mightyhive.com> wrote:

I'm looking to create a Java monorepo-style project with Bazel as the build system and maven jars used throughout, but I'm having a hard time finding any good example projects.

I don't have an open source example, but at Etsy we've been using Bazel for our internal Java+Scala monorepo for about a year.  All in all it's been a success, but we hit did these same questions!

 

I keep hitting questions like:

 * How many WORKSPACE files do I use? Just one or one under every project? (I think the answer is just one)

Just one, at the top level of our repository.

 

 * What is the best way to manage dependencies? If I need to use apache httpclient for example, that maven jar has 5 dependencies of which some have their own dependencies. Where do these go? A global workspace file? In separate workspace files? In the BUILD files? How do you manage these and their versions with the least amount of manual effort?

We use bazel-deps (https://github.com/johnynek/bazel-deps) for managing maven dependencies, and it is a godsend.  We attempted manual management and the generate_workspace tool, but both were unmaintainable.  Automatically resolving transitive deps is a must at any sort of scale, and the bazel-deps approach of a declarative dependency list and generated workspace and BUILD files works nicely.

bazel-deps isn't perfectly integrated with the rest of Bazel; it's a separate tool you have to run and check in the generated Bazel code.  We have a CI script to assert the resolved dependency tree is up-to-date with the declarations in dependencies.yaml, which has helped several times.

Let me know if you have other questions!  Enjoy your Bazeling.

-- Kevin

 

I feel like a solid example would answer a lot of these questions. 

Does anyone know of any good examples?

I'm thinking about creating a small monorepo with 2-3 simple java projects and asking for help with that. 

Thanks!

-Jason

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f3689976-af83-49d9-b516-75596d28bae7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jason Banich]

Awesome. Thanks Kevin, this helps a lot. 

Follow up: 

I am trying out bazel-deps, but I am running into an issue. 

Github gist showing all my files in question: https://gist.github.com/Jdban/04b83fa594467d295ad0a6f55cef0fa5

My simple dependencies.yaml seems to be working. When I run bazel-deps, the 3rdparty folder generates fine.

My WORKSPACE file is identical to what the bazel-deps repo specifies, so that should be fine. 

I'm not sure what to put in the deps section right now, but I have put the 3rdparty values from a run of bazel query //... in there (see gist) for the required libraries (slf4j and apache httpclient).

When I try to build my target, I get a build error that seems like it's having trouble with transitive dependencies required by apache httpclient (ThreadingBehavior.SAFE is from the httpcore library). Is my BUILD file correct or is something else wrong?

When I build this from pom.xml file, these are the only 2 dependencies that I need.

Error:

warning: unknown enum constant ThreadingBehavior.SAFE

  reason: class file for org.apache.http.annotation.ThreadingBehavior not found

java/com/example/library/snitch/DeadMansSnitch.java:45: error: cannot access AbstractHttpMessage

            client.execute(request); 

                  ^

  class file for org.apache.http.message.AbstractHttpMessage not found

 Thanks so much!

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[Ofer Bartal]

Hey Jason,

I have just what you need!

It's a monorepo called StartupOS and it answers all your questions (see below).

Enjoy and let me know how it's working out for ya.

On Fri, Jun 29, 2018 at 2:07 AM Jason Banich <jason....@mightyhive.com> wrote:

I'm looking to create a Java monorepo-style project with Bazel as the build system and maven jars used throughout, but I'm having a hard time finding any good example projects.

Here you go,

https://github.com/google/startup-os 

As for using Maven jars, this process should improve (should merge it into a single command), but currently,

1. Add your dependency:

https://github.com/google/startup-os/blob/master/commands.txt#L14

2. Regenerate all transitive deps:

https://github.com/google/startup-os/blob/master/commands.txt#L24

I keep hitting questions like:

 * How many WORKSPACE files do I use? Just one or one under every project? (I think the answer is just one)

Just one :)

Here it is - 

https://github.com/google/startup-os/blob/master/WORKSPACE

 

 * What is the best way to manage dependencies? If I need to use apache httpclient for example, that maven jar has 5 dependencies of which some have their own dependencies. Where do these go? A global workspace file? In separate workspace files? In the BUILD files? How do you manage these and their versions with the least amount of manual effort?

Here's an example for httpclient.

1. We added org.apache.httpcomponents:fluent-hc:4.5.5 through step #1 above.

That automatically added this to the yaml file:

  org.apache.httpcomponents:

    fluent-hc:

      lang: java

      version: "4.5.5"

See here:

https://github.com/google/startup-os/blob/master/dependencies.yaml#L113

2. We ran step #2 above. That generated the following entry for fluent-hc:

https://github.com/google/startup-https://github.com/google/startup-os/blob/master/third_party/maven/org/apache/httpcomponents/BUILD#L1

Also, it added all transitive dependencies, such as httpclient and httpcore (if those weren't already present):

https://github.com/google/startup-os/blob/master/third_party/maven/org/apache/httpcomponents/BUILD#L19

That file is auto-generated.

I feel like a solid example would answer a lot of these questions. 

Done! :-)

 

Does anyone know of any good examples?

I'm thinking about creating a small monorepo with 2-3 simple java projects and asking for help with that. 

Thanks!

Sure. Let me know what else you need help with :)

[Luc Perkins]

Another repo worth checking out is Heron (https://github.com/apache/incubator-heron). Mostly Java plus healthy doses of C++, Python, Protobuf, etc.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CA%2BndrjoGgCwqsY9snhXv%3DW30w91bKKB5032psDvL-4EGc8%2BrtA%40mail.gmail.com.