Bazel as Clojure build tool

[Kiril Videlov]

Hello,

I have been looking at the Bazel build tool for a multi-language mono-repo and I was wondering if anybody has tried it in the context of Clojure projects. I have only found the lein-monolith plugin which appears to address a similar use case sans the polyglot requirement. Do you think there is any value in attempting to build a rule set similar to the bazel rules for Scala?

Regards,

Kiril Videlov

[Nathan Fisher]

Hi Kiril,

I think you’d need to set your expectations. Faster builds are unlikely to be one of the benefits. A unified build tool is one.

I’ve seen a few similar attempts with Buck which was inspired by Bazel. They either generate lein projects on the fly or call into the Clojure compiler directly.

I think Clojure’s build model works against these tools in part because it’s a dynamic language, in part because it doesn’t implicitly forward declare functions, in part because (I believe) it does breadth first traversal if dependencies, and in part it’s non-trivial/inefficient to create a clean Clojure VM per dependency in the build graph.

Typical usage of these build tools is to have a build file per directory with one or more targets in each file. In order to get the speed that the build tool promises you need the ability to efficiently parse the ABI of each file and only build its dependents when the ABI has changed.

I’m not certain you can do that with Clojures compiler but someone else might be able to chime in with a way that you could.

Cheers,

Nathan

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

- sent from my mobile

[Colin Fleming]

Another tool that works well for polyglot JVM projects is Gradle. I use it to build Cursive, which has Java, Kotlin and AOT'ed Clojure. I released a mostly-unsupported plugin for it, and Andrew Oberstar and others have taken it over and done a lot of work on it - it now lives at https://github.com/gradle-clojure. It's pretty new but is under active development.

Gradle works very well for Cursive, I'm very happy with it.

On 29 January 2018 at 20:07, Nathan Fisher <nfi...@junctionbox.ca> wrote:

Hi Kiril,

I think you’d need to set your expectations. Faster builds are unlikely to be one of the benefits. A unified build tool is one.

I’ve seen a few similar attempts with Buck which was inspired by Bazel. They either generate lein projects on the fly or call into the Clojure compiler directly.

I think Clojure’s build model works against these tools in part because it’s a dynamic language, in part because it doesn’t implicitly forward declare functions, in part because (I believe) it does breadth first traversal if dependencies, and in part it’s non-trivial/inefficient to create a clean Clojure VM per dependency in the build graph.

Typical usage of these build tools is to have a build file per directory with one or more targets in each file. In order to get the speed that the build tool promises you need the ability to efficiently parse the ABI of each file and only build its dependents when the ABI has changed.

I’m not certain you can do that with Clojures compiler but someone else might be able to chime in with a way that you could.

Cheers,

Nathan

On Mon, 29 Jan 2018 at 13:16, Kiril Videlov <kiri...@runbox.no> wrote:

Hello,

I have been looking at the Bazel build tool for a multi-language mono-repo and I was wondering if anybody has tried it in the context of Clojure projects. I have only found the lein-monolith plugin which appears to address a similar use case sans the polyglot requirement. Do you think there is any value in attempting to build a rule set similar to the bazel rules for Scala?

Regards,

Kiril Videlov

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

- sent from my mobile

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

[Gregg Reynolds]

On Mon, Jan 29, 2018, 2:07 PM Nathan Fisher <nfi...@junctionbox.ca> wrote:

...

 In order to get the speed that the build tool promises you need the ability to efficiently parse the ABI of each file and only build its dependents when the ABI has changed.

Can you please elaborate on this a bit? I've been using Bazel for a project that involve C and Java, with cross-compiles, and ABI parsing has never come up. It just works.

I’m not certain you can do that with Clojures compiler but someone else might be able to chime in with a way that you could.

I like boot for clojure stuff, not sure Bazel offers any advantage. Clojurescript may be a different story, tho.

So far I cannot see a good reason to switch from boot to Bazel for 100% clojure projects. Mixed language - clojure, clojurescript, js, Java, kotlin, JNI,etc - may be a different story, still not sure.

Then there's remote caching: https://docs.bazel.build/versions/master/remote-caching.html. a hyuuuuge win for many scenarios, not so sure about clojure.

G

G

[Gregg Reynolds]

On Tue, Jan 30, 2018, 6:07 PM Colin Fleming <colin.ma...@gmail.com> wrote:

Another tool that works well for polyglot JVM projects is Gradle. I use it to build Cursive, which has Java, Kotlin and AOT'ed Clojure. I released a mostly-unsupported plugin for it, and Andrew Oberstar and others have taken it over and done a lot of work on it - it now lives at https://github.com/gradle-clojure. It's pretty new but is under active development.

Gradle works very well for Cursive, I'm very happy with it.

Don't wanna start a religious war, but I've worked with Gradle (and about 150 other build tools). If Gradle makes you happy Bazel will make you rapturous.

See e.g. https://www.google.com/amp/s/www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/amp/

G

On 29 January 2018 at 20:07, Nathan Fisher <nfi...@junctionbox.ca> wrote:

Hi Kiril,

I think you’d need to set your expectations. Faster builds are unlikely to be one of the benefits. A unified build tool is one.

I’ve seen a few similar attempts with Buck which was inspired by Bazel. They either generate lein projects on the fly or call into the Clojure compiler directly.

I think Clojure’s build model works against these tools in part because it’s a dynamic language, in part because it doesn’t implicitly forward declare functions, in part because (I believe) it does breadth first traversal if dependencies, and in part it’s non-trivial/inefficient to create a clean Clojure VM per dependency in the build graph.

Typical usage of these build tools is to have a build file per directory with one or more targets in each file. In order to get the speed that the build tool promises you need the ability to efficiently parse the ABI of each file and only build its dependents when the ABI has changed.

I’m not certain you can do that with Clojures compiler but someone else might be able to chime in with a way that you could.

Cheers,

Nathan

On Mon, 29 Jan 2018 at 13:16, Kiril Videlov <kiri...@runbox.no> wrote:

Hello,

I have been looking at the Bazel build tool for a multi-language mono-repo and I was wondering if anybody has tried it in the context of Clojure projects. I have only found the lein-monolith plugin which appears to address a similar use case sans the polyglot requirement. Do you think there is any value in attempting to build a rule set similar to the bazel rules for Scala?

Regards,

Kiril Videlov

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

- sent from my mobile

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Mark Derricutt]

On 16 Apr 2018, at 8:08, Gregg Reynolds wrote:

Can you please elaborate on this a bit? I've been using Bazel for a project that involve C and Java, with cross-compiles, and ABI parsing has never come up. It just works.

That's because Bazel already handles that for you, I've not followed it lately but originally you had ijars - which were interface jars that the tool would extract for your modules/dependencies which tracked the public interface, which is essentially the ABI here.

This is generated from compiled class files etc. this would work in the Clojure space -IF- you were generating AOT compiled .class files from clojure namespaces, but most people don't do that.

Bazel does favour small build units, so you could just jar up lots of smaller namespaces, and only rebuild if the jar is older than any .clj* file in your globs, which would still garner some improvements depending on the size of the namespaces, or the amount of tests in them.

Mark

---

"The ease with which a change can be implemented has no relevance at all to whether it is the right change for the (Java) Platform for all time." — Mark Reinhold.

Mark Derricutt
http://www.theoryinpractice.net
http://www.chaliceofblood.net
http://plus.google.com/+MarkDerricutt
http://twitter.com/talios
http://facebook.com/mderricutt

[Colin Fleming]

Don't wanna start a religious war, but I've worked with Gradle (and about 150 other build tools). If Gradle makes you happy Bazel will make you rapturous.

I need to use Gradle because the IntelliJ plugin development infrastructure uses it.

That said, I just skimmed that blog post, and they have using Groovy as a negative, and two project-specific languages I've never heard of as a positive. I'm unlikely to agree with them on that - IntelliJ has great Groovy support, and there are a lot of blog posts about any problems I might come across. Groovy is far from my favourite language, but at least it's widespread which helps if you don't want to become an expert in it to just get your job done.

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

- sent from my mobile

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

[Nathan Fisher]

It's not something that as a user you'd be exposed to. The easiest way to demonstrate the impact is to write a library and test target in two different ways for a Java project.

1. using java_library target for the library.

2. using genrule for the library that calls out to javac.

Run the test target, introduce whitespace into the library, and run the test target again. It should only do a full execution in the 2nd example.

[Reid McKenzie]

I previously worked with pants [1] in the context of an extremely large (~50GiB) repository which saw a huge amount of daily churn. In that context build tools like pants and Bazel become absolutely essential because the only way to have a reasonable CI/CD system and build/test times is to do change analysis to predict what tests and other products are impacted. While I think that Pants, Buck [2], Bazel and other Google Blaze derivatives have made fundamental and perhaps undesirable complexity sacrifices in order to be polygot, they do add significant value to a large organization.

Presently, I work on a much smaller Clojure project encompassing several microservices which communicate over Kafka. Each service is its own sub-project in a single repository. We coordinate modules using lein-modules [3]. For a variety of reasons, the majority of our interesting tests involve Kafka and potentially integrating against other services continually built out of the same repo. These tests are wall clock expensive due to a variety of system initialization overheads which are difficult to amortize.

It has already proven very profitable for us to deploy a limited amount of test invalidation by using the inter-module dependency information lein-modules exposes to restrict test runs to those modules which have seen source changes or which according to topological sort order may depend on impacted modules. This is a correct but non-minimal heuristic which leaves time on the table. Our deployment artifacts are also fairly expensive to produce, and so I'm very interested in the possibility of a polygot tool which can capture dependency information both for test invalidation and build product invalidation.

It is in theory possible to write a "macro" (function) for either Pants or Buck which defines a clojure_library or clojure_app target in terms of simply generating a java_library which happens to have your clj(c) files for resources and depend on Clojure. I got this working for Buck at one point, but Buck doesn't support Maven dependencies and requires that you hand-manage all your jars which made the ergonomics unfortunate. I haven't taken a serious crack at getting Pants wired up that way. I worked with the pants developers at one point to understand what they thought it would take to get a clojure_library target integrated "the right way" and it turns out to be a mess because of how pants' JVM model is coupled to Ivy, Scala's incremental compiler.

I've been sketching at Katamari [4] on and off, the thought being that the fundamental dependency and invalidation concerns are well enough understood [7] that maybe by trying to build a framework or fire customers it's possible to achieve a meaningful complexity reduction. I haven't managed that yet. Of particular frustration is notions of when to build classpaths and resolve dependencies - although forcing dependency pinning fixes this which is Buck's answer.

There's also Juxt's mach [5] (not to be confused with Mozilla's build tool of the same name), krei.alpha [6] which has some of the same goals as katamari and I've seen some sketches at multi-module boot.

In  short, I do think that more sophisticated build tools make sense in the Clojure space. Sure Clojure itself doesn't have traditional compilation needs in the sense of say C++ or Scala, but notions of products and static dependency information is more generally valuable and can help with whole applications or systems. After spending a fair bit of time evaluating the options, I concur with Colin. If lein-modules or lein-monolith doesn't solve your problem Gradle is probably the most mature "off the shelf" solution.

Reid

[1] https://pantsbuild.org/

[2] https://buckbuild.com/

[3] https://github.com/FundingCircle/lein-modules

[4] https://github.com/arrdem/katamari

[5] https://github.com/juxt/mach

[6] https://github.com/SevereOverfl0w/krei.alpha

[7] https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf

[ri...@chartbeat.com]

We currently use Pants to manage our huge python deployment and it's been a godsend for dependency management. We also have a bunch of clojure based kafka consumers (and a few APIs) that live in the same repo, but are built manually with lein and versioned/deployed separately. We've wanted a pants-like system for our clojure builds for a while now. I looked at building a plugin for pants (as have others https://groups.google.com/forum/#!topic/pants-devel/PPIAEKzKm8w) as Pants does support java, however it's based on incremental builds with nailgun and seems to build a separate jar per directory (fitting in with the pants philosophy). I'm not sure I like this model, and there doesn't seem to be an nailgun-like solution for clojure anyway. 

We're starting to lean towards gradle as a top-level build system that would allow us to declare internal dependencies and keep 3rd party artifact versions in sync. I think this is the cleanest solution. Gradle's very flexible and it's easy to add new functionality to it. 

[Gregg Reynolds]

On Mon, Jun 11, 2018, 2:30 PM <ri...@chartbeat.com> wrote:

...

We're starting to lean towards gradle 

You do understand that that means you have to deal with groovy? ;)

Fwiw I've been working with Bazel a lot lately and I would never go back.

[Rick Mangi]

Believe it or not, I actually like groovy a lot. I used to use it to write unit tests for java code. It's very easy to learn.

[Gregg Reynolds]

On Mon, Jun 11, 2018, 2:44 PM Rick Mangi <ri...@chartbeat.com> wrote:

Believe it or not, I actually like groovy a lot. I used to use it to write unit tests for java code. It's very easy to learn.

Will wonders never cease? I had to learn a bit of groovy a while back while dealing with Gradle, and came to hate it. Maybe I just have a bad attitude - I blame Clojure!

[Rick Mangi]

Next time you have to write unit tests for java, give groovy a try. It has a lot of features that make this easy, 1 line map construction, overriding of private methods, super easy to make mocks. There's definitely some weird syntax, and it's slow, but I'm fine using it for glue or scripting.

[Gregg Reynolds]

On Mon, Jan 29, 2018, 10:16 AM Kiril Videlov <kiri...@runbox.no> wrote:

Hello,

I have been looking at the Bazel build tool for a multi-language mono-repo and I was wondering if anybody has tried it in the context of Clojure projects.

Let's go back to basics: what do you mean by "multi-language"?

Which languages do you need to support?

G

[Gregg Reynolds]

On Mon, Jun 11, 2018, 2:55 PM Rick Mangi <ri...@chartbeat.com> wrote:

Next time you have to write unit tests for java, give groovy a try.

Well I guess we're getting a little of topic for clojure, but I see your point. Problem is I have only so much gray matter, I'm not sure how much room there is inside my skull for yet another language. Can't we write such tests in clojure?  ;)

[Erik Assum]

Totally off topic, but I once worked on a Java project where some of the tests were written in groovy and they were a huge PITA:

1) The refactoring tools never found, nor refactored the groovy code

2) The compilation errors introduced by 1) never showed up in my IDE

3) Having to relate to tests I didn’t know about in a language I don’t care about when I thought I was done was not great for my general mood at the time

I used to think that sneaking in another language by using it in the test-code was a stellar idea. Now I’m quite certain it isn’t. 

Erik. 

-- 

i farta

[Gregg Reynolds]

On Mon, Jun 11, 2018, 3:03 PM Erik Assum <er...@assum.net> wrote:

Totally off topic, but I once worked on a Java project where some of the tests were written in groovy and they were a huge PITA:

1) The refactoring tools never found, nor refactored the groovy code

2) The compilation errors introduced by 1) never showed up in my IDE

3) Having to relate to tests I didn’t know about in a language I don’t care about when I thought I was done was not great for my general mood at the time

😂

I used to think that sneaking in another language by using it in the test-code was a stellar idea. Now I’m quite certain it isn’t. 

Unless it's clojure!

G

[Colin Yates]

If we are off topic... I once worked with a really smart guy who decided that AOP (using the Spring proxying mechanisms) was a great way of enforcing that customers could only see their line items. Only he did this before he went on holiday, didn’t tell anybody about it and whilst the filtering logic was unit tested there were no integration tests.  Boy, did I (and the other programmers), who hadn’t even heard of AOP, have a fun time figuring out why the 1500 JDBC rows turned into 6 instances of a LineItem.

Oh, the XML hell :-).

For the record, I think AOP is a massively underused yet incredibly powerful tool, but please, for the love of sanity, understand the cognitive load it brings with it.

Another anecdote from when I worked for the the Spring Framework days: the AOP tooling mechanism in Eclipse was shoe-horned in using guess what? Yep, AOP :-).

Sent from my iPhone

[Gregg Reynolds]

On Mon, Jan 29, 2018, 10:16 AM Kiril Videlov <kiri...@runbox.no> wrote:

Hello,

I have been looking at the Bazel build tool for a multi-language mono-repo and I was wondering if anybody has tried it in the context of Clojure projects. I have only found the lein-monolith plugin which appears to address a similar use case sans the polyglot requirement. Do you think there is any value in attempting to build a rule set similar to the bazel rules for Scala?

Hi Kiril,

Any news? What did you decide?

Gregg

[Mark Ingram]

Since the last post here there are are the beginnings of some rules at: https://github.com/simuons/rules_clojure (I put in some PRs needed to make them work with latest versions of Bazel).

Separate from that effort I have been looking at whether it is possible to structure the Bazel project to get by using the built in Java rules. This appears possible if you're willing to live with java / javatests as directory names for the necessary classpath handling.

Example repo here: https://github.com/markdingram/bazel-clojure

with:

- external maven deps

- REPL

- Testing (Kaocha)

- AOT

There's also a larger example that adds some Typescript via the AWS CDK to deploy a Clojure Lambda.

https://github.com/markdingram/bazel-cdk-clojure

Mark

On Sunday, 17 June 2018 22:50:12 UTC+1, Gregg Reynolds wrote: