What's the end goal for tools.deps?

[Didier]

I read the rationale, and I understand that we needed a way to bundle depencies for clj and clojure cli. But in practice, I'm seeing a lot of people move to it, away from boot or lein, and I fail to see what the end goal really is?

Any serious work will need a build tool of some sort. If you use lein, it comes with its own dependency management and config. Same for boot (as far as I know). So in practice, if I use tools.deps, I've now doubled the number of tools I depend on. I need lein and tools.deps.

For me, it seems the benefit would be around unified dependency config format and maybe reuse of the resolver logic. Like say Lein and Boot and all tools needing to specify dependencies adopted the format of tools.deps. This would be nice, especially for source based dependencies, and dealing with transitive dependencies in such case.

So, is there any hope/work to adopt the format in Lein and Boot? Or what is the end goal otherwise?

Regards

[Matching Socks]

The premise that "Any serious work will need a build tool of some sort" is being challenged.

You will check your clj files into Git somewhere and tell people to run your program with "clj...".

If you must build something, such as a web-archive for Servlets, then you will stick with tools that do that.

Reading between the lines, I see a sidelining of AOT compilation. 

[Alex Miller]

On Saturday, November 3, 2018 at 10:46:45 PM UTC-5, Didier wrote:

I read the rationale, and I understand that we needed a way to bundle depencies for clj and clojure cli. But in practice, I'm seeing a lot of people move to it, away from boot or lein, and I fail to see what the end goal really is?

The end goal is to have a library and tool for building classpaths. These are foundational building blocks for other things. As far as I'm concerned, that goal has been achieved. I think there is another round of focus work required before we lock down the api and move tools.deps out of alpha (but that's going to wait until after we release 1.10 and do a round of spec work).

 

Any serious work will need a build tool of some sort.

Many people are finding that this is not true. clj by itself is enough to work on a library and make it available. For project work, clj + some task-specific tools (test runners + packaging/deployment) is sufficient for another chunk of uses. Larger / more complex projects already typically have bespoke build architecture, often built around lein or boot but with additional things thrown in. You can build your own tools on top of clj or custom tools using the logic in tools.deps - examples of both exist already. Generally, I think tools.deps is a much simpler dependency to build on top of than lein or even boot once you get to this stage. Inevitably, I think we will see some interesting compositional tools emerge.

 

If you use lein, it comes with its own dependency management and config. Same for boot (as far as I know). So in practice, if I use tools.deps, I've now doubled the number of tools I depend on. I need lein and tools.deps.

If you want to use both, then yes. But if you are happy using lein or boot, use lein or boot. They are equally as viable choices now as they were before clj existed. 

 

For me, it seems the benefit would be around unified dependency config format and maybe reuse of the resolver logic. Like say Lein and Boot and all tools needing to specify dependencies adopted the format of tools.deps. This would be nice, especially for source based dependencies, and dealing with transitive dependencies in such case.

There are plugins for both lein and boot to do this now.

 

So, is there any hope/work to adopt the format in Lein and Boot? Or what is the end goal otherwise?

Hard for me to say, better answered by lein/boot maintainers.

 

Regards

[Alex Miller]

On Sunday, November 4, 2018 at 6:50:19 AM UTC-6, Matching Socks wrote:

Reading between the lines, I see a sidelining of AOT compilation. 

In general, I'd say distributing libraries as source has always been preferable and is pretty ubiquitous. I can't think of any library distributed as AOT-only (there are a few contribs that ship an additional aot classifier version of the lib, but that's not the one most people consume). 

AOT-compiling production apps is common, but that's not typically distributed. Several tools already exist to AOT compile deps.edn projects and people are doing that.

In general, I see largely the status quo with deps.edn projects - libs are being distributed source-only (sometimes via git deps instead of an artifact) and apps are still being AOT'ed. 

The core team continues to talk about ways to make AOT caching more integrated and automatic (there is no reason to AOT-compile the same version of a lib every time you start a Clojure app). And we continue to talk about ways to make loading faster. Not actively working on anything at the moment, but I expect this will bubble up again soon.

[Sean Corfield]

Any serious work will need a build tool of some sort. If you use lein, it comes with its own dependency management and config. Same for boot (as far as I know). So in practice, if I use tools.deps, I've now doubled the number of tools I depend on. I need lein and tools.deps.

 

At World Singles Networks, we’ve just completed our shift from a complex Boot-based pipeline to just clj/deps.edn and a small shell script to make the invocation simpler. We originally started with Leiningen (back in 2011). We switched to Boot at the end of 2015 (you can read why, and details about our journey, here: http://corfield.org/blog/categories/boot/ ). We still have two small Boot tasks, related to legacy projects, that I will likely convert this week. I’ll write a lengthy blog post (or two or more) about why we made this switch and what pros and cons we encountered, but I’d sum it up as “simplicity”. We are now dependent solely on a small, focused tool that is “standard” with Clojure. For deployment, we still build uberjars (using my fork of depstar that fixes a couple of bugs, and adds “thin” JAR support). We start those JARs with java -cp path/to/the.jar clojure.main -m our.namespace which is pretty much what we were doing with Boot (we used java -jar path/to/the.jar -m our.namespace but switched to the slightly longer form just before we migrated to clj/deps.edn). We use Cognitect’s test-runner, we use Olical’s depot (to track outdated dependencies).

 

I’ve published JARs to Clojars using just clj -Spom, clj -A:jar (invoking depstar), and mvn deploy:deploy-file (the only thing I actually use Maven for!).

 

As I’ve worked on various Contrib libraries (that already have pom.xml and use Maven on the CI system), I’ve removed the “convenience” project.clj files that were originally added to make working with those projects easier (e.g., to start an nREPL server, to run tests with multiple versions of Clojure etc) and replaced them with deps.edn files – and by using aliases for different Clojure versions, I can do multi version testing like this: for v in 1.6 1.7 1.8 1.9 master; do clojure -A:test:$v; done

 

I’ve published my .clojure/deps.edn file on GitHub https://github.com/seancorfield/dot-clojure as an example of tooling integration with a clj-based workflow: starting nREPL servers, running Eastwood, benchmarking, testing, building JAR and uberjar files, even creating new clj-based projects.

 

If you’re happy with Leiningen or Boot, there’s no reason for you to switch. You can still adopt the new deps.edn format and use https://github.com/RickMoynihan/lein-tools-deps or https://github.com/seancorfield/boot-tools-deps to integrate it, if you wish.

 

One important caveat to consider right now is that there is no support for Windows for clj/deps.edn projects (I develop on both macOS and Windows 10 – I just use WSL with Ubuntu on the latter – and at work everything is deployed to Linux).

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

---

From: clo...@googlegroups.com <clo...@googlegroups.com> on behalf of Didier <did...@gmail.com>
Sent: Saturday, November 3, 2018 8:46:45 PM
To: Clojure
Subject: What's the end goal for tools.deps?

 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Didier]

Thanks everyone.

I guess I see the need for a dependency manager. And I love having a CLI to launch Clojure programs. And I understand it is fundamental. That said, it always felt like Lein had solved that problem long ago. Maybe it wasn't official enough. But Lein could have been bundled with Clojure and effectively solved the same problems and more.

That said, I do prefer the tools.deps abstraction. Just feels a little like reinventing the wheel.

I'll be interested to read your blog posts Sean.

At my work, I can't use tools.deps, because we have an internal dependency manager. And all dependencies are imported and vetted in it first. So I've always wanted a build tool that was decoupled from dependency management. Since tools.deps only outputs a classpath, I assume all these extra tools you mentioned using to build and run your tests and deploy somehow work only given a classpath. They might be useful to me.

Regards

[James Reeves]

On Sun, 4 Nov 2018 at 18:58, Didier <did...@gmail.com> wrote:

I guess I see the need for a dependency manager. And I love having a CLI to launch Clojure programs. And I understand it is fundamental. That said, it always felt like Lein had solved that problem long ago. Maybe it wasn't official enough. But Lein could have been bundled with Clojure and effectively solved the same problems and more.

My impression is that the core devs prefer an approach made up of many small, simple tools, rather than the batteries-included approach of Leiningen. I don't think it's a case of Leiningen not being "official", but that it's design isn't quite in the direction the core devs want to head toward.

--

James Reeves

booleanknot.com

[Alex Miller]

On Sunday, November 4, 2018 at 12:58:43 PM UTC-6, Didier wrote:

Thanks everyone.

I guess I see the need for a dependency manager. And I love having a CLI to launch Clojure programs. And I understand it is fundamental. That said, it always felt like Lein had solved that problem long ago. Maybe it wasn't official enough. But Lein could have been bundled with Clojure and effectively solved the same problems and more.

Lein is an excellent "better Maven" for Clojure and it has served an incredibly important role in standardizing project structures and making it easier for Clojure projects to consume and deploy Maven artifacts, particularly to access the vast Java ecosystem to the Clojure community. 

However, we believe there is a better path that starts from reconsidering the problem space without using a Maven-artifact based dependency system, and instead thinking about what Clojure projects need. While consuming Maven artifacts is important, an artifact-centric world is more limited than it needs to be in Clojure. Clojure fundamentally works from source on the classpath. Note the lack of the words artifact, compile, or build in that statement. Taking this approach is what allows us to easily consume some Clojure source that happens to be sitting on github in a repo. Or a db driver sitting in a jar on the disk. The producer does not need to build and deploy an artifact into Maven. The consumer does not need to download an artifact. We can simply consume the source directly without these middle systems. 

 

That said, I do prefer the tools.deps abstraction. Just feels a little like reinventing the wheel.

I know Rich hopes strongly that the Clojure community is a place where people are free and supported when attempting to creatively solve problems in new ways. Certainly Clojure itself wouldn't exist if the community wasn't willing to try something new and different. Sometimes you have to start over, with a new set of assumptions and constraints and see where it goes.

 

I'll be interested to read your blog posts Sean.

At my work, I can't use tools.deps, because we have an internal dependency manager. And all dependencies are imported and vetted in it first. So I've always wanted a build tool that was decoupled from dependency management. Since tools.deps only outputs a classpath, I assume all these extra tools you mentioned using to build and run your tests and deploy somehow work only given a classpath. They might be useful to me.

Generally those tools are just Clojure programs with a clojure.main. If you can build a classpath from your dependency manager, then invoke the tool, then yes, all those tools are open to you as well (and for packaging as lein plugins or boot tasks too). There is no magic here. It's just running Clojure programs.

[Didier]

Generally those tools are just Clojure programs with a clojure.main. If you can build a classpath from your dependency manager, then invoke the tool, then yes, all those tools are open to you as well (and for packaging as lein plugins or boot tasks too). There is no magic here. It's just running Clojure programs.

Currently, we perform all build tasks using custom Clojure code. We have our own test runner, and script to launch cloverage, kibit and eastwood, we AOT compile and package a hybrid JAR of source and .class (to support consumption from Java and Clojure). But I've always wanted to leverage existing work for these, as it can get complicated. I never figured out in Lein how to swap the dependency resolution logic with our own, and just leverage the build features and the accompanying plugins. There's no way to just provide the dependency tree or the classpath to Lein that I could find. It seems tools designed for tools.deps might be easier for me to leverage, like https://github.com/cognitect-labs/test-runner. This looks pretty great, I'll see if I can swap to it instead of out custom test-runner.

 

I know Rich hopes strongly that the Clojure community is a place where people are free and supported when attempting to creatively solve problems in new ways. Certainly Clojure itself wouldn't exist if the community wasn't willing to try something new and different. Sometimes you have to start over, with a new set of assumptions and constraints and see where it goes.

Definitely, though I do hope we won't be experimenting with dependencies much more. It's hard to keep up, maven, lein, boot, now clj, throw in ClojureScript and you need to deal with npm too. It gets quite complicated. To me, this is mostly accidental complexity created by the lack of a common standard.

At least tools.deps has enough of a differentiator, I can recognize its worth the change. Being official is really key to me, so I'm happy.

The risk I see is that you can end up in a place where dependency resolution becomes incompatible. Say you depend on package X, and it uses a different dependency declaration format. Now you can't resolve its dependencies.

This can also happen for build logic. Though it's not as bad a problem in Clojure. There's packages that say need to pull in resources, or must perform AOT, and if you depend on such a package source, you can't transitively build it if the build process declaration is in incompatible formats.

Anyways, I was mostly curious, just wanted to understand better the rationale. I'll look forward to what happens next, and I assume things will standardize around it more and more, which will solve most of my issues.

[Sean Corfield]

I guess I see the need for a dependency manager.

 

I’m curious as to how you define this tool: what exactly do you consider a “dependency manager” to be, and what are it’s specific functions/features? You talk about having an “internal dependency manager” – how do you leverage that from Clojure?

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

From: clo...@googlegroups.com <clo...@googlegroups.com> on behalf of Didier <did...@gmail.com>

Sent: Sunday, November 4, 2018 10:58:43 AM
To: Clojure
Subject: RE: What's the end goal for tools.deps?

 

[Didier]

I’m curious as to how you define this tool: what exactly do you consider a “dependency manager” to be, and what are it’s specific functions/features? You talk about having an “internal dependency manager” – how do you leverage that from Clojure?

By dependency manager, I mean a system that defines "packages" and dependencies (including transitive) between them. Which can also retrieve those packages, and generate the dependency graph.

I'll try to explain, but my work highly values reproducible builds. And so everything stems from that. Think of reproducible build as the idea that you can regenerate the identical artifact and environment produced by a past deployment.

So our packages are managed by some system, each package is just a git repo with some config file that defines the dependencies of the package on others.

This is what I call the "dependency manager". It tracks package dependencies, where in our case, packages are git repos.

You can ask it to generate the dependency graph of a given package, and you can also use it to retrieve all packages into a local copy, and generate a classpath pointing in the correct order to all the local copies.

At this point, it is a lot like tools.deps. Except that it also has a global registry with additional meta data support for packages, like versions.

Another difference is that it is tied into our build and deployment process. When you build a package, the build system uses the dependency manager to pull down all transitive packages of the defined version, it then proceeds to build them in order. All artifacts are then cached in an artifact repository. The package thus references a git repo, each package version maps to an exact commit, and artifact build cache.

When we deploy, it's just the same thing that happens. The host uses the dependency manager to build the dependency tree, but the difference here, is that, instead of retrieving the git source for the specified commits of the specified versions of each package, it will grab there corresponding cached artifacts as a performance optimization.

So now, when it comes to Clojure, I can't use lein or tools.deps to retrieve package dependencies, since they don't know about our internal packages and how to retrieve them. I already have a configuration format to specify dependencies on other packages, and a cli tool which can retrieve them locally (either their source or their corresponding built artifact) from our internal source and artifact repos, and can also generate the classpath.

What I don't have though, is a way to perform Clojure build tasks, like run tests, generate coverage reports, perform AOT, generate Java classes, etc. I can't use lein for that, since I couldn't find how to have it use a custom classpath for all of its tasks, it seems to always implicitly assumes it will retrieve the packages itself and build the classpath.

So I basically use Ant. Which surprisingly, is one of the last build tool that does not complect itself with dependency management as well. So I can use it purely to build. And I created custom Ant tasks for the Clojure related tasks I needed.

I'd love a standard Clojure build tool that is decoupled from a dependency manager, which I could use instead of Ant, and where I didn't have to write the Clojure tasks myself. Though since I do a lot of mixed Java/Clojure project, it would need to also support Java build tasks. So who knows, I might never have the full package I dream off.

Something else that is really painful is bringing in an external Clojure package internally. I'm forced to re-write the dependency format to our internal one. So if it's project.clj or deps.edn, I need to change it to our internal one. Also painful is having to retrieve the source for these, and make an internal source copy. I feel like tools.deps might actually help me here long term, if everything move to source only dependencies, I could leverage it to transitively retrieve and copy over all source and their source dependencies.

Sorry for the long wall of text, but this stuff is actually kind of complicated.

[Austin Haas]

"Several tools already exist to AOT compile deps.edn projects."

Alex, can you please identify some of those projects? I haven't been able to find any.

-austin

[Alex Miller]

See https://github.com/clojure/tools.deps.alpha/wiki/Tools 

[Alex Miller]

I thought several did it, but maybe not that many. At least cambada does.

[Erik Assum]

Sounds like you’re in dire need of an internal installation of artifactory. 

Most build tools let you specify custom repositories, and you could then choose to _only_ use artifactory, both as a repo for the public packages you choose to use, and for your private packages. 

The you can use lein, boot, clj or whatever. 

Erik. 

-- 

i farta

[Khalid Jebbari]

Sean, your deps.edn is a golden mine. Thanks a lot for sharing it.