[ANN] Git Deps for Clojure!

[Alex Miller]

Pleased to announce some new functionality for clj and tools.deps!

https://clojure.org/news/2018/01/05/git-deps

Additionally, there have been new releases of:

- Brew clojure formula (to get it: brew upgrade clojure)

- Linux clojure installer (see https://clojure.org/guides/getting_started for info)

- tools.deps.alpha

- NEW tools.gitlibs

Other than git deps, "clj -Spom" for pom generation has some fixes and an addition to add Maven repositories if needed.

[Gary Trakhman]

Congrats on the release! It's exciting to see this vision move forward.  Wondering if the logical next step is the npm style of trees of duplicate transitive git deps.  In general, how is this going to work for transitive deps?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Miller]

On Fri, Jan 5, 2018 at 1:20 PM, Gary Trakhman <gary.t...@gmail.com> wrote:

Congrats on the release! It's exciting to see this vision move forward.  Wondering if the logical next step is the npm style of trees of duplicate transitive git deps.  In general, how is this going to work for transitive deps?

tools.deps has the ability to read a file-based manifest in a git dep (or a local dep) and traverse its transitive dependencies. This works now for deps.edn based libraries and can be extended for other manifest types. 

I have started work on a pom.xml manifest reader. lein is harder to do right, but maybe easy to do badly. :)  There are still some things to work out wrt how manifest readers like this are dynamically found and loaded; currently they have to be "in the box" but I see that's not the final end state.

[Gary Trakhman]

Specifically, I'm asking about how to resolve duplicate transitive dependencies to actual code, possibly referring to the same lib repo at different git hashes, or different git repos.

I think you answered, 'how do you get a single graph of deps nodes given different kinds of deps'?

For example:

A->B->C#some_tag

A->D->C#other_tag

So, assuming you use a lot of git deps, which I think is the long-term intent here based on the messaging, resolving those pointers to actual loaded code is a potential problem that needs to be addressed somehow.  I mentioned one way, which is to have multiple copies of a lib loaded automatically.

From what I can tell so far, it seems like you can override a dep's specific transitive deps, but that sounds painful when scaled out over number of deps and how many more commit hashes exist than maven release tags for each lib.

Another way to do it might be to make the git repo url+branch itself could be part of the namespace var mapping, but that seems far off based on the way clojure namespaces currently work, and doesn't cover the common case (at least it's what we're used to with maven) of just resolving a single version for each dep.

Is there some existing design thought along these lines that I've missed?

[Alex Miller]

On Friday, January 5, 2018 at 2:08:50 PM UTC-6, Gary Trakhman wrote:

Specifically, I'm asking about how to resolve duplicate transitive dependencies to actual code, possibly referring to the same lib repo at different git hashes, or different git repos.

Ah, good question. The answer is that you need some way to detect version differences and resolve them. Fortunately, this is the identical problem you have in Maven and the solution is effectively the same - pick the newer one. In a git context this means "the commit that has the other as an ancestor". With a growth mindset, this is a safe choice. If two versions do not share a ancestor/descendant lineage, then your build is broken and you will be required to fix it.

 

I think you answered, 'how do you get a single graph of deps nodes given different kinds of deps'?

For example:

A->B->C#some_tag

A->D->C#other_tag

So, assuming you use a lot of git deps, which I think is the long-term intent here based on the messaging, resolving those pointers to actual loaded code is a potential problem that needs to be addressed somehow.  I mentioned one way, which is to have multiple copies of a lib loaded automatically.

Not doing that.

 

From what I can tell so far, it seems like you can override a dep's specific transitive deps, but that sounds painful when scaled out over number of deps and how many more commit hashes exist than maven release tags for each lib.

Either the descendant-most version will be chosen or your deps are broken and will fail to compute a classpath.

Additionally, most Maven artifacts (certainly most in Maven central or clojars) contain SCM metadata about the git url and rev that they represent. This gives us the ability to compare git deps with Maven deps (via their git rev) and apply the same logic. Not yet done, but coming soon...

[Khalid Jebbari]

Great news, great work ! Soon running Clojure will be dead easy.

2 questions :
- where do dependencies get downloaded ? `~/.m2` ? It depends on the procurer (mvn/git/etc.)
- does it work for Cljs ? What does it mean for Cljs ?

[Alex Miller]

> On Jan 6, 2018, at 5:43 AM, Khalid Jebbari <khalid....@gmail.com> wrote:
>
> Great news, great work ! Soon running Clojure will be dead easy.
>
> 2 questions :
> - where do dependencies get downloaded ? `~/.m2` ? It depends on the procurer (mvn/git/etc.)

It depends. Maven deps go to .m2, git deps go to .gitlibs.

> - does it work for Cljs ? What does it mean for Cljs ?

This is targeted at launching Clojure programs. ClojureScript itself is a Clojure program, so it’s relevant in that sense.

[Daniel Compton]

> git deps go to .gitlibs.

Would you consider not putting them in the root of the user's home directory? All the major OS's (and probably the minor ones too) have dedicated folders to put caches. I've opened https://dev.clojure.org/jira/browse/TDEPS-30 with more details on this.

--

Daniel

[Nathan Fisher]

Great work Alex!

Not sure I would want to see duplicates either. If my code breaks because of an old dependency I’d rather that than make the false assumption that I’m running securely with the latest version of a lib.

--

- sent from my mobile

[Nathan Fisher]

Im probably not typical for this but I really value the ability to override the location of where they’re placed. I actually commit my deps to SCM for production code. The internet is fast enough these days that i don’t care about a common code cache. I’m more interested in a reproducible build.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Miller]

On Sunday, January 7, 2018 at 2:37:33 PM UTC-6, Daniel Compton wrote:

> git deps go to .gitlibs.

Would you consider not putting them in the root of the user's home directory? All the major OS's (and probably the minor ones too) have dedicated folders to put caches. I've opened https://dev.clojure.org/jira/browse/TDEPS-30 with more details on this.

Will continue discussion on ticket rather than here...

[Alan Thompson]

Hey - I love the idea.  However, I'm getting an error message trying to run the example:

~/work/fred > cat deps.edn 

{:deps

 {github-clj-time/clj-time

  {:git/url "https://github.com/clj-time/clj-time" :rev "cce58248"}}

}                                                                                                                                                                        

~/work/fred > clj

Error building classpath. Manifest type :lein not loaded when finding deps for github-clj-time/clj-time in coordinate {:git/url "https://github.com/clj-time/clj-time", :rev "cce58248937bc05452ebfc8b65134961227a554e", :deps/manifest :lein, :deps/root "/home/alan/.gitlibs/libs/github-clj-time/clj-time/cce58248937bc05452ebfc8b65134961227a554e"}

​Any pointers on where I'm going wrong?

Alan

On Sun, Jan 7, 2018 at 1:43 PM, Nathan Fisher <nfi...@junctionbox.ca> wrote:

Im probably not typical for this but I really value the ability to override the location of where they’re placed. I actually commit my deps to SCM for production code. The internet is fast enough these days that i don’t care about a common code cache. I’m more interested in a reproducible build.

On Sat, 6 Jan 2018 at 11:25, Alex Miller <al...@puredanger.com> wrote:

> On Jan 6, 2018, at 5:43 AM, Khalid Jebbari <khalid....@gmail.com> wrote:
>
> Great news, great work ! Soon running Clojure will be dead easy.
>
> 2 questions :
> - where do dependencies get downloaded ? `~/.m2` ? It depends on the procurer (mvn/git/etc.)

It depends. Maven deps go to .m2, git deps go to .gitlibs.

> - does it work for Cljs ? What does it mean for Cljs ?

This is targeted at launching Clojure programs. ClojureScript itself is a Clojure program, so it’s relevant in that sense.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

- sent from my mobile

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

[Alex Miller]

On Sunday, January 7, 2018 at 3:41:34 PM UTC-6, Nathan Fisher wrote:

Not sure I would want to see duplicates either. If my code breaks because of an old dependency I’d rather that than make the false assumption that I’m running securely with the latest version of a lib.

Encountering duplicate libs in the transitive tree is common to almost every dependency trees (for example, it's typical to see multiple versions of Clojure as a dependency in *every* transitive tree). clj, like every other dependency tool, will continue to make choices about which version to include and tools to control which one is included.

I do plan to add better tools to understand the tree and the choices made.

[Alex Miller]

On Sunday, January 7, 2018 at 5:45:22 PM UTC-6, Alan Thompson wrote:

Hey - I love the idea.  However, I'm getting an error message trying to run the example:

~/work/fred > cat deps.edn 

{:deps

 {github-clj-time/clj-time

  {:git/url "https://github.com/clj-time/clj-time" :rev "cce58248"}}

}                                                                                                                                                                        

~/work/fred > clj

Error building classpath. Manifest type :lein not loaded when finding deps for github-clj-time/clj-time in coordinate {:git/url "https://github.com/clj-time/clj-time", :rev "cce58248937bc05452ebfc8b65134961227a554e", :deps/manifest :lein, :deps/root "/home/alan/.gitlibs/libs/github-clj-time/clj-time/cce58248937bc05452ebfc8b65134961227a554e"}

Once a dependency has been downloaded, deps must read a project manifest to understand its dependencies. Currently, only deps.edn manifests are supported. It's currently autodetecting the lein project.clj but does not have a reader for it. I have already changed the error handling here so this presents differently, but that has not yet been released. Eventually I expect we will have readers for poms and project.clj files, etc.

[Nathan Fisher]

Hi Alex,

Sounds great!

I strongly agree with your decision of “pick the latest”. While I do understand multiple active versions can make it easier for a developer I don’t think that’s the “right” decision. Besides I can only imagine how much of a pain it would be to implement reliably. I’d expect if I specify a patched version of struts that a transitive dependency wouldn’t have the ability to override that for its own purpose.

Cheers,

Nathan

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Miller]

On Sun, Jan 7, 2018 at 6:53 PM, Nathan Fisher <nfi...@junctionbox.ca> wrote:

 

I strongly agree with your decision of “pick the latest”. While I do understand multiple active versions can make it easier for a developer I don’t think that’s the “right” decision. Besides I can only imagine how much of a pain it would be to implement reliably. I’d expect if I specify a patched version of struts that a transitive dependency wouldn’t have the ability to override that for its own purpose.

I guess I didn't mention that top-level deps always win, so if desired you are always able to decide the specific version in your project.

[Gary Verhaegen]

Have you considered adding an equivalent to lein’s pedantic option, i.e. an option that would die on ambiguous versions rather than make a choice, thereby forcing users to make that choice explicit as a top-level entry? (Or through exclusions etc.)

As an aside, is it even possible on the JVM to have multiple versions of a dependency loaded? Wouldn’t the last version loaded override all the common classes, leaving you with a sort of hybrid that doesn’t correspond to any single version? I’d really like a dependency system that makes each dep’s transitive dependencies only visible to itself, so there would never be any reason to resolve dependencies.

Also, how realistic do you think it is today to expect all of our transitive dependencies to be developed according to that growth mindset you mention? It’s one thing to adopt it in my code, but quite another to assume it’s followed correctly by all of the underlying Java libs.

[Nathan Fisher]

You could do it by “rooting” the package (aka change the package path). Most JVM languages root their dependencies to avoid collisions with user space dependencies. ASM is the most common Clojure does and I think Kotlin and Scala do as well. I think Guavas murmur3 is rooted in Clojure as well for the maps.

[Alex Miller]

On Monday, January 8, 2018 at 2:47:23 AM UTC-6, Gary Verhaegen wrote:

Have you considered adding an equivalent to lein’s pedantic option, i.e. an option that would die on ambiguous versions rather than make a choice, thereby forcing users to make that choice explicit as a top-level entry? (Or through exclusions etc.)

Eventually, maybe. I don't see that ever being the default though.

 

As an aside, is it even possible on the JVM to have multiple versions of a dependency loaded? Wouldn’t the last version loaded override all the common classes, leaving you with a sort of hybrid that doesn’t correspond to any single version?

Actually the way things work the *first* version on the classpath is the one that will be used. Having more than one version of the same lib on the classpath is almost certainly a bug, not a feature.

 

I’d really like a dependency system that makes each dep’s transitive dependencies only visible to itself, so there would never be any reason to resolve dependencies.

You need classloader support for this and indeed this is what OSGi and some early versions of the Java module system do. Working in that kind of environment is very constraining. A lot of discipline is required with respect to your interfaces between loaders and in general I think it's way too much of a burden to use as the normal operating mode (which is one reason you don't see it in the Java module system of java 9). It is a good option for systems where you want users to plug in functionality and the scope of those interface points is small and highly controlled.

 

Also, how realistic do you think it is today to expect all of our transitive dependencies to be developed according to that growth mindset you mention? It’s one thing to adopt it in my code, but quite another to assume it’s followed correctly by all of the underlying Java libs.

I think it's unrealistic, but you already live fully in that world. mvn, lein, boot, gradle, sbt, ivy etc - all dependency resolvers in the Java ecosystem already do exactly what I've described above with choosing Java dependencies. Ususally it works, sometimes it fails and manual nastiness is required to resolve the issues. The "growth not breakage" mindset is the way *out* of that problem - where to start if not in our own code?

[Alex Miller]

There is a new version of clojure tools available (1.9.0.302) that changes the attributes for git coordinates:

:rev - removed attribute

:sha - new required attribute, full sha strongly encouraged (prefix sha support may be removed)

:tag - new optional attribute, should match the sha (not used by tools.deps.alpha)

This release also fixes the XML warning under JDK 9 for -Spom.

If you are listing git coordinate information in your README, we would strongly encourage publishing a full :sha (40 chars) example.

[Bobby Eickhoff]

I find it somewhat ironic given all of the recent discussion of growth vs. breakage in the world of Clojure that this latest release of clojure tools -- if I understand correctly -- includes, as advertised above, a breaking change.  :-P

[Alex Miller]

Yeah, we didn't love doing it, but that's why tools.deps.alpha is still alpha. Given that no one is using it yet, it seemed like the most expedient solution. Over time, we will work harder to avoid things like that.

[Kurt Harriger]

 

I’d really like a dependency system that makes each dep’s transitive dependencies only visible to itself, so there would never be any reason to resolve dependencies.

You need classloader support for this and indeed this is what OSGi and some early versions of the Java module system do. Working in that kind of environment is very constraining. A lot of discipline is required with respect to your interfaces between loaders and in general I think it's way too much of a burden to use as the normal operating mode (which is one reason you don't see it in the Java module system of java 9). It is a good option for systems where you want users to plug in functionality and the scope of those interface points is small and highly controlled.

I have run into incompatible transitive dependencies a lot throughout my career.  In many cases its not that big a deal but for cross cutting libraries such as data serialization etc it can be hard to get all of your dependencies to agree on a common version.  Here is a concrete example I ran into just today: https://github.com/thheller/shadow-cljs/issues/177 and the library that caused the issue is non-other than core.async!

I second the idea of private dependencies.  I have had the displeasure of working with OSGI and I think the reason its complex is not so much because the required metadata is hard to figure out... the complexity mostly comes  from the starting, stopping and restarting bundles within a running jvm in non deterministic order.  

Another issue is that the dependency metadata needs to be included within each jar and since most library authors don't see the value of OSGI they don't provide it to users to rebundle the jar with the required metatata. This problem is similar to that of cljsjs, most javascript devs do not see any value in any significant value is using closure compile over uglyify (myself included as externs are a pain to maintain, minified stacktraces are useless, and errors are subtle and hard to debug), but the clojurescript community does see value in this so they do their best to maintain this metadata even if it means rebuilding projects. 

So what is the alternatives:  

1. Don't upgrade (common solution when there are breaking changes)

2. Try to get upstream maintainer to upgrade (most common when changes are not breaking and/or maintainer willing to do the work)

3. Fork the upstream project and maintain your own release using the desired version

4. Split your app into multiple pieces so they can run in different JVM instances (enter the microservice)

I think the idea that everything can use the same version is oversimplified and creating microservice shouldn't be necessary just because you can't get all your dependencies to agree.

[Isaac Tsang]

can `clj` AOT automatically, and then cache it (save to some directory).

some dependencies need very long time for `require`

[Alex Miller]

> On Jan 18, 2018, at 3:37 AM, Isaac Tsang <ndtm...@gmail.com> wrote:
>
> can `clj` AOT automatically, and then cache it (save to some directory).
> some dependencies need very long time for `require`

Not yet. :) We are working on something along these lines.