Classpath bug re Clojure 1.10.1.645 when using Figwheel.Main

[Alan Thompson]

Hi.  Just helped a colleague debug a vexing problem on a CLJS project using Figwheel.Main.

If we do `brew install clojure/tools/clojure`, it works:

~/work/tmp810/xanadu > clj --help

Version: 1.10.1.561

Usage: clojure [dep-opt*] [--] [init-opt*] [main-opt] [arg*]

       clj     [dep-opt*] [--] [init-opt*] [main-opt] [arg*]

The clojure script is a runner for Clojure. clj is a wrapper

for interactive repl use. These scripts ultimately construct and

....

Note that local ./resources etc are at the beginning of the classpath

~/work/tmp810/xanadu > clj -Spath

resources:target:src/clj:src/cljc:src/cljs:test/cljs:test-figwheel-main:/Users/alanthompson/.m2/repository/com/cognitect/transit-java/0.8.332/transit-java-0.8.332.jar:/Users/alanthompson/.m2/repository/com/google/elemental2/elemental2-core/1.0.0-RC1/elemental2-core-1.0.0-RC1.jar:/Users/alanthompson/.m2/repository/org/clojure/data.json/0.2.6/data.json-0.2.6.jar:/Users/alanthompson/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/alanthompson/.m2/repository/day8/re-frame/test/0.1.5/test-0.1.5.jar:/Users/alanthompson/.m2/repository/commons-codec/commons-codec/1.11/commons-codec-1.11.jar:/Users/alanthompson/.m2/repository/cljsjs/material-ui-currency-textfield/0.8.6-0/material-ui-currency-textfield-0.8.6-0.jar:/Users/alanthompson/.m2/repository/org/clojure/tools.analyzer/1.0.0/tools.analyzer-1.0.0.jar:/Users/alanthompson/.m2/repository/com/bhauman/cljs-test-display/0.1.1/cljs-test-display-0.1.1.jar:/Users/alanthompson/.m2/repository/org/eclipse/jetty/jetty-xml/9.4.28.v20200408/jetty-xml-9.4.28.v20200408.jar:/Users/alanthompson/.m2/repository/com/bhauman/figwheel-repl/0.2.11/figwheel-repl-0.2.11.jar:/Users/alanthompson/.m2/repository/org/eclipse/jetty/jetty-servlet/9.4.28.v20200408/jetty-servlet-9.4.28.v20200408.jar:/Users/alanthompson/.m2/repository/ring/ring-devel/1.8.1/ring-devel-1.8.1.jar:/Users/alanthompson/.m2/repository/com/google/errorprone/error_prone_annotations/2.3.1/error_prone_annotations-2.3.1.jar:/Users/alanthompson/.m2/repository/org/clojure/tools.logging/0.3.1/tools.logging-0.3.1.jar:/Users/alanthompson/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:/Users/alanthompson/.m2/repository/co/deps/ring-etag-middleware/0.2.0/ring-etag-middleware-0.2.0.jar:/Users/alanthompson/.m2/repository/expound/expound/0.7.2/expound-0.7.2.jar:/Users/alanthompson/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/Users/alanthompson/.m2/repository/com/cemerick/url/0.1.1/url-0.1.1.jar:

.....<snip>.....

However, my colleague had accidentally typed `brew install clojure`.  This caused a mysterious failure:

-------------------

~/work/tmp810/xanadu > clojure --help
Version: 1.10.1.645

You use the Clojure tools ('clj' or 'clojure') to run Clojure programs
on the JVM, e.g. to start a REPL or invoke a specific function with data.

....

and the classpath

-----------------

~/work/tmp810/xanadu > clj -Spath

DEPRECATED: Libs must be qualified, change deps-ancient => deps-ancient/deps-ancient (deps.edn)

DEPRECATED: Libs must be qualified, change reagent => reagent/reagent (deps.edn)

DEPRECATED: Libs must be qualified, change ns-tracker => ns-tracker/ns-tracker (deps.edn)

DEPRECATED: Libs must be qualified, change camel-snake-kebab => camel-snake-kebab/camel-snake-kebab (deps.edn)

DEPRECATED: Libs must be qualified, change bidi => bidi/bidi (deps.edn)

DEPRECATED: Libs must be qualified, change orchestra => orchestra/orchestra (deps.edn)

DEPRECATED: Libs must be qualified, change cljs-ajax => cljs-ajax/cljs-ajax (deps.edn)

DEPRECATED: Libs must be qualified, change expound => expound/expound (deps.edn)

DEPRECATED: Libs must be qualified, change re-frame => re-frame/re-frame (deps.edn)

DEPRECATED: Libs must be qualified, change re-frame-utils => re-frame-utils/re-frame-utils (deps.edn)

DEPRECATED: Libs must be qualified, change cljs-bean => cljs-bean/cljs-bean (deps.edn)

/Users/alanthompson/.m2/repository/alandipert/storage-atom/1.2.4/storage-atom-1.2.4.jar:/Users/alanthompson/.m2/repository/com/google/errorprone/error_prone_annotations/2.3.1/error_prone_annotations-2.3.1.jar:/Users/alanthompson/.m2/repository/org/clojure/core.cache/1.0.207/core.cache-1.0.207.jar:/Users/alanthompson/.m2/repository/com/google/jsinterop/jsinterop-annotations/1.0.2/jsinterop-annotations-1.0.2.jar:/Users/alanthompson/.m2/repository/compliment/compliment/0.3.6/compliment-0.3.6.jar:/Users/alanthompson/.m2/repository/ring/ring-headers/0.3.0/ring-headers-0.3.0.jar:/Users/alanthompson/.m2/repository/cljs-bean/cljs-bean/1.4.0/cljs-bean-1.4.0.jar:/Users/alanthompson/.m2/repository/hawk/hawk/0.2.11/hawk-0.2.11.jar:/Users/alanthompson/.m2/repository/binaryage/devtools/1.0.0/devtools-1.0.0.jar:/Users/alanthompson/.m2/repository/org/eclipse/jetty/websocket/websocket-api/9.4.28.v20200408/websocket-api-9.4.28.v20200408.jar:/Users/alanthompson/.m2/repository/org/apache/httpcomponents/httpasyncclient/4.1.3/httpasyncclient-4.1.3.jar:/Users/alanthompson/.m2/repository/org/checkerframework/checker-qual/2.0.0/checker-qual-2.0.0.jar:/Users/alanthompson/.m2/repository/ns-tracker/ns-tracker/0.3.1/ns-tracker-0.3.1.jar:/Users/alanthompson/.m2/repository/cljsjs/react-select/2.4.4-0/react-select-2.4.4-0.jar:/Users/alanthompson/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/alanthompson/.m2/repository/com/google/protobuf/protobuf-java/3.11.1/protobuf-java-3.11.1.jar:/Users/alanthompson/.m2/repository/org/fusesource/jansi/jansi/1.16/jansi-1.16.jar:/Users/alanthompson/.m2/repository/com/cognitect/transit-clj/0.8.309/transit-clj-0.8.309.jar:/Users/alanthompson/.m2/repository/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.jar:/Users/alanthompson/.m2/repository/org/apache/httpcomponents/httpclient/4.5.3/httpclient-4.5.3.jar:test-figwheel-main:/Users/alanthompson/.m2/repository/com/cognitect/transit-cljs/0.8.256/transit-cljs-0.8.256.jar:

.....<snip>.....

:/Users/alanthompson/.m2/repository/zprint/zprint/0.5.1/zprint-0.5.1.jar:resources:/Users/alanthompson/.m2/repository/org/clojure/tools.analyzer/1.0.0/tools.analyzer-1.0.0.jar:/Users/alanthompson/.m2/repository/re-frame/re-frame/0.10.5/re-frame-0.10.5.jar:/Users/alanthompson/.m2/repository/rewrite-cljs/rewrite-cljs/0.4.4/rewrite-cljs-0.4.4.jar:/Users/alanthompson/.m2/repository/net/cgrand/macrovich/0.2.0/macrovich-0.2.0.jar:/Users/alanthompson/.m2/repository/org/clojure/clojurescript/1.10.764/clojurescript-1.10.764.jar:/Users/alanthompson/.m2/repository/com/google/elemental2/elemental2-core/1.0.0-RC1/elemental2-core-1.0.0-RC1.jar:/Users/alanthompson/.m2/repository/org/eclipse/jetty/jetty-client/9.4.28.v20200408/jetty-client-9.4.28.v20200408.jar

with `resources` nearly at the end!  This breaks Figwheel.Main, which cannot find `index.html` correctly and puts up a bogus webpage of 2 words:

Debux Test

which I assume is due to finding some other `index.html` earlier on the classpath.

Questions:

1. What is the difference between the brew targets `clojure` and `clojure/tools/clojure`?  Can these be unified somehow? It seems error-prone as-is.
2. I assume the classpath change in 1.10.1.645 is a bug that needs to be corrected?
3. Is it possible this is a Figwheel.Main problem instead?

Thanks,

Alan

[Alan Thompson]

P.S.  There seems to be no `clojure --version` flag.  Should this be added to the command line tool?

[Alex Miller]

Bunch of things here...

Clojure maintains its own brew tap and a "stable" release that you can obtain with `brew install clojure/tools/clojure` (the brew conventions automatically find the prior repo based on that). That tap also includes prerelease unstable versions that can be obtained with "@version" - more on that is doc'ed in the readme for that repo. The current stable version is 1.10.1.561.

Homebrew core is what you are pulling from if you just do `brew install clojure`. The formula there is no longer maintained by the Clojure team as anyone can update it (and have, with changes we did not agree with). There is no "ownership" model in homebrew-core. I would happily remove it from there but they said they would not accept that PR. Recently, the homebrew-core formula has been updated by members of the homebrew team to newer prerelease (not yet stable) versions of clj. There is not really anything we can do about this.

The classpath difference is interesting. The classpath is a set of path roots and in general the files in these paths should be mutually exclusive. If the roots are mutually exclusive, then the classpath order is unimportant (as the JVM will always find the same class/clj/resource file regardless). Roots that include the same file are nearly always trouble and the source of "jar hell" style problems. clj and deps.edn provide features to cover most of the scenarios where users are typically tempted to use ordering for replacement (:override-deps, :classpath-overrides, etc).

Based on this, we did knowingly make a change since the last stable release that effectively made the classpath unordered (all the roots go through keys of a map at one point). To date, I had not seen any issues with that, so this is a useful data point we can take into consideration. I assume you're getting an index.html out of some library that's either unimportant or accidentally included. For now, I'd recommend that you continue using the latest stable version from the official Clojure homebrew tap.

Re the version - you can see the version with either `clj -h` (first line) or `clj -Sdescribe`.

Relevant links:

* Clojure homebrew tap - https://github.com/clojure/homebrew-tools

* Official clj stable formula - https://github.com/clojure/homebrew-tools/blob/master/Formula/clojure.rb (what you get with `brew install clojure/tools/clojure`)

* Getting started/install doc - https://clojure.org/guides/getting_started

* Homebrew core - https://github.com/Homebrew/homebrew-core

* clj reference - https://clojure.org/reference/deps_and_cli (or see `clj -h` or `man clj`)

[Thomas Heller]

This might be a good incentive for people to keep their published .jar files clean. Unfortunately many published CLJS libs contain the rather common "public" folder with "public/index.html" and often compiled .js artifacts which aren't actually ever used.

I do however think that it is useful to always have :paths first since it provides and easy way to "patch" libraries. You can just easily replace one namespace from a library and provide custom implementations without going through the entire forking process. This is used by many CLJS users as well since on older reagent versions it was required to stub out the reagent.dom [1] namepace in react-native builds. None of this is very pretty but having :paths first (just like lein :source-paths) at least gives you the option of doing this when the alternatives may be much more work.

[1] https://github.com/thheller/reagent-react-native/blob/master/src/main/reagent/dom.cljs

[bed...@yahoo.com]

Changing the order of classpath entry and not having paths be at the beginning of the CLASSPATH will break existing code that uses `(io/resource ...)` to load resources from the CLASSPATH.

While in theory the order of the classpath should not matter, in practice it does, often involuntarily.

Fighweel-main's default behavior is to load `public/index.html` which works fine if the local paths come first, but will fail if any other JAR has a `public/index.html` resource.

And as we learned, they do!

Please consider going back to the old, defined behavior. All other build tools I know have - for good or bad - a defined CLASSPATH order.

Thanks, Jochen

[Alex Miller]

Just to beat the horse...

On Tue, Aug 11, 2020 at 1:05 PM 'bed...@yahoo.com' via Clojure <clo...@googlegroups.com> wrote:

Changing the order of classpath entry and not having paths be at the beginning of the CLASSPATH will break existing code that uses `(io/resource ...)` to load resources from the CLASSPATH.

While in theory the order of the classpath should not matter, in practice it does, often involuntarily.

Just to reiterate so it's clear, this is bad, at least for deps and usually for most other uses. That's why we provide features in deps.edn to encourage you to be specific when you are overriding something that exists in the classpath rather than relying on an ordering mechanism. If there is a use case where shadowing is a feature and not a bug that is not already covered, I'd be interested in hearing about it.

 

Fighweel-main's default behavior is to load `public/index.html` which works fine if the local paths come first, but will fail if any other JAR has a `public/index.html` resource.

And as we learned, they do!

This is bad, and we should loudly complain about it to whoever is doing it, not just accept it. (It's probably accidental, not intentional.)

 

Please consider going back to the old, defined behavior.

I am considering options. 

 

All other build tools I know have - for good or bad - a defined CLASSPATH order.

Do they? I'm not aware of documentation about or guarantees for this with either lein or Maven but would love to be pointed at those docs if you're aware of something. Both Maven and lein have considered how to make builds repeatable/reproducible, but I'm actually not aware that the order is well defined and documented (and I have looked).

 

[bed...@yahoo.com]

Here's some maven-specific discussion: https://stackoverflow.com/questions/793054/maven-classpath-order-issue.

They have a defined order since 2.0.9. and declaration order is considered for transitive dependencies conflict.

Intellij's Dependencies tab in Module settings: You can re-order the dependencies and they reflect in the classpath.

lein classpath -> local paths added first, JARs afterwards

Please consider returning to a :paths first, then :deps in a stable order.

I know it is not pretty and it is not desirable for code to be dependent on that, but resource-loading uses the CLASSPATH and that makes the order of dependencies intrinsically linked to locating resources.

I'd also rather fighweel-main behave differently, but it relies on (io/resource "public/index.html")

Thanks,

 Jochen

[Alex Miller]

On Tue, Aug 11, 2020 at 3:01 PM 'bed...@yahoo.com' via Clojure <clo...@googlegroups.com> wrote:

Here's some maven-specific discussion: https://stackoverflow.com/questions/793054/maven-classpath-order-issue.

They have a defined order since 2.0.9. and declaration order is considered for transitive dependencies conflict.

Unfortunately, neither that 11 year old SO answer nor the referenced jiras actually document, explain, or refer to any documentation about the ordering, or afaict commit to anything specific other than reproducibility. I'm not saying this is your fault or anything, just does not seem well defined to me other than as an artifact of implementation.

For libs, Maven (and I presume lein which relies on Maven libs for this) uses the ordering of deps in the pom wrt the ordering in the classpath. clj intentionally does not include this ordering - the libs are in an unordered map, the version selection algorithm is completely different, etc. If this matters, then one of your deps is broken and should be fixed.

 

Intellij's Dependencies tab in Module settings: You can re-order the dependencies and they reflect in the classpath.

Not sure that has anything to do with Maven or lein, seems orthogonal to the question here.

 

lein classpath -> local paths added first, JARs afterwards

Please consider returning to a :paths first, then :deps in a stable order.

I will consider some options.

 

I know it is not pretty and it is not desirable for code to be dependent on that, but resource-loading uses the CLASSPATH and that makes the order of dependencies intrinsically linked to locating resources. 

I'd also rather fighweel-main behave differently, but it relies on (io/resource "public/index.html")

I think that is perfectly ok - the problem here is whether a jar includes that resource, which is likely to conflict. I'd be very interested to know whether this is actually a jar or an issue with the ordering of your local paths. To check where it's finding index.html:

clj -e "((requiring-resolve 'clojure.java.io/resource) \"public/index.html\")"

[Alan Thompson]

Hi - I just tried your suggestion and no joy:

~/work/tmp810/xanadu >   clj -e "((requiring-resolve 'clojure.java.io/resource) \"public/index.html\")"

DEPRECATED: Libs must be qualified, change deps-ancient => deps-ancient/deps-ancient (deps.edn)

DEPRECATED: Libs must be qualified, change reagent => reagent/reagent (deps.edn)

DEPRECATED: Libs must be qualified, change ns-tracker => ns-tracker/ns-tracker (deps.edn)

DEPRECATED: Libs must be qualified, change camel-snake-kebab => camel-snake-kebab/camel-snake-kebab (deps.edn)

DEPRECATED: Libs must be qualified, change bidi => bidi/bidi (deps.edn)

DEPRECATED: Libs must be qualified, change orchestra => orchestra/orchestra (deps.edn)

DEPRECATED: Libs must be qualified, change cljs-ajax => cljs-ajax/cljs-ajax (deps.edn)

DEPRECATED: Libs must be qualified, change expound => expound/expound (deps.edn)

DEPRECATED: Libs must be qualified, change re-frame => re-frame/re-frame (deps.edn)

DEPRECATED: Libs must be qualified, change re-frame-utils => re-frame-utils/re-frame-utils (deps.edn)

DEPRECATED: Libs must be qualified, change cljs-bean => cljs-bean/cljs-bean (deps.edn)

#object[java.net.URL 0x6c345c5f "file:/Users/alanthompson/work/tmp810/xanadu/resources/public/index.html"]

The call to `requiring-resolve` claims it is finding my local `./resources/public/index.html`.  However, the error remains that it is finding some other `index.html`, which also points to an incorrect JS output file.

Alan

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAOdgdgz88M5jfbSOb2yTkehh3b32uQ6rh0bqa44T7J7hnP7LBQ%40mail.gmail.com.

[bed...@yahoo.com]

Just 2 quick points before I go back to migrate to shadow-cljs & leiningen ;)

"just does not seem well defined "

This is not a line of argument you want to pursue when we are talking about maven, who has had a stable order for what feels like decades now.

If you need a current link: https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies

(search for "first")

To your comment re Intellij: you can actually use Intellij to build and it uses its own project configuration to do that and cobble together the classpath (in many cases it merely syncs with maven/lein etc.). In a way, Intellij acts like clj: It builds a classpath and runs stuff.

Also, arguing that "JARs need to be fixed" is interesting,  as I can get *any* resource with `(io/resource...)` and will get different answers depending on CLASSPATH order. If that order changes in between minor versions of clj, that makes it an unstable build system.

Just try to io-resource '/log4j.properties', for funsies. If in version A of clj I'm getting my own /log4j.properties and in version I'm getting the one from some other JAR, it's a problem.

If I run

find . -name "*.jar" -exec unzip -l {} \; | awk '{print $4}' | sort | less

in ~/.m2

to crudely get all file names for all my cached jars,

I can find 4 instances of 'public/index.html'

3 of 'public/css/style.css' etc. etc.

Please return to at least having :paths at the front of the CLASSPATH.

Thanks,

 Jochen

[Alex Miller]

Can you change that resources file and see if what you're looking at changes to double check? I did actually check a bunch of jars from the prior message and did some searching for that message. Or you could even dump the classpath with -Spath, then move resources to the front, then use -Scp (which will force the classpath you say, ignoring everything else).

Or could be that it's not the index.html but something it refers to getting picked up from elsewhere?

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/WI3ddZRK4Bg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAN67zA36F4Hsobu%3DOt4-DPrmcxsHVK9f%3DsUo1p9g1shg%3DZaOuQ%40mail.gmail.com.

[Alex Miller]

On Tue, Aug 11, 2020 at 6:41 PM 'bed...@yahoo.com' via Clojure <clo...@googlegroups.com> wrote:

Just 2 quick points before I go back to migrate to shadow-cljs & leiningen ;)

"just does not seem well defined "

This is not a line of argument you want to pursue when we are talking about maven, who has had a stable order for what feels like decades now.

If you need a current link: https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies

(search for "first")

I understand it may feel like that to you, but I have been using Maven for at least 15 years and have spent a lot of time fighting classpath ordering issues while using it. That link says nothing about classpath order and I spent some more time today looking for something like that, without success. There are plenty of Maven tickets about this, it's changed over the years, and a lot of complexity to cover when you start to consider parent poms, dependency management, etc. I don't think it's as fixed as you think it is, probably more that they just don't touch it much (and most differences don't actually matter). It may be stable largely by fear.

 

To your comment re Intellij: you can actually use Intellij to build and it uses its own project configuration to do that and cobble together the classpath (in many cases it merely syncs with maven/lein etc.). In a way, Intellij acts like clj: It builds a classpath and runs stuff.

Yeah, I use it every day. My point was just that it's independent from what we're talking about.

 

Also, arguing that "JARs need to be fixed" is interesting,  as I can get *any* resource with `(io/resource...)` and will get different answers depending on CLASSPATH order.

You can only get different answers if two roots have the same resource, which again, is bad, and the cause of a big percentage of dependency issues I've tracked down at one point or another. Same resource = shadowing = eventually you will reorder your deps and see something different.

 

If that order changes in between minor versions of clj, that makes it an unstable build system.

I'd say if that's coming from dependencies, then it's your build deps that are unstable, you just didn't know it. The ordering of project paths before deps is a separate issue.

 

Just try to io-resource '/log4j.properties', for funsies. If in version A of clj I'm getting my own /log4j.properties and in version I'm getting the one from some other JAR, it's a problem.

If I run

find . -name "*.jar" -exec unzip -l {} \; | awk '{print $4}' | sort | less

in ~/.m2

to crudely get all file names for all my cached jars,

I can find 4 instances of 'public/index.html'

3 of 'public/css/style.css' etc. etc.

You are proving my point. That's all broken. Those libs cannot control where they are placed in an application's classpath and whether they will shadow or be shadowed by something else. They are time bombs waiting to break your build. You should complain to their creators.

 

Please return to at least having :paths at the front of the CLASSPATH.

As I said, that's a different issue and I'm inclined to agree with you. The details of changing it are more complicated than it appears.

 

[Alex Miller]

FYI, I've been working with the homebrew team today and homebrew-core now points to the latest stable clojure tools version and will track that automatically going forward.

[bed...@yahoo.com]

Ok, let's reset as we start talking about different things now.

None of these libraries are broken. They just include resources. Also, I don't think it is realistic to tell library authors to please move certain files out of the way because my build tool randomizes my classpath. That is not going to happen. People will keep including things like log4j.properties that are in potential conflict with a local file or files in other JAR paths.

The question remains: Should my build tool allow me control over the order of the classpath? I believe the answer is yes.

(Even gradle gives you that control and/or offers ways to ensure that my chosen paths and libraries are first on the classpath.

Regarding maven: Even if there would be no documentation at all, their algorithm has not changed since 2.0.9. It's the Normative Kraft des Faktischen at play. )

If the build tool from version A to version B changes this order, it is not a usable tool. I can't make this point any clearer.

We might agree to disagree here, but a build tool should not be at liberty to sort top-level dependencies at will.

Note that this is completely independent from the thought that order *shouldn't* matter.

When in fact it does matter. It is an unfortunate reality. That's simply how resource loading works in Java.

I understand that building dependency trees is not trivial, but it is rather trivial to ensure :paths comes before everything else.

Please define some stable order and establish a top-level order for explicit dependencies. That makes builds deterministic.

Thanks for looking in this issue,

 Jochen

PS: happy to learn you got in touch with the homebrew team. That is excellent news.

[Thomas Heller]

None of these libraries are broken. They just include resources. Also, I don't think it is realistic to tell library authors to please move certain files out of the way because my build tool randomizes my classpath. That is not going to happen. People will keep including things like log4j.properties that are in potential conflict with a local file or files in other JAR paths.

These packages should be cleaned up. They contain files they shouldn't contain. It mostly goes unnoticed and library authors often aren't even aware this is happening. I have reported this to a couple CLJS libraries over the years and all of them were fixed pretty much immediately, since it was always accidental. The burden is probably on the community to make people aware of this. Heck I'd even say all tools used for building library .jar files should even complain about certain files from the outset. While it is mostly harmless and usually goes unnoticed it is also a potential security risk. I don't know anyone that audits their dependencies properly and often people just serve "public" resources straight over HTTP. That means any dependency you don't audit may contain "public/some-exploit/foo.html" which then is often automatically available under "https://your-domain.com/some-exploit/foo.html". Probably not something you want to have on your domain.

Regardless :paths should be first, just because of the options it enables I outlined earlier. IMHO dependencies otherwise can stay unordered since everything should be namespaced properly and unique anyways so order shouldn't matter.

[Sean Corfield]

I'd like to point out that tools.deps.alpha can build a differently-ordered classpath from the exact same input files when run on a different machine (i.e., the same source/deps, the same version of t.d.a./CLI).

This has bitten me a couple of times with depstar which I use for building JARs/uberjars (and which I maintain) because a JAR built on one machine can behave differently than a JAR built on another machine -- because of classpath ordering of things that overlap on the classpath. This is completely separate from the :paths issue (with which I have some sympathy).

The most recent "gotcha" for me was with log4j2 which has a Log4J2Plugins.dat file containing a cache of the plugins that control formatting in log files, so any library that includes formatter plugins tends to have its own copy of this file. There have been a couple of long-standing bug reports against Log4J2 because of this -- because people in Maven-land keep tripping over classpath ordering issues and finding logging breaks "randomly" in their applications. Years of complaints and the official answer is "We'll stop doing the plugins cache thing in log4j 3". There is a Maven plugin that knows how to merge Log4J2Plugins.dat files but, of course, it is specific to log4j2 and Maven. There is a Leiningen plugin for it as well -- also specific to log4j2 code -- because Leiningen is built on the same "reproducible" base as Maven...

So this _is_ an issue in Maven-land. It's an acknowledged problem with libraries that contain overlapping files and has been for years. That t.d.a/CLI makes this a bit more visible is a positive, in my opinion, because you're more likely to realize that you have a problem upstream with badly-behaved dependencies.

p.s., My "hack" for depstar was to have it overwrite Log4J2Plugins.dat in the JAR until it found one bigger than ~10K and then ignore subsequent ones (the plugin cache file in the main log4j2 JAR is 20K, the plugin cache files in all the other libs I've seen are 5K or smaller). Sure, that means you don't get _all_ the formatter plugins, you only get the core ones, but in the real world that is almost always going to be the right choice.

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/81e969d2-91d2-4fc3-bdfd-8e509fc304bdn%40googlegroups.com.

--

Sean A Corfield -- (904) 302-SEAN
An Architect's View -- https://corfield.org/
World Singles Networks, LLC. -- https://worldsinglesnetworks.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Alan Thompson]

Hi Alex,

1. Great news that the Homebrew team has responded to your request to point only to stable versions.

2. The resources directory is the contains the path `resources/public/index.html`.  The local one is definitely not the one being served in my original example (2nd half) re 1.10.1.645 when the `resources` dir was listed near the end. The `index.html` being found by the Figwheel.Main server had different contents, and was pointing to a different *.js executable file (possibly in a different *.jar file?).  That is the source of the 2-word `Debux Test` webpage.

3. Unless I'm missing something, I believe we have already run your suggested test. Using 1.10.1.561 from `brew install clojure/tools/clojure`, I got the `resources` and `target` dirs as items #1 and #2 on the classpath; everything worked as expected.  Using 1.10.1.645 from `brew install clojure`, `resources` was near the end of the classpath and it failed (I didn't track down where the `target` dir wound up in that instance).

4. I could possibly try to replicate your proposed experiment explicitly, but I no longer have easy access to 1.10.1.645 since Homebrew has been fixed.  I did find the `brew-install` repo on GH, but am not certain how to replicate the broken install of *.645.

Thank you for the attention you are giving to this issue.

Alan

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAOdgdgzMs1bscsnYAj6M_ksFpKHtOkhMKnOfLsFm1_hDFwUqoQ%40mail.gmail.com.

[Sean Corfield]

Alan,

> 4. I could possibly try to replicate your proposed experiment explicitly, but I no longer have easy access to 1.10.1.645 since Homebrew has been fixed.  I did find the `brew-install` repo on GH, but am not certain how to replicate the broken install of *.645.

Per the clojure/homebrew-tools readme: https://github.com/clojure/homebrew-tools#version-archive-tool-releases

brew install clojure/tools/clo...@1.10.1.645

You may also need:

brew link --overwrite clo...@1.10.1.645

This allows you to easily switch back and forth between different versions for testing.

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAN67zA2N436wYz%3DQfaXn70XWFHTXmNFr3q_8cTfSRJamtpV3NA%40mail.gmail.com.

[Matching Socks]

Too fragile.  This reminds me of the notion of "situated programming", featured in the talk by Rich Hickey:  you and your programs operate in the middle of a bizarre and changing situation.  For Clojure, the Java ecosystem is part of that situation.  Even if some jars do not overlap today, you will be forced to take a minor update someday that introduces a clash.  Or perhaps (quite likely) jars do overlap today, but you will take a minor update someday that causes the classpath to emerge from the hash-map differently and your program won't work anymore.  The insight of the theory of "situated programs" is, not to hit a cliff when a perfectly legal quirk arises in the situation.

[bed...@yahoo.com]

I couldn't agree more.

It really boils down to the simple fact that class loading in the JVM - for the standard classloaders - is well-defined.

If a build tool is not able to reflect this, it is an unreliable build tool. It doesn't matter if anyone thinks CLASSPATH order shouldn't matter from a philosophical point of view. (And inconveniencing deps.edn users to go tell the authors of dependencies to "fix their resources" is not a realistic approach)

Providing dependencies in an unsorted map like deps.edn is doing now is inherently unstable.

That doesn't necessarily make deps.edn unusable, but - as others have pointed out - a source of surprising errors.

Until an order is defined that survives changing OS/JVMs, I will advise anyone to not use deps.edn.

Furthermore, all build tools - with the exception of Bazel and derivates - do rely on transitive dependency resolution and thus also are inherently fragile.

But either through established practices or defined standards: they offer a way to define an order should conflicts arise.

(Take a look at https://issues.apache.org/jira/browse/MNG-1412 again for a longer discussion).

At the very least: The location of your own classpath entries must be well-defined. They are by default first on regular CLASSPATH.

I wasn't following the latest development on this, but I would assume that is a reasonable demand and has been fixed in newer tools.deps versions?

Thanks,

 Jochen

[Alex Miller]

Just FYI, we have a plan to address this and it should be in the next stable version.

On Sep 14, 2020, at 1:00 AM, 'bed...@yahoo.com' via Clojure <clo...@googlegroups.com> wrote:



--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/WI3ddZRK4Bg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/e229b29e-304d-4e96-bf26-59b255e3920bn%40googlegroups.com.

[Alex Miller]

These changes are in the new 1.10.1.693 prerelease of clj if you want to test it. Assuming things look good, I'll promote that to a stable release soon.

In this release, things will be ordered :extra-paths (in order, if used), then :paths (in order, if used), then libs. That reverts the changes in the release that led to the issue here.

Additionally, libs are now sorted by tree depth (roots first, then their deps, etc), then alpha per depth so this should greatly improve reproducibility of lib ordering across builds.

[bed...@yahoo.com]

This is excellent news! Thank you very much!