maven-shade-plugin issues with Clojure core (or any clj) AOT compilation

[Mike Rodriguez]

Background:

This problem is specific to building jars that contain AOT (Ahead Of Time) compiled Clojure code using Maven and the maven-shade-plugin.

Clojure AOT compilation depends on timestamps of .class files vs .clj files being accurate.  When both .class files and their associated .clj files exist, the AOT .class files are only used by the compiler if their last modified timestamp is strictly greater than the last modified timestamp of the associated .clj file.

Also note that the Clojure core jar itself is deployed AOTed.

I know that much of the Clojure ecosystem uses Leiningen as a build tool (and boot now too I guess).  This problem doesn't apply to Leiningen (and I haven't looked at boot).

Problem:

The maven-shade-plugin is popular for building shaded/uber/standalone jars in Maven.  Typically this means the jar will include some/all of its dependency jars' files.  The maven-shade-plugin has an unfortunate property though.  It does not preserve the timestamps on files that are added to this final shaded jar.  The resulting jar actually ends up with all files inside of it having the same timestamp (when the jar was created).  In particular, if you originally had AOT Clojure .class files and .clj files with different last modified timestamps, now they will have the same timestamps in the shaded jar.

I've brought this up before @ http://stackoverflow.com/questions/19594360/preserving-timestamps-on-clojure-clj-files-when-building-shaded-jar-via-maven-s

I have rarely seen people bring up issues around this with maven-shade-plugin beyond this particular case.  I believe I have seen a complaint or two around the timestamp loss during shading (I can't find them now), but nothing that has gained any traction (I may try to bring it up to the plugin people soon though).

When the AOTed .class file is ignored in favor of the .clj file, the namespace is JIT (Just-In-Time) compiled.  There are several issues with this.

1) Performance:  It makes the AOT files mostly worthless since they are not saving you on startup time costs anymore.  Everything is JITed anyways.

2) Errors:  The reloading of the .clj files is a forced reload of the .clj namespaces involved.  This can cause classpath clashes among ClassLoaders.

- There are quite a few CLJ Jiras out there that faced trouble dealing with the mix of reloading namespaces and AOT compilation.

You may be thinking, "Just don't build your Clojure jars with Maven.  Use Leiningen instead perhaps?"

This is fine when it is something you control.  However, what if I want to use Clojure to develop a library that may be consumed by Java consumers who very likely will be using Maven.  If my library is going to be shaded into a standalone jar with maven-shade-plugin by this Java consumer (again, likely) then this scenario can happen.

Another thought that may occur is to just avoid AOT compilation of my library application to avoid the problem (this is recommended in the community I believe).  However, Clojure core is AOT compiled and it will get included in the shaded jar.  That alone is enough to cause the issue.

Example:

I have a GitHub repo to show a minimum example of where this can be a problem.  In particular it shows a way for the problem (2) to occur.

@ https://github.com/mrrodriguez/mvn-shade-test

This repo has a Java API through shade.ShadeJava that will cause the Clojure compiler to require the shade.main namespace using JIT compilation.  However, shade.main uses clojure.pprint, which is AOT compiled via Clojure core.

clojure.pprint was chosen here just because it one of the cases I've seen come up that actually fail with problem (2) from above.  Even if there were no failures though, the Clojure core namespaces would be getting recompiled with problem (1).

[Gary Trakhman]

My workaround in a multi-module maven shaded java project with a clojure module was to strip out CLJ files in the top-level build (shipped with clojure.core jar). 

I had also stripped CLJ files from my project artifact, but AOT compiles classfiles from all referenced namespaces, so I attempted to strip those out too earlier with the maven jar plugin. This necessitated manually whitelisting AOT'd protocol classes from some of those 3rd party deps to resolve further classloader issues.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Rodriguez]

I appreciate your input.

When we have controlled the Maven shade projects, we have been able to workaround the issue as well.  We actually add a org.codehaus.mojo/exec-maven-plugin step where we re-open the shaded jar, create a new jar, and re-add every file into the new jar via a script.  While re-adding the files, we change all .clj files' timestamps to some time in the past and all other files just get the current time.  This ensures that all .clj files will look "older" than any AOT .class file that happens to be in the jar.

My main problem statement is primarily concerning when you *do not* control the build.  If I have a Clojure lib that is not-shaded and not AOT-compiled and that I have a Java API exposed on (which is what I demonstrate in my example project as shade.ShadeJava), this problem will come up for any Java consumer who happens to use maven-shade-plugin.  In this sort of case it reflects poorly on Clojure if arbitrary consumers have to do workarounds in their shade builds just because in my lib I'm wanting to use Clojure as an "implementation detail".

So I bring up this issue to just get a sense of what others think and what others think should/could be done about it.  Perhaps using Clojure behind Java APIs is just not a very common use-case people are working with.  It has been a pretty common one for me.  It does seem like a good use-case though to be supported to allow Clojure to sneak into existing Java-centric ecosystems.

[Mike Rodriguez]

I also forgot to include the error message from my example GitHub project, in case it is easier than actually running it.  I understand this issue for the most part.  It has still been a bit tricky for me to fully understand how this AOT class interface/class is being referred to by any Clojure ClassLoader later on causing this.  I dug some and "sort of" understood at this point.  I know there have been many AOT + forced reloading of certain Clojure namespace issues in the past.  This one doesn't directly look like one I've seen.

However, remember, this isn't the only problem I'm bringing up here.  It is that AOT files in general would be ignored after shading with maven-shade-plugin.

The errors:

Exception in thread "main" java.lang.ClassCastException: clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 cannot be cast to clojure.pprint.PrettyFlush

        at clojure.pprint$pretty_writer$fn__4700.invoke(pretty_writer.clj:391)

        at clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848.flush(Unknown Source)

        at clojure.core$flush.invokeStatic(core.clj:3609)

        at clojure.core$flush.invoke(core.clj:3603)

        at clojure.core$prn.invokeStatic(core.clj:3620)

        at clojure.core$prn.doInvoke(core.clj:3612)

        at clojure.lang.RestFn.invoke(RestFn.java:397)

        at clojure.pprint$pprint.invokeStatic(pprint_base.clj:252)

        at clojure.pprint$pprint.invoke(pprint_base.clj:241)

        at clojure.pprint$pprint.invokeStatic(pprint_base.clj:245)

        at clojure.pprint$pprint.invoke(pprint_base.clj:241)

        at shade.main$_main.invokeStatic(main.clj:6)

        at shade.main$_main.doInvoke(main.clj:4)

        at clojure.lang.RestFn.invoke(RestFn.java:397)

        at clojure.lang.Var.invoke(Var.java:375)

        at shade.ShadeJava.main(ShadeJava.java:14)

[Alex Miller]

This seems like a pretty straightforward bug with maven-shade-plugin not preserving information that it should, so it seems like it should be fixed there.

On Wednesday, December 28, 2016 at 7:06:11 AM UTC-6, Mike Rodriguez wrote:

[Mike Rodriguez]

Thanks for the feedback.  I will try to track this issue via the maven-shade-plugin then.  Hopefully there can be some momentum to change it there.

[Philip Aston]

I just tripped over this and opened https://issues.apache.org/jira/browse/MSHADE-272 . Affected users might like to vote it up.

FWIW, my workaround:

            <plugin>

                <artifactId>maven-shade-plugin</artifactId>

                <executions>

                    <execution>

                        <configuration>

...

                            <filters>

                                <filter>

                                    <artifact>*:*</artifact>

                                    <includes>

                                        <include>**/*.class</include>

                                        <include>**/*.properties</include>

                                        <!-- Conditionally loaded, clojure.jar doesn't have a compiled version. -->

                                        <include>clojure/core_instant18.clj</include>

                                    </includes>

                                </filter>

                            </filters>

                        </configuration>

                    </execution>

                </executions>

            </plugin>