Issue when moving to Clojure 1.10

[Didier]

When compiling with Clojure 1.10, I get the following error, which I did not use to get on 1.7, 1.8 and 1.9:

Exception in thread "main" Syntax error compiling at (com/org/namespace_test.clj:1:1).
  at clojure.lang.Compiler.load(Compiler.java:7647)
  at clojure.lang.RT.loadResourceScript(RT.java:381)
  at clojure.lang.RT.loadResourceScript(RT.java:372)
  at clojure.core$load$fn__6824.invoke(core.clj:6126)
  at clojure.core$load.invokeStatic(core.clj:6125)
  at clojure.core$load.doInvoke(core.clj:6109)
  at clojure.lang.RestFn.invoke(RestFn.java:408)
  at clojure.core$load_one.invokeStatic(core.clj:5908)
  at clojure.core$load_one.invoke(core.clj:5903)
  at clojure.core$load_lib$fn__6765.invoke(core.clj:5948)
  at clojure.core$load_lib.invokeStatic(core.clj:5947)
  at clojure.core$load_lib.doInvoke(core.clj:5928)
  at clojure.lang.RestFn.applyTo(RestFn.java:142)
  at clojure.core$apply.invokeStatic(core.clj:667)
  at clojure.core$load_libs.invokeStatic(core.clj:5985)
  at com.org.ant.clojure.test_runner$exec_tests.invoke(test_runner.clj:10)
  at clojure.lang.RestFn.invoke(RestFn.java:397)
  at clojure.lang.AFn.applyToHelper(AFn.java:152)
  at clojure.lang.RestFn.applyTo(RestFn.java:132)
  at com.org.ant.clojure.TestRunner.main(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:348)
  at clojure.lang.RT.classForName(RT.java:2207)
  at clojure.lang.RT.classForName(RT.java:2216)
  at clojure.lang.RT.loadClassForName(RT.java:2235)
  at clojure.lang.RT.load(RT.java:453)
  at clojure.lang.RT.load(RT.java:428)
  at clojure.lang.Compiler.ensureMacroCheck(Compiler.java:6957)
  at clojure.lang.Compiler.checkSpecs(Compiler.java:6969)
  at clojure.lang.Compiler.macroexpand1(Compiler.java:6987)
  at clojure.lang.Compiler.macroexpand(Compiler.java:7074)
  at clojure.lang.Compiler.eval(Compiler.java:7160)
  at clojure.lang.Compiler.load(Compiler.java:7635)
  ... 30 more
Caused by: clojure.lang.ArityException: Wrong number of args (3) passed to: clojure.spec.alpha/amp-impl
  at clojure.lang.AFn.throwArity(AFn.java:429)
  at clojure.lang.AFn.invoke(AFn.java:40)
  at clojure.core.specs.alpha$fn__47.invokeStatic(alpha.clj:107)
  at clojure.core.specs.alpha$fn__47.invoke(alpha.clj:107)
  at clojure.core.specs.alpha__init.load(Unknown Source)
  at clojure.core.specs.alpha__init.<clinit>(Unknown Source)
  ... 43 more

Anyone knows why?

Here's the code at 1:1 :

(ns com.org.namespace-test
  (:require [clojure.test :as test :refer :all]
            [com.org.namespace :as namespace])
  (:import [com.org.persistence Response]
           [com.org.encrypt Encrypter]))

[Alex Miller]

That file looks fine to me - can you post what libraries/classpath you're using? I can't repro this with 1.10.

[Matching Socks]

This might have nothing to do with it, but, doesn't :import use parens?

[Alex Miller]

:import works with either (personally I prefer []).

On Jan 15, 2019, at 7:00 PM, Matching Socks <phill...@gmail.com> wrote:

This might have nothing to do with it, but, doesn't :import use parens?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/qIA7GoAVtbE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Didier]

Okay, I found my issue, mostly.

It seems that Clojure 1.9 and 1.10 both now have a clojure.core.specs.alpha.clj file.

This file seems to be the only file that is not AOT compiled by the maven compile script for Clojure. Thus the Jar for Clojure only has it as source, and is missing the .classes for it.

We have custom Ant tasks to compile Clojure, where we call compile-ns ourselves on our source namespaces.

I was using Clojure 1.9 in our Ant tasks, which meant that as part of running compile-ns on our namespace, because they depend on Clojure core as well, clojure.core.specs.alpha.clj from the Clojure 1.9 jar was also being compiled.

Thus, our project jar now contained the 1.9 classes for core specs.

Now, we would use Clojure 1.10 to run our project, but the Clojure 1.9 core specs fail to validate amp-impl of Clojure 1.10. I'm guessing it was changed along with its spec inside core specs.

So we were getting a weird syntax error at runtime for every namespace that depend on clojure.core.

I have a few questions based on this:

1) Why is clojure.core.specs.alpha.clj the only thing not AOT compiled in the Clojure jar?

2) I realized that I probably shouldn't compile Clojure 1.10 code with Clojure 1.9, and I bumbed our Ant code to use Clojure 1.10, and that fixed it. So what are the compiler compatibility? Can newer Clojure versions compile older Clojure code? Is it only two version back? Is this tested? Or should I always use equal versions to compile? Is forward compatibility a thing? Etc.

3) Is compile-ns transitive? I was surprised to see that it also compiles clojure.core.specs.alpha.

4) Isn't it a bit strange that if I AOT compile my code, my Jar gets .class files from Clojure core as well? Even though I don't get the error anymore, I still have clojure.core.specs.alpha classes from my jar on the classpath, and the source from Clojure on it as well.

Thanks

[Alex Miller]

On Tue, Jan 15, 2019 at 10:43 PM Didier <did...@gmail.com> wrote:

Okay, I found my issue, mostly.

It seems that Clojure 1.9 and 1.10 both now have a clojure.core.specs.alpha.clj file.

This file seems to be the only file that is not AOT compiled by the maven compile script for Clojure. Thus the Jar for Clojure only has it as source, and is missing the .classes for it.

We have custom Ant tasks to compile Clojure, where we call compile-ns ourselves on our source namespaces.

I was using Clojure 1.9 in our Ant tasks, which meant that as part of running compile-ns on our namespace, because they depend on Clojure core as well, clojure.core.specs.alpha.clj from the Clojure 1.9 jar was also being compiled.

Thus, our project jar now contained the 1.9 classes for core specs.

Now, we would use Clojure 1.10 to run our project, but the Clojure 1.9 core specs fail to validate amp-impl of Clojure 1.10. I'm guessing it was changed along with its spec inside core specs.

So we were getting a weird syntax error at runtime for every namespace that depend on clojure.core.

I have a few questions based on this:

1) Why is clojure.core.specs.alpha.clj the only thing not AOT compiled in the Clojure jar?

clojure.core.specs.alpha is actually not part of Clojure proper, it's in the dependencies. We don't compile the specs as we would then be bound to a particular version of the spec.alpha dependency. We wanted it to be possible to bump those versions but not have to be binary-compatible. (And this is exactly the issue you're seeing - binary incompatibility of compiled specs between versions.)

There are actually a few other namespaces in Clojure itself that are not AOT compiled because they are conditionally included, or for other reasons. Some of these special cases went away in Clojure 1.10 due to the bump in minimum JDK which removed a lot of the special cases.

 

2) I realized that I probably shouldn't compile Clojure 1.10 code with Clojure 1.9, and I bumbed our Ant code to use Clojure 1.10, and that fixed it. So what are the compiler compatibility? Can newer Clojure versions compile older Clojure code? Is it only two version back? Is this tested? Or should I always use equal versions to compile? Is forward compatibility a thing? Etc.

I'm not sure you've accurately described the scenario here. In general, we don't guarantee binary compatibility between versions of Clojure, however, we do make every effort to maintain stability of the interfaces that compiled Clojure classes rely on inside the language (all the Java core like RT, etc). However, I don't think the problem here is with Clojure itself, it's with spec.

What you're seeing are specs compiled with one version of spec.alpha, then being run with a different version of spec.alpha. We have not been trying to maintain binary compatibility for compiled specs (and have not been compiling the core.specs to avoid this being an issue). Protocols tend to be particularly problematic in this area (the spec interface is a protocol), so that's a complicating factor here.

 

3) Is compile-ns transitive? I was surprised to see that it also compiles clojure.core.specs.alpha.

Yes. When compilation is on, it transitively compiles every namespace loaded.

4) Isn't it a bit strange that if I AOT compile my code, my Jar gets .class files from Clojure core as well? Even though I don't get the error anymore, I still have clojure.core.specs.alpha classes from my jar on the classpath, and the source from Clojure on it as well.

Yes, it's one of the downsides of AOT. This has been discussed for a long time (see https://dev.clojure.org/jira/browse/CLJ-322 for example). Having both source and classes is totally fine - Clojure prefers the class file, but there are issues (particularly with protocols) in having classes that depend on source files.

 

[Didier]

Okay, I think I have a good understanding now.

Internally, I was using the Clojure build which bundles the dependency with it. That's why that namespace shows up inside my Clojure Jar.

It makes sense also not to guarantee binary compatibility. We will make sure to always build and run using the same version of Clojure from now on.

Is source compatibility guaranteed to some extent? Lile Clojure 1.10 compiling Clojure 1.8 source? I'd assume if so, its only guaranteed backward, like old Clojure is not guarantee to support newer Clojure code?

[Mark Derricutt]

On 16 Jan 2019, at 18:17, Alex Miller wrote:

Yes, it's one of the downsides of AOT. 

I'd say it's not so much a downside of AOT - but of having a single output path for classes. I've long wanted Chas's patch to be applied, or something like it.

Having everyone reinvent the mechanism whenever they happen to need AOT is kind of annoying - rare, but still annoying.

---

"The ease with which a change can be implemented has no relevance at all to whether it is the right change for the (Java) Platform for all time." — Mark Reinhold.

Mark Derricutt
http://www.theoryinpractice.net
http://www.chaliceofblood.net
http://plus.google.com/+MarkDerricutt
http://twitter.com/talios
http://facebook.com/mderricutt

[Didier]

So I got to the bottom bottom of the problem here.

This is a scenario:

1) Library A depends on library B and Clojure 1.10

2) Library B must be AOTed due to a gen-class, but depends on Clojure 1.9

3) Library A does not work, because it is now using the Clojure core spec 1.9 compiled transitively through the Library B AOT, and found in its Jar, since .class get precedence over source files.

So, this means that a library with a dependency on another one that is AOT can cause spec conflicts.

This is new now that spec is out. And so I first caught this when some of our libs were using 1.9, and I started moving others to 1.10.

The way I solved this is by hacking our build, so that the clojure compiled classes from the AOT don't get included in the Jar after they are compiled.

In my opinion, this is a bit of a problem. What would be nice is either to have a way to compile only a gen-class, and not the namespace that contains it. Or not compile things transitively. Or maybe just a way for compile to exclude clojure core namespaces.

Now, if you depend on libraries that don't need AOT, it is not an issue. But if you do, it forces you to re-compile all your dependencies and do a full big bang upgrade to the new Clojure version. You can't just use libs compiled with older versions of spec. While before, you were able to use libs compiled with older version of Clojure from packages that use newer version of Clojure.

[Didier]

Okay, so after reading through the linked issue here: https://dev.clojure.org/jira/browse/CLJ-322 I'm not sure, as a tool builder, what is the ideal path forward.

This is what I seem to understand would be ideal, let me know if I'm wrong:

1) AOT compile everything. So basically, always AOT every lib. This is to make sure that we don't face this issue: https://dev.clojure.org/jira/browse/CLJ-1544

2) Figure out some mechanism within the built tooling to only include the classes for which the source exist in the current project being built into the resulting Jar. Meaning, only namespaces found in the source folder should have their compiled classes be included into the Jar.

This setup should minimize conflicts, from what I understand. The only thing I'm not sure about is, are classes compiled with an older version of Clojure and JDK always going to work with a newer version of Clojure and JDK?

[Alex Miller]

On Jan 26, 2019, at 12:43 AM, Didier <did...@gmail.com> wrote:

Okay, so after reading through the linked issue here: https://dev.clojure.org/jira/browse/CLJ-322 I'm not sure, as a tool builder, what is the ideal path forward.

This is what I seem to understand would be ideal, let me know if I'm wrong:

1) AOT compile everything. So basically, always AOT every lib. This is to make sure that we don't face this issue: https://dev.clojure.org/jira/browse/CLJ-1544

I think the key is that if you are going to AOT, then everything you depend on should be AOTed. The ideal place to do this is when you build the final application - uberjar and aot the world. To do so, it’s best if libs never AOT to give app compilers the opportunity to do so.

2) Figure out some mechanism within the built tooling to only include the classes for which the source exist in the current project being built into the resulting Jar. Meaning, only namespaces found in the source folder should have their compiled classes be included into the Jar.

You can avoid this by a) not aot compiling or b) compiling only when dependent on things that are compiled, in which case new classes are only your lib. But some tools (like the Maven Clojure plugin) do have the ability to filter which aot’ed classes you include.

This setup should minimize conflicts, from what I understand. The only thing I'm not sure about is, are classes compiled with an older version of Clojure and JDK always going to work with a newer version of Clojure and JDK?

As much as possible, yes. But we don’t guarantee it. So again, make the compile decision as late as possible.

On Friday, 25 January 2019 22:11:46 UTC-8, Didier wrote:

So I got to the bottom bottom of the problem here.

This is a scenario:

1) Library A depends on library B and Clojure 1.10

2) Library B must be AOTed due to a gen-class, but depends on Clojure 1.9

3) Library A does not work, because it is now using the Clojure core spec 1.9 compiled transitively through the Library B AOT, and found in its Jar, since .class get precedence over source files.

So, this means that a library with a dependency on another one that is AOT can cause spec conflicts.

This is new now that spec is out. And so I first caught this when some of our libs were using 1.9, and I started moving others to 1.10.

The way I solved this is by hacking our build, so that the clojure compiled classes from the AOT don't get included in the Jar after they are compiled.

In my opinion, this is a bit of a problem. What would be nice is either to have a way to compile only a gen-class, and not the namespace that contains it. Or not compile things transitively. Or maybe just a way for compile to exclude clojure core namespaces.

Now, if you depend on libraries that don't need AOT, it is not an issue. But if you do, it forces you to re-compile all your dependencies and do a full big bang upgrade to the new Clojure version. You can't just use libs compiled with older versions of spec. While before, you were able to use libs compiled with older version of Clojure from packages that use newer version of Clojure.

On Wednesday, 16 January 2019 20:47:43 UTC-8, Mark Derricutt wrote:

On 16 Jan 2019, at 18:17, Alex Miller wrote:

Yes, it's one of the downsides of AOT. 

I'd say it's not so much a downside of AOT - but of having a single output path for classes. I've long wanted Chas's patch to be applied, or something like it.

Having everyone reinvent the mechanism whenever they happen to need AOT is kind of annoying - rare, but still annoying.

---

"The ease with which a change can be implemented has no relevance at all to whether it is the right change for the (Java) Platform for all time." — Mark Reinhold.

Mark Derricutt
http://www.theoryinpractice.net
http://www.chaliceofblood.net
http://plus.google.com/+MarkDerricutt
http://twitter.com/talios
http://facebook.com/mderricutt

--

[Sean Corfield]

2) Library B must be AOTed due to a gen-class, but depends on Clojure 1.9

 

My approach is to isolate the pieces that need to be AOT compiled to as few namespaces as possible and have them only depend on other namespaces at runtime, i.e., avoid :require in the ns clause where possible and use require/resolve in the AOT-compiled functions to get at the larger body of Clojure code. And, of course, to try to avoid gen-class as much as possible too 😊

 

The transitive AOT compilation thing is a giant pain but there are ways around it with tooling.

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

---

From: clo...@googlegroups.com <clo...@googlegroups.com> on behalf of Didier <did...@gmail.com>
Sent: Friday, January 25, 2019 10:11:45 PM
To: Clojure
Subject: Re: Issue when moving to Clojure 1.10

 

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Didier]

I think the key is that if you are going to AOT, then everything you depend on should be AOTed. The ideal place to do this is when you build the final application - uberjar and aot the world. To do so, it’s best if libs never AOT to give app compilers the opportunity to do so.

The build systems we rely on do not re-build dependent libraries unfortunately, and we don't use uberjar for our deployments either. Which means I can't delay the AOT to the last possible minute. Also, a lot of mixed projects happen, where Java needs the Clojure compiled classes at compile time. So these also can't be delayed AOT.

b) compiling only when dependent on things that are compiled

This is what we used to do, as we would compile everything, except Clojure now depends on something that isn't compiled, core specs, which causes conflicts. 

I'll modify our build tooling to somehow filter classes here, I already excluded the Clojure directory, so that solved our problem related to Clojure. Now I'm just wondering what is best going forward.

The transitive AOT compilation thing is a giant pain but there are ways around it with tooling.

With tooling or with code changes? Because you mentioned writing your namespaces in a way so that gen-classes limit their AOT imact as much as possible. My issue is, I work at a very big company, and I maintain the Clojure build tools internally. I can't guide everyone to refactor their code to isolate their gen-classes, so I need to make the build tool as resilient as possible to all scenarios, assuming no control on the code.

So I have two options:

1) AOT everything all the time, but only include the current project classes in the resulting Jar.

2) AOT as little as possible, basically only namespaces with gen-class in them. And also only include the current project classes in the resulting Jar. Unless there is a dependency with protocol, now these should be transiently compiled as well, so I need to give a build options so people could specify to include certain transient namespaces into the Jar, in the case of protocols.

I'm personally leaning on one, as it seems like it should work in all cases. Am I forgetting something?

Regards!

[Didier]

For anyone interested, the approach I ended up taking was to AOT everything all the time. But, require the class namespaces to be explicitly defined in the build config file for any which needs to be included in the Jar. And when packaging the Jar, only the Clojure source and the AOT class files of the namespaces that were explicitly listed are included.

I do that by compiling every source files, and then deleting all .class files except for the ones that match the explicitly listed namespaces. If no namespace are listed, then the resulting Jar is thus similar to a non AOT build, and only includes source.

The advantages are that the AOT compilation bubbles compile issues early at build time. And the resulting Jars do not include transitive classes that could cause version conflicts. And when you need gen-class, you can include only the gen-class .class in the Jar, not even have its containing namespace. Minimizing the blast radius of AOT. And if you face the protocol issue, you can specify these namespaces as well to ne included in the Jar.