java/clojure compilation dependencies

[cburroughs]

I'm using the standard <executions> snippet [1] to cause
clojure:compile. This succeeds in invoking clojure:compile after the
standard java compilation. The problem is that the java code depends
on clojure gen-classes, and thus fails to compile. Is there a
workaround to this problem? Is there a general maven solution to
cross langauge dependencies?

[1] http://github.com/talios/clojure-maven-example/blob/master/pom.xml#L36

[Chas Emerick]

Are the dependencies circular? i.e. do you have clojure code that
depends on java code that depends on clojure code in the same
project? If so, you need to have a javac / clojure:compile invocation
for each "stage" of the dependency graph, if you will. Anything
beyond that would require hooking the clojure compiler into javac so
that clojure code could be compiled as necessary to satisfy the import
requirements of each java compilation unit. That's certainly
possible, but not easy to implement -- AFAIK, groovy is the only JVM
language that provides that sort of circular compiler dependency
resolution.

In terms of real-world solutions, I'd suggest that this sort of
compilation circularity indicates the need for a refactoring or two to
cut that knot. Also, you can use maven modules in order to gain more
control (IMO, in a more comprehensible way to future maintainers
compared to complicated hand-configured interleaved compilation
executions).

If the dependencies aren't circular, and you just need to compile the
clojure code before all of your java code, you can certainly bind
clojure:compile to a different phase that comes before 'compile' --
'process-resources', perhaps. That's a *slight* wart in your pom, but
an easy solution. Again, you can use maven modules to have more
explicit control over the dependencies between different codebases
within the same project.

Cheers,

- Chas

[cburroughs]

Thanks. For the record my dependencies were not circular and the
process-resources wart seemed to do the trick.

[Stuart Sierra]

Maven modules are the "correct" solution, but more complicated.
-S

[Laurent PETIT]

Hi,

2010/6/3 Stuart Sierra <the.stua...@gmail.com>

Maven modules are the "correct" solution, but more complicated.
-S

I would rather say "Maven modules are a working solution". Why "the correct solution" ? Inside a module, java allows circular dependencies between classes, and since the circular dependency is "local" to the module, nobody cares (furthermore, it's justified as a "cohesive design" :-) ).

So really, I think the ideal "correct" solution (not existing yet) would be to plug the clojure compiler in the java compiler, creating in-memory appropriate stubs for types, records, genclasses and geninterfaces in the compiler model.


Anyway, the "Maven modules" solution forces you to go back to an acyclic dependency graph between your modules. For this you would have been forced to separate the code for the interfaces from the code for the implementation. Either by having e.g. the gen-class have a different implementation namespace, and then compiling all the "interface" parts of the gen-classes before java compilation, and then compile all the rest of the clojure code (raw clojure code, namespaces implementing genclasses). Either by creating regular java interfaces / classes.

Tough topic :)

[Mark Derricutt]

I've often done the process-resources trick to get around this, but it does feel dirty.  Personally I prefer splitting things to separate modules, it can feel a bit "overkill" having another module, another pom.xml, and another full directory structure, but I find the benefits of doing that separation works out for the best in the end.

At times I wish maven had pre/post meta-phases, but I could see that bringing all maner of more confusion and problems.

--
Pull me down under...

[Laurent PETIT]

If this is an option for you, one modification to your code could be to create the contract between clojure and java in terms of java interfaces, written in java.
And, to avoid using the constructors of your clojure implementations in your java code, you could (hopefully) use dependency injectors such as spring xml configuration files.

2010/5/28 Chas Emerick <ceme...@snowtide.com>

[Jason Smith]

I think, and this may not work right off, but I think that if you put in a Java stub class manually, your Java compile will be happy (it only needs the class, methods, javadoc, etc., to compile), and then Clojure compile + gen-class can replace the stubbed version.  The stub just defines the class and javadoc, and methods.  Each method throws an exception when called, so you can be sure Clojure has replaced it with it's gen-class.

 

If that doesn't work immediately, you can try putting in some script to delete the stubbed class between Javac and Clojure compile.  If you need to do that, put another message out on the forum, and I'll go into more detail.  Let me know if you need more info on how to manually create the stubs.

 

See also: http://groups.google.com/group/clojure/browse_thread/thread/ea39801f8e907640/d9e998a999d2833d?lnk=gst&q=stubs#d9e998a999d2833d

[Chas Emerick]

On Jun 3, 2010, at 9:42 AM, Laurent PETIT wrote:

> So really, I think the ideal "correct" solution (not existing yet)
> would be to plug the clojure compiler in the java compiler, creating
> in-memory appropriate stubs for types, records, genclasses and
> geninterfaces in the compiler model.

Of course, this is decidedly nontrivial -- but the integration would
actually go in the other direction. I looked at what this would
entail a year or so ago, and quickly came to the conclusion that the
way the groovy folk do it (IIRC, having groovyc be the "lead" compiler
and call out to javac when necessary, providing classfile stubs of
what will eventually be compiled from groovyland in order to satisfy
potential Java code dependencies) is the only "sane" approach.

I'm not sure that this is a common enough situation to warrant the
individual or community effort that would be required to make the
above a reality in Clojure (especially given a variety of project
configuration and refactoring workarounds). That might change, but I
cringe at the thought of even scoping out the work required.

Cheers,

- Chas

[Laurent PETIT]

Hi Chas,

2010/6/6 Chas Emerick <ceme...@snowtide.com>:

>
> On Jun 3, 2010, at 9:42 AM, Laurent PETIT wrote:
>
>> So really, I think the ideal "correct" solution (not existing yet) would
>> be to plug the clojure compiler in the java compiler, creating in-memory
>> appropriate stubs for types, records, genclasses and geninterfaces in the
>> compiler model.
>
> Of course, this is decidedly nontrivial -- but the integration would
> actually go in the other direction.  I looked at what this would entail a
> year or so ago, and quickly came to the conclusion that the way the groovy
> folk do it (IIRC, having groovyc be the "lead" compiler and call out to
> javac when necessary, providing classfile stubs of what will eventually be
> compiled from groovyland in order to satisfy potential Java code
> dependencies) is the only "sane" approach.

It could work in most cases, but I fear not in the general case.
Because Clojure has macros. So
* a pure static analysis of the code for deriving all possible stub
is not possible in the general case without running the macros
* and running the macros requires having everything evaled before
the macros being "in good shape". So if some of your clojure macros
have a dependency on a java class which in the call stack will run
directly or indirectly clojure code, chess mat ! :'(

Groovy AFAIK does not have this problem, so the groovy stubs -> javac
-> groovyc sequence should work in all cases.

Now, I know almost nothing about the possibilities to plug behaviour
in javac. It may well be that there's no possibility at all ? :-(

> I'm not sure that this is a common enough situation to warrant the
> individual or community effort that would be required to make the above a
> reality in Clojure (especially given a variety of project configuration and
> refactoring workarounds).  That might change, but I cringe at the thought of
> even scoping out the work required.

Yes, it's daunting. Living with that for the moment, and lobbying
collaboratively with other "JVM dynamic languages" to have javac
enhanced with pluggable compiler extensions could be a way to follow ?

[Chas Emerick]

On Jun 6, 2010, at 3:48 AM, Laurent PETIT wrote:

> It could work in most cases, but I fear not in the general case.

...

Yes, good point.

>> I'm not sure that this is a common enough situation to warrant the
>> individual or community effort that would be required to make the
>> above a
>> reality in Clojure (especially given a variety of project
>> configuration and
>> refactoring workarounds). That might change, but I cringe at the
>> thought of
>> even scoping out the work required.
>
> Yes, it's daunting. Living with that for the moment, and lobbying
> collaboratively with other "JVM dynamic languages" to have javac
> enhanced with pluggable compiler extensions could be a way to follow ?

Well, javac is already pluggable -- javadoc is a javac "extension",
for example. But the APIs involved are all extraordinarily
complicated and fundamentally undocumented AFAIK.

- Chas

[Laurent PETIT]

2010/6/6 Chas Emerick <ceme...@snowtide.com>:

Does that mean that it's not even standardized ? (That is it would
work for Oracle/Sun's javac, but not IBM's , etc. ?)

[Jason Smith]

If anyone is willing to write a generic stub generator, I'll volunteer to help with the Maven side.  I'm a contributor on the GMaven project, and I understand the Maven portions.  I'm not good enough with Clojure yet to write the stub generator myself, but it doesn't look like it would be that hard for someone who knows the language well.  Should be much easier than in Groovy, where you are dependent on ANTLR, and it has to be integrated in core Groovy to be viable.

[Jason Smith]

I've worked with both the newer annotation APIs that plug into Javac and with extending JavaDoc via their APIs.  They don't solve this problem, or even come close.  Or in other words, "There is no possibility at all."

 

Working on GMaven for a while, I've had ample time to think about this problem, and I have to agree that the only sane way - at present - is the Groovy way.  This is a general problem for all languages running on the JVM, so Sun may eventually come up with a general solution.  Until then, the cleanest solution I've seen is cross-compilation using stubs, and treating Java as the least-common-denominator language.

 

***I see what you are saying about the macros.***  Can you give a more concrete example of the macro problem?  I am thinking about it, and all I can come up with is edge cases (where the macro needs the Java class to evaluate).  Since a macro is code generating code, I am not sure that case will come up often.  How often will I write a Java class to assist me in a macro (probably never, for me)?  Why not just use Clojure?  Theoretically, mathematically it is a problem.  Pragmatically speaking, it should be rare.

 

Clojure is very different from Java, so to get the level of interoperability that stubbing would add - that's huge.

[Laurent PETIT]

Hello,

2010/6/7 Jason Smith <ja...@lilypepper.com>:

> I've worked with both the newer annotation APIs that plug into Javac and
> with extending JavaDoc via their APIs.  They don't solve this problem, or
> even come close.  Or in other words, "There is no possibility at all."
>
> Working on GMaven for a while, I've had ample time to think about this
> problem, and I have to agree that the only sane way - at present - is the
> Groovy way.  This is a general problem for all languages running on the JVM,
> so Sun may eventually come up with a general solution.  Until then, the
> cleanest solution I've seen is cross-compilation using stubs, and treating
> Java as the least-common-denominator language.
>
> ***I see what you are saying about the macros.***  Can you give a more
> concrete example of the macro problem?  I am thinking about it, and all I
> can come up with is edge cases (where the macro needs the Java class to
> evaluate).  Since a macro is code generating code, I am not sure that case
> will come up often.  How often will I write a Java class to assist me in a
> macro (probably never, for me)?  Why not just use Clojure?  Theoretically,
> mathematically it is a problem.  Pragmatically speaking, it should be rare.

Well, not sure either it's just a theoretical problem or not. And for
sure having stubs which would clearly solve 90% of the problem would
be great.
My fear was coming from the point that maybe, one of the real selling
points of clojure - macros, e.g. "throw your androMDA cartridges away"
:-) -, when used for real, would make this not work anymore.

I can see 2 points where macros could make this more difficult:

a) macros assisting in removing boiler plate, and, in the process,
ending up in generating bigs (do ...) with genclass, types, protocols,
interfaces definitions. This sole point would imply creating stubs not
only by doing static clojure code analysis, but also dynamic code
evaluation. If there is a cycle at one point between java and clojure
code, this will become problematic to manage.
b) macros themselves using java classes from the project. Maybe, as
you said, not as problematic as point a), but if the macros don't use
java classes but just require them to be on the classpath if e.g. the
macros use gen-class to derive a java class or interface of the
project, then we're back into trouble.

Here are some ideas of scenario which involve points a), b) or both:

* imagine one wants to leverage the ability to easily create
immutable datastructures, usable from java. One could then decide to
create the problem domain concepts with java interfaces, and then let
clojure macroification generate the types for the interfaces. This
implies a cycle, macros, and would be problematic.
* another example : one uses the superb web framework "YAWF" (Yet
Another Web Framework) and has written a splendid abstract class for
his project, namely AbstractMyProjectYAWFController. Now he wants to
leverage clojure and wants it to remove some boilerplate and wants to
write a macro for generating AbstractMyProjectYAWFController
subclasses from declarative clojure datastructures. And of course he
wants to be able to instanciate those subclasses from his java code
(or hopefully from his AOC container). Here again, point b) will
bother us. The defined macro, when called, will look like
(defcontroller MyAwesomeUserLoginController [:splendidActionHook
[:login :password*] ....). And now try to understand from this code
that you must generate a stub fro MyAwesomeUserLoginController,
deriving from AbstractMyProjectYAWFController, with an additional
public YAWFControllerResultAbstraction splendidActionHook(String
login, String[] password) :-( :-(

Does this make sense ?

--
Laurent

[Meikel Brandmeyer]

Hi,

Am 07.06.2010 um 23:48 schrieb Laurent PETIT:

> a) macros assisting in removing boiler plate, and, in the process,
> ending up in generating bigs (do ...) with genclass, types, protocols,
> interfaces definitions. This sole point would imply creating stubs not
> only by doing static clojure code analysis, but also dynamic code
> evaluation. If there is a cycle at one point between java and clojure
> code, this will become problematic to manage.

A report from the trenches: VimClojure - using Nailgun - had
to generate a classes for the different commands, which were
then invoked by the nailgun server. This was abstracted away
by a macro, basically wrapping everything in call which basically
looked like a defn in the end.

This structure was now rewritten to work differently (a central
nail written in Java dispatching to normal clojure functions),
but it worked like charm and I wouldn't dismiss this approach
as too exotic.

Sincerely
Meikel

[Jason Smith]

Okay, skipping to the concrete examples at the bottom...

1) I am not sure if you mean that Clojure is generating the interfaces, or using them.  If it is using them, Java already knows about them.  If Clojure is generating the interfaces and the concrete implementation, then you'd have to generate stub interfaces before compiling the Java.  I'm not sure why you consider this an unsolvable cycle, so perhaps I don't understand the point.

2) I don't understand the point.  :-)  Is this meant to show a cycle, or is it meant to show the difficulty of extracting class information?  Perhaps the approach is wrong.  After all, gen-class can create class files. Perhaps if we redefined gen-class temporarily, we could extract the stub information rather than generating the class files.  Run it twice, each time with a different implementation.  It seems to me that it should be easier to create a stub than it is to generate class files, or at least no more difficult.

And it's likely that I am missing the point entirely.  Speak slowly and use small words.