ClojureCLR updated

[dmiller]

I've posted a major update to the ClojureCLR code to clojure-contrib.
However, if you want to stay up with the latest changes, I recommend
gitting the code at http://github.com/dmiller/ClojureCLR. Consult the
wiki there for information on installing / compiling / running.

This release brings the code up to revision 1370 of the JVM version,
about a week ago. This means that all the lazy changes have been
incorporated. The latest version of the DLR is used.

The compiler has been completely rewritten. It now follows the JVM
version very closely, though it still uses the DLR (specifically,
Expression Trees V2) for most code generation. Type hints are
followed. Most important, AOT-compilation now works and loading and
compiling are pretty complete. Because the bootstrap files such as
core.clj can be AOT-compiled into CLR assemblies, the startup time has
been significantly reduced.

Of the bootstrap files, core.clj is 95% complete. core_print, set,
and zip are at 100%. And main.clj loads, so the standard REPL is
available. Proxying and genclass haven't been attempted yet.

Speed is still an issue. For basic code generation, the MSIL
produced by the ClojureCLR compiler is as close as possible to the
bytecodes produced by the Clojure(JVM) compiler. The next phase f
work is to do IL-level comparisons of the compiler output of each
implementation. If anyone has some *.clj benchmarks to recoommend,
let me know.

Overall, this release is a very large step forward in performance and
functionality. There are still enough rough edges and gaps in testing
that daily use cannot be recommended, but those wanting to play and
test will find this release much easier to deal with. Bon appetit.

David Miller

[Shawn Hoover]

Wow, great work! I followed the instructions and it worked as stated except for one missing detail: nothing happened because I didn't realize I had to pass arguments to the bootstrapcompiler. I recommend adding a note to compile-run doc that you need command line arguments like "clojure.core clojure.set clojure.zip clojure.main". Once I got that working, it spat out dlls and subsequent startup went from about 9 seconds on my laptop down to 2!

Shawn

[David Miller]

I'll add a note on the necessary command-line args to
BootstrapCompile. I didn't think of it because I'm always running
debug mode inside VS and the command line args are set up in the
project properties.

The speedup in startup you saw is consistent with my experience.

Thanks for the feedback.

David

[kinghajj]

On May 31, 1:08 am, dmiller <dmiller2...@gmail.com> wrote:
> Proxying and genclass haven't been attempted yet.

Can you guess how long they should take to implement?

> Speed is still an issue.   For basic code generation, the MSIL
> produced by the ClojureCLR compiler is as close as possible to the
> bytecodes produced by the Clojure(JVM) compiler.

Why is it important that the MSIL is similar to the JVM bytecode?
Shouldn't the compiler use techniques specific for performance on the
CLR?

> Overall, this release is a very large step forward in performance and
> functionality.  There are still enough rough edges and gaps in testing
> that daily use cannot be recommended, but those wanting to play and
> test will find this release much easier to deal with.  Bon appetit.

It'd be much easier to play with if you provide a precompiled
executable :)

Keep up the good work! I hope to someday use ClojureCLR for real
projects, so I can have all the functional, concurrent goodness of
Clojure in .NET.

[David Miller]

:>> Proxying and genclass haven't been attempted yet.

:> Can you guess how long they should take to implement?

Probably not too long. I cannot do a straight conversion of
core_proxy.clj and genclass.clj as I did with other bootstrap files --
the code is heavily dependent on the ASM bytcode library. I'll have
to recode all that in System.Reflection.Emit. That will take some
work. If I move it to the front of the queue, it could be done in a
week or two.

:> Why is it important that the MSIL is similar to the JVM bytecode?

:> Shouldn't the compiler use techniques specific for performance on
the
:> CLR?

It's important in that it means that the generated MSIL is not
completely junk, in that I'm not missing any important optimizations,
that I'm taking full advantage of type hints, avoiding reflection,
etc.. The JVM bytecodes are an important check for me. What's
interesting is that the methods used to generate the IL in the each
implementation are very different. The two implementations generate
AST from the clojure source that are pretty much identical. From the
ASTs, the JVM implementation generates bytecodes using the ASM
bytecode library--essentially, it's close to hand-generated. Rich and
company have that code pretty finely tuned. The CLR implementation
transforms the ASTs into DLR-Expression-Tree-Version-2 expressions.
The DLR code handles compiling those expressions into either dynamic
methods or into static methods for saving in assemblies. The DLR
expression compiler should be generating decent MSIL. I think the
closeness of the results are encouraging.

The JVM implementation does pull some tricks, such as nulling method
arguments before tail calls or storing some temp values on the stack,
that I can't figure out how to duplicate with ExpressionTrees. Also,
the expression tree compiler can only generate static methods.
Clojure functions are instances of a class implementing the IFn
interface. I have to hand-code the class definitions and code the
'invoke' instance methods to call out to static methods in a base
class. That adds an extra method call in many places. Whether all
these little things add up to some of the performance differences is
beyond my knowledge or the granularity of the profiling tools at my
disposal. Unless some MSIL expert surfaces to give advice, I'll be
spending time hand-coding some alternatives to benchmark certain
constructs to see what works better. Worst case, I will end up
discarding the MSIL and doing the MSIL generation myself.

:> It'd be much easier to play with if you provide a precompiled
:> executable :)

I thought about that. Adding assembilies of my code as a download is
easy enough. However, to get the thing running, you also need vjslib
from the J# Redistributatable library plus DLLs generated from the DLR
source -- care to advise me about the legal ramifications of me doing
that directly? :)

:> Keep up the good work! I hope to someday use ClojureCLR for real
:> projects, so I can have all the functional, concurrent goodness of
:> Clojure in .NET.

I think that day is not too far off.

[Konrad Hinsen]

On 01.06.2009, at 19:57, David Miller wrote:

> :> It'd be much easier to play with if you provide a precompiled
> :> executable :)
>
> I thought about that. Adding assembilies of my code as a download is
> easy enough. However, to get the thing running, you also need vjslib
> from the J# Redistributatable library plus DLLs generated from the DLR
> source -- care to advise me about the legal ramifications of me doing
> that directly? :)

I remember reading that Mono now supports DLR. Would it be
envisageable to supply a fully Mono-based precompiled executable?
That would also be nice for us non-Windows users...

BWT, ClojureCLR is a very interesting development for me, as
colleagues of mine have a lot of .NET code (running under Linux with
Mono).

Konrad.

[Shawn Hoover]

If Mono supports the DLR, the only thing I see holding back that dream is the Visual J# dependency. Even that could go away if ClojureCLR had a BigInteger implementation, or something similar that enables Clojure's arbitrarily large integers.

[David Miller]

Getting rid of the vjslib dependency is on the todo list. I started
looking at it yesterday. I was going to switch to the
Microsoft.Scripting.Math.BigInteger class that comes with the DLR and
then implement a BigDecimal on top of that. However, there are some
missing pieces for our needs, and their implementation does not expose
enough internals to allow me to do certain algorithms efficiently. so
now I'm contemplating grabbing a copy of Knuth, crossing my fingers,
and digging in.

I've watched Mono for a long time, but haven't dived in. I guess now
is the time.

David

On Jun 1, 3:56 pm, Shawn Hoover <shawn.hoo...@gmail.com> wrote:

[Johan Berntsson]

On Jun 2, 6:51 am, David Miller <dmiller2...@gmail.com> wrote:
> I've watched Mono for a long time, but haven't dived in.  I guess now
> is the time.

Would be great. I'm very excited about ClojureCLR, but won't be able
to use it until it runs on Linux and isn't tightly bound to a specific
IDE.

/Johan

[Jeff Heon]

On Jun 1, 1:57 pm, David Miller <dmiller2...@gmail.com> wrote:
> It's important in that it means that the generated MSIL is not
> completely junk, in that I'm not missing any important optimizations,
> that I'm taking full advantage of type hints, avoiding reflection,
> etc..  The JVM bytecodes are an important check for me. What's
> interesting is that the methods used to generate the IL in the each
> implementation are very different.  The two implementations generate
> AST from the clojure source that are pretty much identical.  From the
> ASTs, the JVM implementation generates bytecodes using the ASM
> bytecode library--essentially, it's close to hand-generated.  Rich and
> company have that code pretty finely tuned.  The CLR implementation
> transforms the ASTs into DLR-Expression-Tree-Version-2 expressions.
> The DLR code handles compiling those expressions into either dynamic
> methods or into static methods for saving in assemblies.  The DLR
> expression compiler should be generating decent MSIL.  I think the
> closeness of the results are encouraging.

I wonder if that project could be of some help/inspiration.
It implements a JVM in .NET
http://www.ikvm.net/