Clojure/LLVM

[Antony Blakey]

Has anyone looked at driving LLVM from Clojure and then doing code generation to LLVM? I've had some experience doing it in a JIT context for VisualWorks Smalltalk, but I'd rather not reinvent the wheel. I'd love to be able to use Clojure on the iPad.

Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

Did you hear about the Buddhist who refused Novocain during a root canal?
His goal: transcend dental medication.

[Per Vognsen]

Clojure depends on aggressive dynamic optimization for performance.
It's a very different set of techniques than what is used in the
ahead-of-time static compilation world from which LLVM hails. The
Google guys working on Unladen Swallow seem to have had enough
problems getting it to work well even as a basic JIT code generator;
the layers of abstraction make it very slow. There's a good reason V8
is using unlayered, direct code generation.

In my judgement it would be a multi-year project for an experienced
specialist in the field. Mike Pall took several years to do just the
x86 version of LuaJIT 2.0, but he had already done LuaJIT 1.0, and he
is arguably one of the world's top experts in this area.

Besides, one of the Clojure's main raisons d'etre is the vast pool of
existing libraries in the JVM (and now CLR) world to draw from.

-Per

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>
> To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
>

[mac]

Like Per says, if you want to compile Clojure to LLVM code that's
probably a huge project, have you looked at VMKit? I think that's a
JVM/CLR implementation based on LLVM, perhaps it could run Clojure?

If you just want to use LLVM from Clojure however, you can use my C
FFI clj-native with the C interface for LLVM. I actually made it
because I wanted to play around with LLVM but I never got around to
actually doing that again once I had a working version of clj-native.
It would be very interesting to see if clj-native is suitable for the
task.

http://github.com/bagucode/clj-native

/Markus

[Konrad Hinsen]

On 24 Mar 2010, at 07:40, mac wrote:

> If you just want to use LLVM from Clojure however, you can use my C
> FFI clj-native with the C interface for LLVM. I actually made it
> because I wanted to play around with LLVM but I never got around to
> actually doing that again once I had a working version of clj-native.
> It would be very interesting to see if clj-native is suitable for the
> task.

Just wondering: what are the performance implications of interfacing
to C via clj-native? Are all data items necessarily copied when
crossing the interface? I think that's what happens with JNI though I
am not sure. For big data items, that would be a disaster.

Konrad.

[mac]

Clarification of lettersoup:
JNI = java native interface, the C/C++ interface to java.
JNA = java native access, an API that aims to simplify java/C interop.

clj-native is built upon JNA and is at present a rather thin wrapper
so it's good to learn a bit about JNA to get the most out of clj-
native.

I have been careful to make clj-native as performant as possible, but
I haven't actually tested it much so I can't promise anything.
clj-native uses the "direct mapping" feature of JNA which is more
performant with regards to the function calls themselves than the more
dynamic JNA alternatives. This can make calls into native code quite
fast but calls from native code into Clojure via callbacks are slower
because JNI has to use a kind of reflection API to call methods :(

With regards to copying of data, clj-native expects nio buffers to be
used as input where a native function requires a typed pointer. These
buffers can be allocated outside of the JVM heap specifically to avoid
copying. For example (ByteBuffer/allocateDirect 1000) will allocate
1000 bytes outside of the JVM heap, which means it does not have to be
copied during native interop because the GC will never move it.
JNA also has facilities to create ByteBuffers from pointers which
means you can gain access to large blobs of memory that reside in
native code.

[Konrad Hinsen]

On 24.03.2010, at 11:22, mac wrote:

> With regards to copying of data, clj-native expects nio buffers to be
> used as input where a native function requires a typed pointer. These
> buffers can be allocated outside of the JVM heap specifically to avoid
> copying. For example (ByteBuffer/allocateDirect 1000) will allocate
> 1000 bytes outside of the JVM heap, which means it does not have to be
> copied during native interop because the GC will never move it.

OK, that clarifies how it works, thanks!

The essential question thus is how fast get and put operations to a direct buffer are, compared to an access to an array element. If the difference is not important, one could indeed use direct buffers for storing big data objects and access them efficiently from both sides of the interface.

Konrad.

[mac]

I think get/put are pretty fast but there is also the possibility of
slicing arrays in and out of nio buffers on the java side which maybe
could be faster if you want to modify many values and have some memory
to spare.

For example if you have a nio buffer several hundred megabytes big and
you want to change every value on the java/clojure side then you could
use a java array that's one or a few megs in size and continously fill
that upp from the buffer, change it's values and then write it back.
Don't know if it's actually faster but it sounds reasonable since it
will save you millions of method calls.
On the native side you should be able to get a direct pointer to the
memory in the buffer so you can just go crazy and do whatever you like
(hello segfault) ;)

[Jeff Heon]

I know it's not Clojure, but you can at least scratch the Lisp on the
iP* itch.

Scheme for the iPhone:
http://jlongster.com/software/iphone/scheme-iphone-example/
http://jlongster.com/blog/2010/02/23/farmageddon/