Natively Compiled Clojure

[octopusgrabbus]

I use Clojure primarily as a very reliable tool to aid in data transformations, that is taking data in one application's database and transforming it into the format needed for another applications' database.

So, my question is would a natively compiled Clojure make sense or turn the language into something that was not intended? In almost all instances I have not found a problem with Clojure's execution speed so my question is not about pro or anti Java.

Thanks.

[Timothy Baldridge]

If you haven't found a performance problem, and it's working well for your needs, why are you interested in making the code run "natively". What problems have you encountered that pique your intrest in this area?

Timothy
 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

--
“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[octopusgrabbus]

I've had no problems. I am just curious.

What triggered this was this morning I saw something on stackoverflow.com about Clojure's possibly interacting with C code, and the natively compiled question just popped into my head.

[Shantanu Kumar]

> What triggered this was this morning I saw something on stackoverflow.comabout Clojure's possibly interacting with C code, and the natively compiled

> question just popped into my head.

Could Clojure-JNA help there, or is it some other use-case?
https://github.com/Chouser/clojure-jna

Shantanu

[octopusgrabbus]

Thanks. I took your link and posted it as a comment on stackoverflow.

[Philip Potter]

you might want to look at clj-native:

https://github.com/bagucode/clj-native

It's used, among other things, by overtone to interact with
libscsynth, the supercollider synthesis library:

https://github.com/overtone/overtone/blob/master/src/overtone/sc/machinery/server/native.clj

I don't know how it compares to clojure-jna; clj-native also uses jna
under the hood.

Phil

[Chouser]

I'm pretty sure clj-native is more recent, faster, better, and more actively maintained. I ought to update clojure-jna to say all that.

--Chouser

[Bronsa]

It is also not longer actively mantained.
https://github.com/bagucode/clj-native/issues/6#issuecomment-11930841

2013/1/21 Chouser <cho...@n01se.net>

[Philip Potter]

That may be the case, but it still has an active community of users.
At worst, you'll still be able to get answers to questions by asking
here; at best, someone might be able to step up and fill the role.

I can definitely help with questions.

Phil

[Mikera]

A natively compiled Clojure would be very very interesting (perhaps targeting LLVM?)

However it would also be very hard to implement. Clojure depends on a lot of features provided by the JVM (JIT compilation, interop with Java libraries, garbage collection being the most significant ones). It would be very hard to reimplement all of these from the ground up. The JVM is already a very good host platform, why fix something that isn't broken?

Arguably the effort would be better spend improving the JVM with extra features that would help Clojure (e.g. TCO). 

[Marko Kocić]

On Friday, January 25, 2013 6:12:07 AM UTC+1, Mikera wrote:

A natively compiled Clojure would be very very interesting (perhaps targeting LLVM?)

However it would also be very hard to implement. Clojure depends on a lot of features provided by the JVM (JIT compilation, interop with Java libraries, garbage collection being the most significant ones). It would be very hard to reimplement all of these from the ground up. The JVM is already a very good host platform, why fix something that isn't broken?

What about native ClojuresScript? It doesn't have to implement everything Clojure have already, and many people could consider it good enough alternative to Clojure. I could personally live without runtime macros and eval if it would gain me small and performant native executable. 

[Nahuel Greco]

Check the clojure-py2 project, they plan to use LLVM to generate native modules (as C compiled) for Python. When that objective is reached probably you will have almost all the machinery to compile python-less native binaries: 

http://lanyrd.com/2013/clojurewest/sccgmm/

Saludos,
Nahuel Greco.

--

[Tony Pitluga]

There are a couple projects that might be worth looking at although it seems both have not been updated in a few months.

ClojureC: https://github.com/schani/clojurec 

Clojure-Scheme: https://github.com/takeoutweight/clojure-scheme

[Timothy Baldridge]

The important question to ask yourself (and I'll cover this in my talk), is why do you want native Clojure?

A native implementation of Clojure will fail to deliver on several fronts:

Interop with systems - Java has one of the biggest ecosystems on the planet

Performance - The JVM JIT and GC are really the best that the world has to offer. Beating them in the general use case is very, very hard. 

However, a native clojure implementation could succeed on a few other fronts:

Deterministic behavior - The JVM JIT/GC are not very deterministic, and are hard to reason about. In this area they trade off simplicity for performance. 99% of the time that is what you want. But perhaps in some cases you would like different behavior. 

Embedding in applications - Python is used very often as a scripting language in 3d apps, games, mapping software, etc. I've yet to hear of the JVM ever being used for this. 

Systems where the JVM isn't available/allowed - think iOS

So while I think there is a use for a non JVM/JS implementation of Clojure, the use cases are much smaller than many think.

I've gotten a bit sidetracked on the more general topic of generating native code via Clojure and LLVM lately, so while my talk will show some basic Clojure code running in the Python VM via LLVM, it will also cover the more broad concept of leveraging LLVM directly from Clojure on the JVM.

The latter is a prerequisite to the former, so the library I'll be showcasing can be used for generating code for either situation. Should be fun times for all!

Timothy

[John Gabriele]

On Friday, January 25, 2013 11:28:32 AM UTC-5, tbc++ wrote:

The important question to ask yourself (and I'll cover this in my talk), is why do you want native Clojure?

  * easy direct access to C libs
  * fast startup time
  * minimal footprint

Well, that's what would attract *me* to a native Clojure, anyway.

---John

[Konrad Hinsen]

Timothy Baldridge writes:

> The important question to ask yourself (and I'll cover this in my talk), is why do
> you want native Clojure?

...

> Interop with systems - Java has one of the biggest ecosystems on the planet

The Java ecosystem is big but concentrated on some application domains.
Other application domains (including mine, scientific computing) rely on
a "native code" ecosystem (C and Fortran libraries). The JVM is very bad
at interop with native code when large data is involved, because the
JNI puts security before efficiency.

The CLR does a better job there, so those looking for native interop
could explore ClojureCLR. The main problem there is the
Windows-centricity of the CLR ecosystem. Even if the CLR is in theory
portable, with Mono providing an implementation for Linux and MacOS,
many important tools and libraries for the CLR are available only for
Windows, or are a pain to use elsewhere.

Konrad.

[Mark Rathwell]

> Embedding in applications - Python is used very often as a scripting language in 3d apps, games, mapping software, etc. I've yet to hear of the JVM ever being used for this.

Related to this, do you have any thoughts on the viability of
embedding clojure-py into a C++ application for similar use cases?
Seems that performance could be a concern, but sandboxing might be
more straightforward.

Haven't had time to try this myself yet.

Thanks.

[Paul deGrandis]

As far as embedding Clojure is concerned, another option is ClojureScript-Lua + LuaJit + C.
I've recently started going through the CLJS-Lua source to see how viable this is.

I am holding off committing any serious effort until I see the talk at Clojure/West.  C + Python + ClojurePy2 could fit most of my needs (even if I have to give up some speed and memory).

Paul

[Mark Rathwell]

On Sun, Jan 27, 2013 at 1:31 PM, Paul deGrandis
<paul.de...@gmail.com> wrote:
> As far as embedding Clojure is concerned, another option is
> ClojureScript-Lua + LuaJit + C.
> I've recently started going through the CLJS-Lua source to see how viable
> this is.

Thanks, I had forgotten about ClojureScript-Lua. (And hopefully the
content of the clojure-py2 talk will be available afterwards, very
interesting work.)

[Paul deGrandis]

Additionally, you might also consider ClojureScript itself.  With a little work you could embed V8 into your application (https://developers.google.com/v8/embed).

Martin Trojer just wrote a really nice blog post on embedded runtimes - http://martinsprogrammingblog.blogspot.co.uk/

All that said, for most apps where an embedded, scripting runtime is advantageous, ClojureScript-Lua +LuaJIT and PyClojure+cFFI+Python/PyPy are two very attractive options.  In the best case scenario, the former delivers more speed and a smaller footprint, trading off some library and ecosystem comforts.  The latter has a great ecosystem, but will be slower and have a larger memory footprint.

Paul

[Timothy Baldridge]

That's why I think it's a good idea to ask what the goals are for native Clojure. The ClojureScript and Clojure-Py options while nice both don't allow for a good concurrency story. On top of that, I'm not sure either of those would actually run on iOS. 

However, a pure, from-scratch option has a much smaller ecosystem to draw from. But on the upside, you're also not limited by the host VM. This means that the type system could be based purely on protocols instead of having to fit protocols into a OOP type system. 

These are the questions I'd like to see people answer here: what hard limits have you hit with Clojure/ClojureScript that you think could be resolved with a native Clojure implementation?

Timothy Baldridge

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
 
 

[Mark Rathwell]

I haven't hit any hard limits at this point, but you hit on a use case
where Python and Lua currently hit a sweet spot that I think would be
nice to use Clojure:

C/C++ systems that want to expose scripting capabilities to users
(e.g. game engines, robotics systems).

For these types of use cases, the ability to deploy native Clojure
executables is not as important as the ability embed a low-resource
runtime, the ability to call C/C++ code, the ability to sandbox the
environment, etc.

[AtKaaZ]

although I'm sure everybody's seen this, I believe it is relevant here, this clojureconj by Chris Granger
https://www.youtube.com/watch?v=V1Eu9vZaDYw
maybe only applies to clojurescript(that is, being slow in this case)
the important stuff is at from 13:59

I may be wrong or incomplete.
Please express any corrections / additions,
they are encouraged and appreciated.
At least one entity is bound to be transformed if you do ;)

[Mikera]

On Monday, 28 January 2013 07:56:58 UTC+8, tbc++ wrote:

That's why I think it's a good idea to ask what the goals are for native Clojure. The ClojureScript and Clojure-Py options while nice both don't allow for a good concurrency story. On top of that, I'm not sure either of those would actually run on iOS. 

However, a pure, from-scratch option has a much smaller ecosystem to draw from. But on the upside, you're also not limited by the host VM. This means that the type system could be based purely on protocols instead of having to fit protocols into a OOP type system. 

These are the questions I'd like to see people answer here: what hard limits have you hit with Clojure/ClojureScript that you think could be resolved with a native Clojure implementation?

The hard limits I have hit in Clojure are around three specific areas:

1. GC latency on the JVM (usually not a problem, but can be an issue for games and realtime work)

2. Startup latency

3. Access to hardware level features (GPU access, OpenGL etc.)

1. Can be avoided by avoiding memory allocation (there are Java libraries that do this like Javaolution), but this is really hard to do in idiomatic Clojure. Even something as simple traversing a single vector as a sequence is O(n) in new object allocations.

2. Is actually more about the time taken to load / compile the Clojure core libraries than it is about the JVM. On my machine, the JVM itself starts up in about 0.1 sec which is fine for anything other than chaining thousands of script invocations. Generally I think that people who complain about JVM startup time probably have the wrong architecture - you don't spin up a new virtual server to handle every web request, do you?

3. Can be done, but it's a pain to configure

Personally, I think the advantages of being on the JVM vastly outweigh the disadvantages:

- Extremely fast and mature JIT compiler / GC algorithms

- Huge library ecosystem 

- Easy portability across platforms

Hence what I'd really like to see are better ways to integrate and / or generate native code within JVM Clojure. In particular:

- Better ways to load / execute native code libraries at runtime

- Macros / tools for accessing natively managed memory while still keeping a Clojure style

- A really good JNA wrapper (clj-native seems along the right track.....)

- A native code compiler that can be executed at runtime. So we can actually compile to native code and run it dynamically. Possibly something based on LLVM would work well here.

If we had all that, we would genuinely have the best of both worlds.

 

[Konrad Hinsen]

Timothy Baldridge writes:

> That's why I think it's a good idea to ask what the goals are for native Clojure. The
> ClojureScript and Clojure-Py options while nice both don't allow for a good concurrency
> story. On top of that, I'm not sure either of those would actually run on iOS. 
>
> However, a pure, from-scratch option has a much smaller ecosystem to draw from. But on
> the upside, you're also not limited by the host VM. This means that the type system
> could be based purely on protocols instead of having to fit protocols into a OOP type
> system. 

Another approach that might be worth exploring (for someone who has
lots more time than me for such a project) is hijacking the
implementation of the Julia language (http://julialang.org/). It's a
language with many Lisp inspirations, despite its non-Lisp syntax, and
it's based on LLVM with no virtual machine in between, but a run-time
system with a garbage collector. It's data model is much more similar
to Clojure's than to classic OOP, with all functions essentially being
multimethods. A native Clojure implementation could probably re-use
much of Julia's implementation.

Konrad.

[Mikera]

On Tuesday, 22 January 2013 00:29:54 UTC+8, octopusgrabbus wrote:

On this general topic: the following project (VMKit) looks very interesting:

 - http://vmkit.llvm.org/index.html

It basically implements a full JVM on top of LLVM, which should in theory be sufficient to run Clojure on the LLVM.

This opens some interesting opportunities for combining Clojure with the LLVM infrastructure for native code generation.

[Paul deGrandis]

I personally have no limitations with Clojure and the JVM for general development.

Like Mark mentioned, I have a need to embed runtimes within C systems, with a small memory footprint (ideally less than 5 Mb) fast execution, and a solid FFI.
Giving up some raw performance to get some benefits of Clojure is a worthy trade-off for me (macros, persistent data structures, etc).  Concurrency would be awesome if it existed, but it's not a requirement - I can always fall back on lower-level/error-prone/hand-controlled concurrency mechanisms within the C system itself.

In other systems I've been working on where the core library is C and I have a lot of memory available, JNA/JNI/BridJ has been fine as long as I've followed best practices regarding those technologies.

When I needed a smaller memory footprint, and could give up some speed, ClojureScript + Node.js has been awesome for throwing quick things together.  I'm very much inclined to switch this to clojure-py2 in the future if language-feature support is on-par with CLJS or there was a speed/memory benefit.

Unlike Mike, the small amount of OpenCL stuff I've done has been fine - I'll be doing more in the future, so that may change :)
It took some time to adjust to best practices to make the GPU worth it.  Hopefully Project Sumatra will make this a moot point: http://openjdk.java.net/projects/sumatra/

Paul

[Konrad Hinsen]

Mikera writes:

> On this general topic: the following project (VMKit) looks very interesting:
>
> - http://vmkit.llvm.org/index.html
>
> It basically implements a full JVM on top of LLVM, which should in theory be sufficient
> to run Clojure on the LLVM.

I tried this a while ago and had some exchanges with VMKit authors. Back then,
VMKit crashed on Clojure initialization because of a memory limit in VMKit
concerning the size of a single method. Clojure initializes tons of data in a
single JVM method.

But I agree that VMKit is an interesting project in principle. Someone will have
to take up active development again to make it more usable.

Konrad.

[Aaron]

I have been working on a library for ClojureCLR that actually uses a Clojure DSL to generate, compile, and load C code live into a running REPL.  It's called c-in-clj (see https://github.com/aaronc/c-in-clj).  It works quite well for me so far, but is still in what I would call Alpha stage.  Eventually, this code may be used to compile Clojure code to a native binary via a C compiler.

You can also use this little library that I have for ClojureCLR to do painless native function calls via the CLR's P/Invoke mechanism: https://github.com/aaronc/ClojureClrEx.