|Natively Compiled Clojure||octopusgrabbus||1/21/13 8:29 AM|
I use Clojure primarily as a very reliable tool to aid in data transformations, that is taking data in one application's database and transforming it into the format needed for another applications' database.
So, my question is would a natively compiled Clojure make sense or turn the language into something that was not intended? In almost all instances I have not found a problem with Clojure's execution speed so my question is not about pro or anti Java.
|Re: Natively Compiled Clojure||tbc++||1/21/13 9:19 AM|
If you haven't found a performance problem, and it's working well for your needs, why are you interested in making the code run "natively". What problems have you encountered that pique your intrest in this area?
“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
|Re: Natively Compiled Clojure||octopusgrabbus||1/21/13 9:25 AM|
I've had no problems. I am just curious.
What triggered this was this morning I saw something on stackoverflow.com about Clojure's possibly interacting with C code, and the natively compiled question just popped into my head.
|Re: Natively Compiled Clojure||Shantanu Kumar||1/21/13 10:15 AM|
> What triggered this was this morning I saw something on stackoverflow.comabout Clojure's possibly interacting with C code, and the natively compiled
> question just popped into my head.Could Clojure-JNA help there, or is it some other use-case?
|Re: Natively Compiled Clojure||octopusgrabbus||1/21/13 10:44 AM|
Thanks. I took your link and posted it as a comment on stackoverflow.
|Re: Natively Compiled Clojure||Philip Potter||1/21/13 11:21 AM|
you might want to look at clj-native:
It's used, among other things, by overtone to interact with
libscsynth, the supercollider synthesis library:
I don't know how it compares to clojure-jna; clj-native also uses jna
under the hood.
|Re: Natively Compiled Clojure||Chouser||1/21/13 11:25 AM|
I'm pretty sure clj-native is more recent, faster, better, and more actively maintained. I ought to update clojure-jna to say all that.
|Re: Natively Compiled Clojure||Nicola Mometto||1/21/13 11:27 AM|
|Re: Natively Compiled Clojure||Philip Potter||1/21/13 12:03 PM|
That may be the case, but it still has an active community of users.
At worst, you'll still be able to get answers to questions by asking
here; at best, someone might be able to step up and fill the role.
I can definitely help with questions.
|Re: Natively Compiled Clojure||Mikera||1/24/13 9:12 PM|
A natively compiled Clojure would be very very interesting (perhaps targeting LLVM?)
However it would also be very hard to implement. Clojure depends on a lot of features provided by the JVM (JIT compilation, interop with Java libraries, garbage collection being the most significant ones). It would be very hard to reimplement all of these from the ground up. The JVM is already a very good host platform, why fix something that isn't broken?
Arguably the effort would be better spend improving the JVM with extra features that would help Clojure (e.g. TCO).
|Re: Natively Compiled Clojure||Marko Kocić||1/25/13 7:18 AM|
What about native ClojuresScript? It doesn't have to implement everything Clojure have already, and many people could consider it good enough alternative to Clojure. I could personally live without runtime macros and eval if it would gain me small and performant native executable.
|Re: Natively Compiled Clojure||Nahuel Greco||1/25/13 7:35 AM|
Check the clojure-py2 project, they plan to use LLVM to generate native modules (as C compiled) for Python. When that objective is reached probably you will have almost all the machinery to compile python-less native binaries:
|Re: Natively Compiled Clojure||Tony Pitluga||1/25/13 8:08 AM|
|Re: Natively Compiled Clojure||tbc++||1/25/13 8:28 AM|
The important question to ask yourself (and I'll cover this in my talk), is why do you want native Clojure?
A native implementation of Clojure will fail to deliver on several fronts:
Interop with systems - Java has one of the biggest ecosystems on the planet
Performance - The JVM JIT and GC are really the best that the world has to offer. Beating them in the general use case is very, very hard.
However, a native clojure implementation could succeed on a few other fronts:
Deterministic behavior - The JVM JIT/GC are not very deterministic, and are hard to reason about. In this area they trade off simplicity for performance. 99% of the time that is what you want. But perhaps in some cases you would like different behavior.
Embedding in applications - Python is used very often as a scripting language in 3d apps, games, mapping software, etc. I've yet to hear of the JVM ever being used for this.
Systems where the JVM isn't available/allowed - think iOS
So while I think there is a use for a non JVM/JS implementation of Clojure, the use cases are much smaller than many think.
I've gotten a bit sidetracked on the more general topic of generating native code via Clojure and LLVM lately, so while my talk will show some basic Clojure code running in the Python VM via LLVM, it will also cover the more broad concept of leveraging LLVM directly from Clojure on the JVM.
The latter is a prerequisite to the former, so the library I'll be showcasing can be used for generating code for either situation. Should be fun times for all!
“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
|Re: Natively Compiled Clojure||John Gabriele||1/25/13 11:28 PM|
On Friday, January 25, 2013 11:28:32 AM UTC-5, tbc++ wrote:
* easy direct access to C libs
* fast startup time
* minimal footprint
Well, that's what would attract *me* to a native Clojure, anyway.
|Re: Natively Compiled Clojure||Konrad Hinsen||1/27/13 3:59 AM|
Timothy Baldridge writes:...
The Java ecosystem is big but concentrated on some application domains.
Other application domains (including mine, scientific computing) rely on
a "native code" ecosystem (C and Fortran libraries). The JVM is very bad
at interop with native code when large data is involved, because the
JNI puts security before efficiency.
The CLR does a better job there, so those looking for native interop
could explore ClojureCLR. The main problem there is the
Windows-centricity of the CLR ecosystem. Even if the CLR is in theory
portable, with Mono providing an implementation for Linux and MacOS,
many important tools and libraries for the CLR are available only for
Windows, or are a pain to use elsewhere.
|Re: Natively Compiled Clojure||Mark Rathwell||1/27/13 7:07 AM|
> Embedding in applications - Python is used very often as a scripting language in 3d apps, games, mapping software, etc. I've yet to hear of the JVM ever being used for this.Related to this, do you have any thoughts on the viability of
embedding clojure-py into a C++ application for similar use cases?
Seems that performance could be a concern, but sandboxing might be
Haven't had time to try this myself yet.
|Re: Natively Compiled Clojure||Paul deGrandis||1/27/13 10:31 AM|
As far as embedding Clojure is concerned, another option is ClojureScript-Lua + LuaJit + C.
I've recently started going through the CLJS-Lua source to see how viable this is.
I am holding off committing any serious effort until I see the talk at Clojure/West. C + Python + ClojurePy2 could fit most of my needs (even if I have to give up some speed and memory).
|Re: Natively Compiled Clojure||Mark Rathwell||1/27/13 12:47 PM|
On Sun, Jan 27, 2013 at 1:31 PM, Paul deGrandisThanks, I had forgotten about ClojureScript-Lua. (And hopefully the
content of the clojure-py2 talk will be available afterwards, very
|Re: Natively Compiled Clojure||Paul deGrandis||1/27/13 3:32 PM|
Additionally, you might also consider ClojureScript itself. With a little work you could embed V8 into your application (https://developers.google.com/v8/embed).
Martin Trojer just wrote a really nice blog post on embedded runtimes - http://martinsprogrammingblog.blogspot.co.uk/
All that said, for most apps where an embedded, scripting runtime is advantageous, ClojureScript-Lua +LuaJIT and PyClojure+cFFI+Python/PyPy are two very attractive options. In the best case scenario, the former delivers more speed and a smaller footprint, trading off some library and ecosystem comforts. The latter has a great ecosystem, but will be slower and have a larger memory footprint.
|Re: Natively Compiled Clojure||tbc++||1/27/13 3:56 PM|
That's why I think it's a good idea to ask what the goals are for native Clojure. The ClojureScript and Clojure-Py options while nice both don't allow for a good concurrency story. On top of that, I'm not sure either of those would actually run on iOS.
However, a pure, from-scratch option has a much smaller ecosystem to draw from. But on the upside, you're also not limited by the host VM. This means that the type system could be based purely on protocols instead of having to fit protocols into a OOP type system.
These are the questions I'd like to see people answer here: what hard limits have you hit with Clojure/ClojureScript that you think could be resolved with a native Clojure implementation?
|Re: Natively Compiled Clojure||Mark Rathwell||1/27/13 4:53 PM|
I haven't hit any hard limits at this point, but you hit on a use case
where Python and Lua currently hit a sweet spot that I think would be
nice to use Clojure:
C/C++ systems that want to expose scripting capabilities to users
(e.g. game engines, robotics systems).
For these types of use cases, the ability to deploy native Clojure
executables is not as important as the ability embed a low-resource
runtime, the ability to call C/C++ code, the ability to sandbox the
|Re: Natively Compiled Clojure||AtKaaZ||1/27/13 5:05 PM|
although I'm sure everybody's seen this, I believe it is relevant here, this clojureconj by Chris Granger
maybe only applies to clojurescript(that is, being slow in this case)
the important stuff is at from 13:59
I may be wrong or incomplete.
Please express any corrections / additions,
they are encouraged and appreciated.
At least one entity is bound to be transformed if you do ;)
|Re: Natively Compiled Clojure||Mikera||1/27/13 6:06 PM|
On Monday, 28 January 2013 07:56:58 UTC+8, tbc++ wrote:
The hard limits I have hit in Clojure are around three specific areas:
1. GC latency on the JVM (usually not a problem, but can be an issue for games and realtime work)
2. Startup latency
3. Access to hardware level features (GPU access, OpenGL etc.)
1. Can be avoided by avoiding memory allocation (there are Java libraries that do this like Javaolution), but this is really hard to do in idiomatic Clojure. Even something as simple traversing a single vector as a sequence is O(n) in new object allocations.
2. Is actually more about the time taken to load / compile the Clojure core libraries than it is about the JVM. On my machine, the JVM itself starts up in about 0.1 sec which is fine for anything other than chaining thousands of script invocations. Generally I think that people who complain about JVM startup time probably have the wrong architecture - you don't spin up a new virtual server to handle every web request, do you?
3. Can be done, but it's a pain to configure
Personally, I think the advantages of being on the JVM vastly outweigh the disadvantages:
- Extremely fast and mature JIT compiler / GC algorithms
- Huge library ecosystem
- Easy portability across platforms
Hence what I'd really like to see are better ways to integrate and / or generate native code within JVM Clojure. In particular:
- Better ways to load / execute native code libraries at runtime
- Macros / tools for accessing natively managed memory while still keeping a Clojure style
- A really good JNA wrapper (clj-native seems along the right track.....)
- A native code compiler that can be executed at runtime. So we can actually compile to native code and run it dynamically. Possibly something based on LLVM would work well here.
If we had all that, we would genuinely have the best of both worlds.
|Re: Natively Compiled Clojure||Konrad Hinsen||1/28/13 12:23 AM|
Timothy Baldridge writes:Another approach that might be worth exploring (for someone who has
lots more time than me for such a project) is hijacking the
implementation of the Julia language (http://julialang.org/). It's a
language with many Lisp inspirations, despite its non-Lisp syntax, and
it's based on LLVM with no virtual machine in between, but a run-time
system with a garbage collector. It's data model is much more similar
to Clojure's than to classic OOP, with all functions essentially being
multimethods. A native Clojure implementation could probably re-use
much of Julia's implementation.
|Re: Natively Compiled Clojure||Mikera||1/28/13 12:35 AM|
On Tuesday, 22 January 2013 00:29:54 UTC+8, octopusgrabbus wrote:
On this general topic: the following project (VMKit) looks very interesting:
It basically implements a full JVM on top of LLVM, which should in theory be sufficient to run Clojure on the LLVM.
This opens some interesting opportunities for combining Clojure with the LLVM infrastructure for native code generation.
|Re: Natively Compiled Clojure||Paul deGrandis||1/28/13 6:54 AM|
I personally have no limitations with Clojure and the JVM for general development.
Like Mark mentioned, I have a need to embed runtimes within C systems, with a small memory footprint (ideally less than 5 Mb) fast execution, and a solid FFI.
Giving up some raw performance to get some benefits of Clojure is a worthy trade-off for me (macros, persistent data structures, etc). Concurrency would be awesome if it existed, but it's not a requirement - I can always fall back on lower-level/error-prone/hand-controlled concurrency mechanisms within the C system itself.
In other systems I've been working on where the core library is C and I have a lot of memory available, JNA/JNI/BridJ has been fine as long as I've followed best practices regarding those technologies.
When I needed a smaller memory footprint, and could give up some speed, ClojureScript + Node.js has been awesome for throwing quick things together. I'm very much inclined to switch this to clojure-py2 in the future if language-feature support is on-par with CLJS or there was a speed/memory benefit.
Unlike Mike, the small amount of OpenCL stuff I've done has been fine - I'll be doing more in the future, so that may change :)
It took some time to adjust to best practices to make the GPU worth it. Hopefully Project Sumatra will make this a moot point: http://openjdk.java.net/projects/sumatra/
|Re: Natively Compiled Clojure||Konrad Hinsen||1/28/13 8:08 AM|
Mikera writes:I tried this a while ago and had some exchanges with VMKit authors. Back then,
VMKit crashed on Clojure initialization because of a memory limit in VMKit
concerning the size of a single method. Clojure initializes tons of data in a
single JVM method.
But I agree that VMKit is an interesting project in principle. Someone will have
to take up active development again to make it more usable.
|Re: Natively Compiled Clojure||Aaron||2/12/13 12:50 PM|
I have been working on a library for ClojureCLR that actually uses a Clojure DSL to generate, compile, and load C code live into a running REPL. It's called c-in-clj (see https://github.com/aaronc/c-in-clj). It works quite well for me so far, but is still in what I would call Alpha stage. Eventually, this code may be used to compile Clojure code to a native binary via a C compiler.
You can also use this little library that I have for ClojureCLR to do painless native function calls via the CLR's P/Invoke mechanism: https://github.com/aaronc/ClojureClrEx.
On Sunday, January 27, 2013 6:59:09 AM UTC-5, Konrad Hinsen wrote:
Timothy Baldridge writes: