Porting Clojure to Native Platforms

[JvJ]

I've been wondering lately about getting Clojure to compile to native code, and so I've been doing some looking around.

There are a few projects that are on the right track, such as TinyClojure and Ferret, but they are incomplete and don't seem to be under active development any more.

I'm wondering about the requirements, how much work it would take, and whether or not it would be worth it.  My current thinking is of a Clojure->C/C++ compiler.

So far, I have a few topics for discussion that I'm unclear about that might be necessary for this kind of project:

• Can the Immutable Persistent Data Structures be implemented with a reference-counting scheme rather than a garbage collector?  This may improve performance.
• In a similar vein, can the allocation strategy be controlled for these structures in such a way as to optimize cache locality?
• Can we take advantage of tail-call optimization in existing C++ compilers?
• It wouldn't have to depend on an existing runtime environment like JVM or JavaScript.
• Could this reduce reliance on reflection and increase performance?
• Could a new, clojure-optimized runtime be created that improves performance?
• Could certain anonymous functions be optimized or inlined in a way that improves performance over JVM/JS implementations?
• Is there a way to compile C++ code at runtime?  This would be essential for the REPL and for Macros.

Let me know if anyone has any thoughts on the matter.

Thanks

[Dan Girellini]

Not knowing the problem you’re specifically trying to solve, would using using the GNU java compiler work to take byte code to native?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jason Felice]

There was talk of an LLVM backend a while back, but I believe LLVM was deemed too low-level to be useful.  That was, in fact, why I signed the contributor agreement.  So, I'd love to see some movement on a C back-end.

Gambit Scheme has a special form that emits C (or C++) code.  It's very useful.  I can't remember which Lisp it was, but there was definitely a lisp which had an emit-C as it's *only* special form.  It might have been one of the embedded schemes.

Last time I looked, I also saw ClojureC:  It looked useful, but hasn't been touched in 3 years.

https://github.com/schani/clojurec

And here's a clojure-to-scheme compiler that used Gambit to target native:

https://github.com/takeoutweight/clojure-scheme

(In fact, this might be what I was thinking about with the special form above.)

IMHO, native is useful for embedded systems and command-line programs (where JVM start-up time can make a comand simply unusable).

--

[Raoul Duke]

things like robovm are another possible approach.

[JvJ]

I wasn't trying to solve any particular problem.  I was just wondering about the possibility of it.

The main motivation would be performance gains.

[Raoul Duke]

> The main motivation would be performance gains.

blah? so many impedance mismatches and layers of indirection that i
don't think it will gain much? i mean, it would probably be better to
spend time tuning gc parameters or something. just a rant / guess.
e.g. robovm is for some use cases perfectly fine performance wise
believe it or not.

[Timothy Baldridge]

As someone who has spent a fair amount of time playing around with such things, I'd have to say people vastly misjudge the raw speed you get from the JVM's JIT and GC. In fact, I'd challenge someone to come up with a general use, dynamic language that is not based on the JVM and comes even close to the speed of Clojure. 

A LLVM/C++/RPython based version of Clojure would on a good day come in at about 1/10 the speed of Clojure on the JVM for general use cases. 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[lvh]

Hi Tim,

On Apr 25, 2016, at 3:50 PM, Timothy Baldridge <tbald...@gmail.com> wrote:

A LLVM/C++/RPython based version of Clojure would on a good day come in at about 1/10 the speed of Clojure on the JVM for general use cases. 

Whoa! The RPython one is particularly interesting; are those the figures you saw out of your efforts in porting Clojure to PyPy? Did you ever get a chance to interact with the PyPy folks when you saw such (IMHO) pathological performance figures? (No blame; a genuine question!)

lvh

[Timothy Baldridge]

I wrote Pixie and did a fair amount of benchmarking on it. As it stands tracing JITs work well on tight loops. That means in a perfect world you would only have one hot trace through a given set of functions. That's actually harder to pull off than it sounds. 

Take for instance the PersistentHashMap. If you dig into that implementation you'll see dozens of if statements that are hit in different conditions. If you're adding a new branch, adding a new leaf, copying nodes of different types (there are three node types). So simply adding to a hashmap requires dozens of conditional branches and the result of those branches changes every time you add a new item. 

So what I ended up doing for Pixie is disabling the JIT when digging into the hashmap implementations. The PHM was implemented in RPython directly and the JIT wouldn't try to optimize it. However, sometimes the JIT calls back into user code (equals comparisons, and hashcode calculations), so then you have to re-enable the JIT in a place where there isn't much of a loop context. 

There's many other locations like that. You may think you have a simple loop, but then somewhere deep in your code you have a call to update-in, which creates a lazy seq for the path, then iterates on that while digging into several hashmaps and boom, the JIT just bails out. 


The more I read up on it though, the more I see examples of tracing JITs working poorly for "application code". That is, normal code you'd run in a webserver. That sort of stuff just has so many if statements that the JIT almost always optimizes the wrong thing.

[JvJ]

Interesting point about the performance of JIT and GC on JVM.  I didn't realize that they could be so highly performant.

Also, I had thought "Metal Clojure" would have been a good name for this project, but you went ahead and stole it before I even thought of it!

[Gregg Reynolds]

On Apr 25, 2016 3:50 PM, "Timothy Baldridge" <tbald...@gmail.com> wrote:
>
> As someone who has spent a fair amount of time playing around with such things, I'd have to say people vastly misjudge the raw speed you get from the JVM's JIT and GC. In fact, I'd challenge someone to come up with a general use, dynamic language that is not based on the JVM and comes even close to the speed of Clojure. 
>
> A LLVM/C++/RPython based version of Clojure would on a good day come in at about 1/10 the speed of Clojure on the JVM for general use cases. 
>

Now that's pretty darned interesting.  Next month I'm going to start doing some serious work with the Java APIs on the Intel Edison.  Java is one of the std environments for Edison.  But naturally I'm very interested in finding out if what works in theory works in practice.  Running a Clojure nrepl on an Edison is obviously a big win, in theory.  Is std Clojure fast enough for iot stuff?  We'll see.

The other interesting doodad is the Curie.  Now in the Arduino 101, but due to have an RTOS soon, see the zephyr project.  My pipedream is to write Clojure code for Zephyr. Is that insane?

-g

[Michael Ball]

Not JVM based but I've been wondering about ClojureCLR on top of the new CoreCLR might be a potential path for native. There's a native compiler in the works (a brief mention a couple weeks ago in the standup https://youtu.be/cfxuJsZIK4A?t=1h13m32s). 

I have to say with the MIT license, tail call support, native compilation, and the open development CoreCLR could have a bright future as a host platform, even though it's not nearly as widely used as the JVM today.

[Mark Engelberg]

On Mon, Apr 25, 2016 at 1:50 PM, Timothy Baldridge <tbald...@gmail.com> wrote:

As someone who has spent a fair amount of time playing around with such things, I'd have to say people vastly misjudge the raw speed you get from the JVM's JIT and GC. In fact, I'd challenge someone to come up with a general use, dynamic language that is not based on the JVM and comes even close to the speed of Clojure. 

Julia reportedly outperforms Clojure (I haven't benchmarked it myself).  Julia is designed primarily for numeric computation, but unlike a lot of other "math languages" like Octave, R, and Matlab whose general programming constructs are horribly slow, the overall general-purpose machinery of Julia (function calls, looping, data structures, dispatch, etc.) is said to be quite fast. 

Speaking of numeric computation, it is, in my opinion, a real weak point of Clojure.  There is a huge perf. penalty for boxed numbers, and since anything in a Clojure data structure gets boxed, it's rather difficult to work around Clojure's poor numeric performance for anything more algorithmically complicated than a super-simple loop.  A number of dynamic languages use tagged numbers (primitive types with a couple bits set aside to mark the type of the number), and these are going to do much better than Clojure for a wide variety of math-oriented computational tasks.  I've benchmarked some specific programs in Clojure vs Racket and Racket tended to come out ahead when there was a lot of arithmetic. 

[Ilya Ivanov]

I have no first-hand experience with it, but isn't Excelsior JET what you're looking for? They claim to provide AOT compilation to native binaries. 

[Mars0i]

On Monday, April 25, 2016 at 3:50:45 PM UTC-5, tbc++ wrote:

As someone who has spent a fair amount of time playing around with such things, I'd have to say people vastly misjudge the raw speed you get from the JVM's JIT and GC. In fact, I'd challenge someone to come up with a general use, dynamic language that is not based on the JVM and comes even close to the speed of Clojure. 

I was going to say that I'd be surprised if Clojure were as fast as SBCL (overall, on average, depends on your application, depends on how you code it, ymmv, etc. ...).  Then I stopped back to check the little benchmarks on the Computer Language Benchmarks Game .  Whatever it is that those comparisons do, or don't prove, I would no longer be surprised.

[Mars0i]

On Tuesday, April 26, 2016 at 12:19:23 AM UTC-5, Mars0i wrote:

I was going to say that I'd be surprised if Clojure were as fast as SBCL (overall, on average, depends on your application, depends on how you code it, ymmv, etc. ...).  Then I stopped back to check the little benchmarks on the Computer Language Benchmarks Game .  Whatever it is that those comparisons do, or don't prove, I would no longer be surprised.

I forgot how much faster Clojure got in 1.7 and 1.8.  I remember Java wiping the floor with Clojure on most of those benchmarks a couple of years ago, but now it's just a little bit faster on average on those benchmarks.

[Niels van Klaveren]

The only conclusion I can draw from these benchmarks over the years is that the speed of Clojure is inversely proportional to it's idiomaticity

[Colin Fleming]

This is true of a lot of those benchmarks, especially for functional languages. I haven't looked recently, but back in the day Haskell seemed very fast on the shootout game. However the code that actually runs that fast is very very far from idiomatic. Again, I haven't looked at the Clojure versions recently but I recall several of them being basically Java written in Clojure form, i.e. lots of interop and not a lot of the sort of higher-order code that makes Clojure such a pleasure to program. The rules around functions accepting and returning primitives are way into expert-level territory.

--

[Mikera]

I would definitely second Tim's points. The JVM is very hard to beat once you factor in the GC and JIT requirements.

Worth noting that persistent data structures with structural sharing are used pretty much ubiquitously in Clojure and that these are *exactly* the kinds of data structures that benefit most from GC. 

An alternative scheme such as reference counting would perform extremely badly in comparison because structural sharing implies a *lot* of reference count updates, and writes to reference counts scattered across memory are expensive on modern machines (especially in concurrent situations). 

[Dragan Djuric]

On the other hand, Julia relies on external native libraries for performance (BLAS, GPU, whatever). The speed of those libraries is not due to the language (usually C or ASM) but to the fact that they use algorithms that are heavily optimized for each hardware architecture. If you'd write vector and matrix functions in plain everyday C, you'd get the same speed as with Java arrays in Clojure or Java, sometimes even slower. In Cloure and Java, you can link to those same libraries that Julia links to, with the same tradeoffs.

[Aleksander Sumowski]

A scala native port is due to be announced soon. It would be interesting to see whats their approach and performance characteristics are:

http://www.scala-native.org/

Cheers,

Aleksander

uSwitch is a trading name of uSwitch Ltd. Registered in England and Wales (Company No. 03612689). Registered Address: Notcutt House, 36 Southwark Bridge Road, London, SE1 9EU

This communication and any attachments contains information which is confidential and may be subject to legal privilege. It is for intended recipients only. If you are not the intended recipient you must not copy, distribute, publish, rely on or otherwise use it without our consent. Some of our communications may contain confidential information which it could be a criminal offence for you to disclose or use without authority. If you have received this email in error please notify the sender immediately and delete the email from your computer.

uSwitch Ltd reserves the right to monitor all email communications for compliance with legal, regulatory and professional standards.

[Nurullah Akkaya]

My 2¢ on the subject,

Ferret is not dead, I've been working on it for sometime now. Latest builds/docs are available at [1].

> Can the Immutable Persistent Data Structures be implemented with a reference-counting scheme rather than a garbage collector?  This may improve performance.

That is what Ferret uses. But reference-counting is not cheap. This snippet,

    (reduce + 0 (range 10000))

will spend most of its time inc/dec reference counts (When running in single threaded mode with a memory pool.).

> In a similar vein, can the allocation strategy be controlled for these structures in such a way as to optimize cache locality?

Currently when running on embedded systems or in single threaded mode ferret can be configured to run using a memory pool. This avoids calling malloc/heap at runtime, improves performance and determinism you can also tell how much memory will be used at compile time.  In single threaded mode this also improves cache locality. (A linked list is just a sequential memory blocks from the pool allocated on the stack.)

> It wouldn't have to depend on an existing runtime environment like JVM or JavaScript.

Ferret uses its own runtime. No third party dependencies. Generated code is strict ISO C++11.

>  Could certain anonymous functions be optimized or inlined in a way that improves performance over JVM/JS implementations?

Ferret does simple escape analysis when you are not passing a fn as variable, it is created on the stack. A ferret function is a C++ functor and no runtime lookup is necessary, compiler should be able to inline them depending on the optimization level.

> Is there a way to compile C++ code at runtime?  This would be essential for the REPL and for Macros.

This is not necessary for macros. Ferret is intended for embedded systems a REPL is not really useful when you have to jump through hoops to communicate with the device. Ferret does supports macros. Everything is compiled there is no reader or eval, you can't add/remove/resolve symbols during runtime. (Unless you setup your own system for it.)

> The main motivation would be performance gains.

Speed is not my main motivation. Ferret is not lighting quick but allows you to  easily interact with C,C++ libraries or drop in arbitrary C++ code to squeeze that last bit of performance for critical sections of code. i.e As someone already said in the thread with ferret you can take the julia route.

> Julia relies on external native libraries for performance (BLAS, GPU, whatever). The speed of those libraries is not due to the language (usually C or ASM) but to the fact that they use algorithms that are heavily optimized for each hardware architecture

[1] http://dropbox.nakkaya.com/builds/ferret-manual.html

Best,

--

Nurullah Akkaya
http://nakkaya.com

[Alex Miller]

I rewrote most of the Alioth programs back in the Clojure 1.6 timeframe to get them in the ballpark of the Java programs. I have timing comparisons across 1.6/1.7/1.8 somewhere but the differences are slight (and not all improvements) - really the subsequent Clojure versions had little effect on the program timings because of the kind of programs they are.

Almost all of the Alioth programs are dominated by tight primitive math loops - the bytecode produced by those programs is very similar to the output from Java (or Scala for that matter) and thus there's little advantage left to gain.

Most of the perf differences between the Java/Scala and Clojure versions are attributable to Clojure startup time overhead. As the total program timings are mostly in the 5-10 second range, a half second of additional startup time is a big factor, percentage-wise. When I've tested lazy var prototypes, that definitely helps a bit in reducing this overhead.

Regarding the idiomatic-ness of these programs, I think it's a fair point that fast primitive math loops do not fit well with the typical sequence (or even transducer) model of data manipulation due to boxing. Rich has done some design work on supporting primitive (long/double) transformation functions and eventually (not 1.9) that will probably surface in the language. When it does, things like primitive vectors will become far more useful than they currently are.

I think the compensating factors are that 

1) 99% of most typical (your "typical" may vary) programs is not this kind of code

2) it is possible to write most Java primitive/array code in Clojure loop/recurs and see similar performance to Java

3) you can always simply write that hot loop in Java or invoke a native lib when that is necessary

[Raoul Duke]

RC & GC might complement. Don't throw out RC. Also, there are different kinds of 'performance'. Horses for courses, you know.

https://www.google.com/search?q=bacon+gc+reference+counting+equation

[Gregg Reynolds]

Wow.  I was going to suggest Lua, but according to the benchmarks it's not even in the same league.  I wonder what the benchmarks would look like if they included calls out to C libs.

[Raoul Duke]

Horses for courses. Ask all the game people who use Lua big time. :-)

[Timothy Baldridge]

> Ask all the game people who use Lua big time....

See that's the problem. When we say "horses for courses" we have to actually dig down into what we're saying there. The reason Lua is used in the gaming community is that it's quite fast, very very small, and simple for what it does. It also has a fairly decent JIT (in the form of LuaJIT). But it's still an interpreted language, with a very basic set of data-structures (basically hash-maps for everything even arrays). 

So yeah, the reason it's used in the gaming community is that integrating the interpreter is fairly easy, and it's "fast enough" to be used for game logic. But I seriously doubt anyone picked Lua because it had a world-class GC (because it doesn't), or because out-performed other languages. 

On Tue, Apr 26, 2016 at 2:30 PM, Raoul Duke <rao...@gmail.com> wrote:

Horses for courses. Ask all the game people who use Lua big time. :-)

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gregg Reynolds]

On Apr 26, 2016 3:30 PM, "Raoul Duke" <rao...@gmail.com> wrote:
>
> Horses for courses. Ask all the game people who use Lua big time. :-)
>

Sorry, never heard of horses for courses.  Does it mean sth like different strokes for different folks?  Re: big time users of Lua, I don't know any.  I guess performance isn't so great?

This thread may save me some serious heartache.  As I mentioned previously I'm about to start a project using the Intel Edison, and maybe also the Curie. I'm definitely going to experiment with Clojure but I've been kinda assuming that Lua might be better than Clojure for such constrained environments.  But maybe I assumed too much.

On the other hand speed is only one aspect of performance, maybe not so important for IoT stuff.  Space and energy are just as important, maybe more so.  I wonder if there are any benchmark comparisons for those.

gregg

[Raoul Duke]

> Sorry, never heard of horses for courses. Does it mean sth like different
> strokes for different folks?

yessir.

[Timothy Baldridge]

I wouldn't underestimate ClojureScript running on V8. That's another platform that's quite fast. And there are a lot of people trying very hard to make JS engines fast, could consider leveraging that. 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Weilbach]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 25.04.2016 22:02, Jason Felice wrote:
> There was talk of an LLVM backend a while back, but I believe LLVM
> was deemed too low-level to be useful. That was, in fact, why I
> signed the contributor agreement. So, I'd love to see some
> movement on a C back-end.
>
> Gambit Scheme has a special form that emits C (or C++) code. It's
> very useful. I can't remember which Lisp it was, but there was
> definitely a lisp which had an emit-C as it's *only* special form.
> It might have been one of the embedded schemes.
>
> Last time I looked, I also saw ClojureC: It looked useful, but
> hasn't been touched in 3 years.h

Just because you mentioned it, I played a lot with the ClojureC
compiler and implemented the metacircular evaluator for it two years
ago. There is a really nice REPL for it:

https://github.com/bertfrees/cljc.repl

But ClojureC is not fast, because statically compiling all the
abstractions in (look at the emitted C code), makes it slow. You can
directly call C though and get a self-contained binary. I guess Ferret
is better for embedded development though.

I found mjolnir really interesting and would like to see a way to
build native building blocks from Clojure that way. At least one could
implement performance critical numeric routines directly in Clojure
including annotations and call them from the JVM (e.g. on the GPU).
This should cover a lot of what Julia does, but I am not familiar
enough with its internals (e.g. JIT and how it standardizes tensor
memory layout etc.).


Cheers,
Christian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJXH99KAAoJEICbLivqiPOFTNUP/ik2l/Bg528fvcBlX4WG+kOr
zLlNL7YgXu+rx/kZ+yVZuN/IQo5MW/TdRmOuZz7qkqb0nOuYNJ2J/fij4pUXISnn
tvlgVF4Pasj4sV0e/i4ZL5vo7T8KMd4UIb2D8YFdmNMXXFUw1OgBf10ucQvtljxt
eHGiDIf2etLYIfZupJAS5jtZqW17FK+K10sTZGdd5ZlCMUctl708BPXLlwayjpXM
erIXc7/hS5YIemC0iHEgNLYfyFLxIPa/Jid6PW3kZ7bJeB1QmwBTn472nI9GYMln
LF3DVx6QWK9B210qFz4HemgEKhkKj8/wNRR2uNpsBsYtYoMkUvB2tGeWuw6620G+
YFA2VOsVGpHaPPBxPgJLpT6/mN+ELmpmHdDk+L/UlFHw4eP/V/FlCNc1rnoR0HlM
j9Sqz11eJyy4rZRINC1N6PyXNDlTAFVOUypbb9dTAx7s2h/nBvqqLRAuMQ2kHHPf
8+Mp13EsSu2AaHDW/rFkM158zARn04gHiuFvtlCSjqZY6aCSCYAfKgY+v6YKEG1T
lhqyOlVo9i/tXCDCE3xov30Tp1BdqHDjuGyckfkkxFAJdzoylbuyCewR5GMppyie
FH93BZ63niWtwVXo8vuMAQHkiJnnVcw9BNqlFCI5cJH3LUTx0Kt/Jk1LhKxs0256
TZg/fBZqNQAYQtHMorJ1
=VsLc
-----END PGP SIGNATURE-----

[Rangel Spasov]

tbc++ - given your experience, would you consider a Clojure port to Erlang VM a viable idea for production workloads? I know Elixir comes pretty close, but I still prefer Lisp : ) .

[Colin Fleming]

Of course, most people using Lua seriously use LuaJit, which is surprisingly fast. However it's very prone to the branching problems you described earlier for complex application code. I believe LuaJit does optimise hashmaps when they're used as arrays etc. LuaJit really shines when used for something with a really tight inner loop with perhaps a couple of main branches - it's used for packet filtering, for example, and it's possible to get more or less the same performance as C under those restricted conditions but with dynamic adaptation to traffic conditions for things like DDOS attacks.

One thing that LuaJit does offer is much better control over memory layout than Java/JS etc. You can manually allocate blocks of memory and treat them as arrays of floats etc - it would be interesting to see how fast it would run something like a raytracer or a mandelbrot calculation, I'd imagine it could get pretty fast.

[Plínio Balduino]

Hi, J

Based on my yet small experience:

Can the Immutable Persistent Data Structures be implemented with a reference-counting scheme rather than a garbage collector?  This may improve performance.

- Yes, using Smart Pointers. I'm trying something like this, but I'm still in the very beginning. I don't have any info about the behavior of these structures under heavy usage yet.

In a similar vein, can the allocation strategy be controlled for these structures in such a way as to optimize cache locality?

- AFAIK, not with the default memory allocator. There are some custom memory allocators for C/C++, but I never used it.

Can we take advantage of tail-call optimization in existing C++ compilers?

- Since you can't control by default the call stack, in a first moment you can't do it. You could do it writing your own way to control the call stack, where it surely will make you develop your own virtual machine or your own compiler with this optimization. Two herculean tasks IMHO. Or you can repeat the Clojure approach and use explicit loop/recur implementing it internally as a loop.

It wouldn't have to depend on an existing runtime environment like JVM or JavaScript.

* Could this reduce reliance on reflection and increase performance?

- You will need to implement your own way to control reflection. There's no guarantee that this could improve performance.

* Could a new, clojure-optimized runtime be created that improves performance?

- No idea

* Could certain anonymous functions be optimized or inlined in a way that improves performance over JVM/JS implementations?

- Since the compiler is yours, yes. But I don't know how much effort it would take.

* Is there a way to compile C++ code at runtime?  This would be essential for the REPL and for Macros.

- I never heard about on demand native code generation and execution. I think it could be a paradise for viruses, but I know almost nothing about this field. Unless of course you're generating code for (your own|a) virtual machine, so the rules are different.

I hope I helped and I really would like to see this project, if you start to develop it.

Regards

Plínio

On Mon, Apr 25, 2016 at 4:47 PM, JvJ <kfjwh...@gmail.com> wrote:

I've been wondering lately about getting Clojure to compile to native code, and so I've been doing some looking around.

There are a few projects that are on the right track, such as TinyClojure and Ferret, but they are incomplete and don't seem to be under active development any more.

I'm wondering about the requirements, how much work it would take, and whether or not it would be worth it.  My current thinking is of a Clojure->C/C++ compiler.

So far, I have a few topics for discussion that I'm unclear about that might be necessary for this kind of project:

• Can the Immutable Persistent Data Structures be implemented with a reference-counting scheme rather than a garbage collector?  This may improve performance.
• In a similar vein, can the allocation strategy be controlled for these structures in such a way as to optimize cache locality?
• Can we take advantage of tail-call optimization in existing C++ compilers?
• It wouldn't have to depend on an existing runtime environment like JVM or JavaScript.
• Could this reduce reliance on reflection and increase performance?
• Could a new, clojure-optimized runtime be created that improves performance?
• Could certain anonymous functions be optimized or inlined in a way that improves performance over JVM/JS implementations?
• Is there a way to compile C++ code at runtime?  This would be essential for the REPL and for Macros.

Let me know if anyone has any thoughts on the matter.

Thanks

--

[Timothy Baldridge]

>> would you consider a Clojure port to Erlang VM a viable idea for production workloads? I know Elixir comes pretty close, but I still prefer Lisp : ) .

I looked into that at one point, but sadly the Erlang VM is a really poor platform for a Clojure port. Here are a few reasons why:

Unit of compilation is the module, not sure how that all plays out with the Elixir repl, but it's not optimal. From what I can tell, re-deffing a defn would require re-loading an entire namespace (module) or require one-defn-per-module. 

No native polymorphism. Elixir fakes it with structs (think defrecords) with a custom hidden field, then looks up functions in a hash map on that field to dispatch.

Total lack of shared state. This one doesn't sound like a problem till you think of how to do things like a REPL that can re-def defns at runtime. Also, clojure makes fairly liberal use of atoms, vars, and volatile! cells. Even stuff like lazy seqs leverage limited mutability and set-once behaviors. You don't have any of that in BEAM, just actors. And converting all those things to actors would be a major mistake. 

So can you have a lisp flavored Erlang? Sure. But what you're left with would not be Clojure. 

Timothy

[Plínio Balduino]

LISP Flavoured Erlang:

http://lfe.io/

[Rangel]

Thanks for the detailed write up! I guess a carefully chosen subset of
Clojure can make sense then, but definitely not all of Clojure.

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/8DUkKiXV77U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Colin Fleming]

Unit of compilation is the module, not sure how that all plays out with the Elixir repl, but it's not optimal. From what I can tell, re-deffing a defn would require re-loading an entire namespace (module)...

Reloading a whole namespace on an evaluation doesn't seem like a show-stoppingly bad idea though, does it? We're only talking during interactive use at the REPL, if it takes 100ms to evaluate a form I don't think anyone is going to care that much. I mean, at the REPL no-one complains about how long require takes except in extreme cases, and that's generally reloading many namespaces. Of course, in a reload-entire-ns model you're likely to have to reload namespaces that depend on the one you're reloading as well, but I'm still not sure it's so horrible.
 

[Stig Brautaset]

Colin Fleming <colin.ma...@gmail.com> writes:

> Unit of compilation is the module, not sure how that all plays out with
> the Elixir repl, but it's not optimal. From what I can tell, re-deffing a
> defn would require re-loading an entire namespace (module)...
>
> Reloading a whole namespace on an evaluation doesn't seem like a
> show-stoppingly bad idea though, does it?

It means the whole namespace has to compile at all times, as opposed to
just the form I'm currently working on. Not show-stoppingly bad
perhaps, but certainly workflow-altering.

Stig

[Charles Harvey III]

You can watch Robert Virding's talk about creating Lisp Flavored Erlang (LFE).

https://www.youtube.com/watch?v=Br2KY12LB2w

It is really informative. It tells you a lot about Erlang and the BEAM. And it explains why LFE is not Clojure and why Clojure would not work.

Virding really likes his Erlang as you can see from his Github profile: https://github.com/rvirding?tab=repositories
He's ported Lisp, Lua and Prolog to the BEAM.

[Jason Felice]

On Tue, Apr 26, 2016 at 7:42 PM, Plínio Balduino <pbal...@gmail.com> wrote:

* Is there a way to compile C++ code at runtime?  This would be essential for the REPL and for Macros.

- I never heard about on demand native code generation and execution. I think it could be a paradise for viruses, but I know almost nothing about this field. Unless of course you're generating code for (your own|a) virtual machine, so the rules are different.

 

There is a Scheme (I can't remember which) that generates C, spawns the compiler, links and loads a dynamic library, then invokes the code.  I'm having a brain fart because I want to say it's Gambit but Gambit actually has an interpreter.  Maybe it was CHICKEN?

[Max Penet]

He happens to be one of the original Erlang authors/designers.

There's also a side project of LFE that attempts to make things more clojur'esque in LFE https://github.com/lfex/clj. As for right now LFE shares more similarities with Common Lisp than with Clojure or Racket (lfe is a Lisp2). Personally I find more pleasant to work with Erlang directly at the moment, even coming from Clojure. The syntax is very simple once you get used to it (I heard that before about parens :)).

I know that LFE has a way to define functions in the REPL now, it's something fairly new, but this might just be a hack for repl development, I have no clue how it's done under the hood. 

[Plínio Balduino]

Racket has this feature.

[Colin Fleming]

That could be hidden from the user by tooling though, perhaps by recompiling the previous version of the namespace with just the form the user wants to load modified. That would also fix one serious problem with per-form evaluation, which is that the line numbers for subsequent forms get messed up (assuming that the form you're working on is a different number of lines to the previous version). This affects exception stacktraces and debugging.

Also, anyone using CLJS and Figwheel is already loading ns-at-a-time. It's a slightly different workflow, but most people seem to love it.

[Mark Engelberg]

Relatedly, Chez Scheme was just released as open source:
https://github.com/cisco/chezscheme

which is a native code optimizing scheme compiler.  Maybe there are some ideas that can be drawn from this.

[Christian Weilbach]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

https://github.com/bertfrees/cljc.repl also works this way.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJXIwRVAAoJEICbLivqiPOFyo8P/0PgnVHkAroD+fTUTFR5ULjX
CzrEBycHYd4ot1qbRKGW4kNJ/MC+NhuT7TPGcMbnnhULrTZX+A04NTaXaUhKahtO
pO4xspkxjXRJdXQwno7AyqLSOnutdSvjU3aECsqTJN17dxt0+YNjMvyb1dp1PHMr
2aofDx6Pu4UwEbBtqDe8qPWADwcBhNj+Ci5oT447qFZyBD4V1+wQwsdX6xJjiYz3
nyghxasZv9kx8U+qNeBEOKTcPKoD7N+WQs5AjcIq9aRiHNcIf0xAKztd68/+bpg7
lzR46fQg6McdzV8ohomcajlpeg0SMe2hOMmugAMUab9Wuz9JzTMBq0RuZ6GZEjQ6
vGVQTwqfVqbD1dU48MmSPivtd8Ijou03dtq/CuwV3//QePUqUqgKUA8OMdoiQlNv
s124MznjQHfA+YkZxNCo5+7Atc9+OafsIlfH7+WDnmJcy3kDYStksKuNRX+heIpH
j4Fv5e9ILvUHYltgM0Pi1VozpLXodS9d9t/IRyG/SZtk/JqUmmHohNbYGzLuLT1Q
dXXJoVxRLo+qHjaT0ZdmKyRayonVNaV7yLczXVyqWjKVLM8Q3NbanaRQofzSM6jD
M9KtW4CRc22skfWaPKS/pNMJjvkIMHmgyMSwD+HF0rG1SIh06u1Qno/1Wqf7ciiF
u/dBpT3Li4ELAkJVojMI
=3h7o
-----END PGP SIGNATURE-----

[John Gabriele]

On Monday, April 25, 2016 at 4:12:13 PM UTC-4, JvJ wrote:

The main motivation would be performance gains.

Sounds like competing performance-wise with the JVM is extremely difficult.

My best guess is that a successful new Clojure implemention will have these qualities:

  * be interpreted (performance not a goal --- already have Clojure)
  * be interpreted (compilation not a goal --- already Ferret, ClojureC, etc)
  * not run in a browser (already ClojureScript)
  * not compile to $other-language/bytecode (already Hy, clojure-scheme)
  * not be designed _only_ for embedding (already TinyClojure (... TinyScheme, ... Lua))
  * easy interop with C/native libs
  * fast startup
  * small footprint
  * probably not a _complete_ subset of Clojure, and that's _OK_.
  * have a common, community-accepted free software license, such as GPL3 or LGPL3

It's main drawback would be lack of libraries at first, but people really like writing Clojure, so I suspect it wouldn't be a problem for long.