[ANN] ClojureC - A Clojure to C compiler - Work in Progress

[Mark Probst]

Dear Clojurians,

I'm excited to announce ClojureC, an effort to produce a Clojure implementation that targets C:

  https://github.com/schani/clojurec

My personal goals with this are to be able to write self-contained (command-line) Clojure programs that have (essentially) zero start-up time, as well as to have a Clojure implementation that plays well with iOS (no work on Objective-C bindings has been done so far, but I have plans...).

So far the best overview of how far along ClojureC is is the test suite:

  https://github.com/schani/clojurec/blob/master/test/clojurec/core_test.clj

as well as cljc.core:

  https://github.com/schani/clojurec/blob/master/src/cljc/cljc/core.cljc

The ClojureC compiler as well as core.cljc are adapted versions of ClojureScript's code.  I intend to unify the compiler with ClojureScript's again, once things have settled.

Collaborators are very much welcome!

Cheers!

Mark

PS: I'm "schani" on IRC.

[Ambrose Bonnaire-Sergeant]

Cool!

Ambrose

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Baishampayan Ghose]

> I'm excited to announce ClojureC, an effort to produce a Clojure
> implementation that targets C:
>
>   https://github.com/schani/clojurec
>
> My personal goals with this are to be able to write self-contained
> (command-line) Clojure programs that have (essentially) zero start-up time,
> as well as to have a Clojure implementation that plays well with iOS (no
> work on Objective-C bindings has been done so far, but I have plans...).

This is incredibly exciting! Thanks for the great work, Mark.

Regards,
BG

--
Baishampayan Ghose
b.ghose at gmail.com

[Paulo Pinto]

On Jul 9, 1:03 pm, Mark Probst <mark.pro...@gmail.com> wrote:
> Dear Clojurians,
>
> I'm excited to announce ClojureC, an effort to produce a Clojure
> implementation that targets C:
>
>  https://github.com/schani/clojurec
>
> My personal goals with this are to be able to write self-contained
> (command-line) Clojure programs that have (essentially) zero start-up time,
> as well as to have a Clojure implementation that plays well with iOS (no
> work on Objective-C bindings has been done so far, but I have plans...).
>
> So far the best overview of how far along ClojureC is is the test suite:
>

>  https://github.com/schani/clojurec/blob/master/test/clojurec/core_tes...

>
> as well as cljc.core:
>
>  https://github.com/schani/clojurec/blob/master/src/cljc/cljc/core.cljc
>
> The ClojureC compiler as well as core.cljc are adapted versions of
> ClojureScript's code.  I intend to unify the compiler with ClojureScript's
> again, once things have settled.
>
> Collaborators are very much welcome!
>
> Cheers!
>
> Mark
>
> PS: I'm "schani" on IRC.

Great!

A question that I also mentioned on the HN thread, any idea to bypass
C and
compile directly to native, for example via LLVM?

--
Paulo

[Mark Probst]

On Mon, Jul 9, 2012 at 4:05 PM, Paulo Pinto <paulo....@gmail.com> wrote:
> A question that I also mentioned on the HN thread, any idea to bypass
> C and
> compile directly to native, for example via LLVM?

No particular plans. Why would you prefer that?

Mark

[Timothy Baldridge]

> No particular plans. Why would you prefer that?

Simplicity for one thing, self hosting for another. This is the
problem I see with a project like this:

1) My macros are written in Clojure for the JVM, my code is written in
Clojure for C
2) My code path is then ClojureScript->JVM Macros->C->GCC. So I'm
basically touching what...5 platforms/languages? (JVM, Clojure,
ClojureScript, C)
3) What if I have a syntax error in C? Now I have to understand how to
deal with syntax/runtime errors in 3 languages (JVM for macros,
Clojure for runtime errors, and C for compile errors).

With LLVM you can get rid of the C and GCC steps. And, if you use LLVM
bindings for Java, you can skip the source code output part of all
this. At that point your tool chain would consist mostly of JVM, and
Clojure and a few binary blobs. You can basically treat LLVM as an
AST.

As an example see this: http://code.google.com/p/pymothoa/

So the idea is, compile your stdlib to .o files, then have LLVM load
them. At this point it's a simple tree transform from Clojure's data
structures to LLVM, and from there to a binary output. There are no C
syntax errors, since there is no C code being compiled. Since errors
could be caught in the ClojureScript compiler, you could output
meaningful information like (error in line 33 of core.clj) instead of
worthless messages like (syntax error at line 13233 in FF3RXG.c).
Never been much of a fan of source-to-source translators to be honest.

Timothy

[Mark Probst]

On Mon, Jul 9, 2012 at 4:44 PM, Timothy Baldridge <tbald...@gmail.com> wrote:
>> No particular plans. Why would you prefer that?
>
> Simplicity for one thing, self hosting for another.

Self-hosting is orthogonal to this question. In principle one can
make the ClojureC compile run on ClojureC and be self-hosting.

> 3) What if I have a syntax error in C? Now I have to understand how to
> deal with syntax/runtime errors in 3 languages (JVM for macros,
> Clojure for runtime errors, and C for compile errors).

In a perfect world the compiler always outputs correct C, just like it
would always output correct LLVM intermediate code. In both cases
things can go wrong.

I do see advantages to using LLVM. Better GC integration would be
one, (better) self-containment of the toolchain, as you say, another.

Mark

[Paul Stadig]

Mark,
This looks interesting. Thanks for working on this.

I checked out the code and followed the instructions to setup glib2
and the GC. When I run `lein test` "I get Exception in thread "main"
java.io.FileNotFoundException: Could not locate clojurec/
core__init.class or clojurec/core.clj on classpath:"

Do you know what might be going wrong?


Thanks,
Paul

[Mark Probst]

On Mon, Jul 9, 2012 at 5:47 PM, Paul Stadig <pa...@stadig.name> wrote:
> I checked out the code and followed the instructions to setup glib2
> and the GC. When I run `lein test` "I get Exception in thread "main"
> java.io.FileNotFoundException: Could not locate clojurec/
> core__init.class or clojurec/core.clj on classpath:"
>
> Do you know what might be going wrong?

No, I don't think so. A stab in the dark: Are you using leiningen 2?

Mark

[Paul Stadig]

I'm using lein 1.7.1. Should I be using lein2?

[Mark Probst]

Yes. I'm sorry I forgot to mention that. Please let me know if it
works and I'll update the README.

Mark

[Paul Stadig]

On Mon, Jul 9, 2012 at 12:07 PM, Mark Probst <mark....@gmail.com> wrote:
> Yes. I'm sorry I forgot to mention that. Please let me know if it
> works and I'll update the README.

It works with lein2. The tests are running now, but there are some
failures. I'll keep poking at it.

Thanks!
Paul

[Paulo Pinto]

On Jul 9, 4:11 pm, Mark Probst <mark.pro...@gmail.com> wrote:

> On Mon, Jul 9, 2012 at 4:05 PM, Paulo Pinto <paulo.jpi...@gmail.com> wrote:
> > A question that I also mentioned on the HN thread, any idea to bypass
> > C and
> > compile directly to native, for example via LLVM?
>
> No particular plans.  Why would you prefer that?
>
> Mark

I would rather not depend on an external C compiler.

--
Paulo

[Paul Stadig]

The test failures seem to be because libgc wasn't on my
LD_LIBRARY_PATH. After I did that the tests all passed fine.


Paul

[Mark Probst]

Good to hear! Patch welcome ;-)

Mark

[j1n3l0]

Hi,

There is also a package for Ubuntu if you wish to install it that way:

  sudo apt-get install libgc-dev

Will be keeping an eye on this one :)

Nelo Onyiah

[Peter Taoussanis]

Man- the day there's a mature, production-ready C(something) compiler, I'm going to flip out. Can't wait.

Thank you for working on this!

[Eduardo Bellani]

I would take a look at bigloo and gambit, those implementations produce
solid C code from a Scheme base.

[charlie]

I think the benefits outweight the negatives, yes it's more pure going straight to LLVM and takes out one more compilation step  , but having it translate to C opens up the possibility of it running on any machine architecture that has a C compiler.

[Adam King]

  Please don't bypass C!  Almost all platforms have some sort of C compiler - it's like javascript for native platforms, instead of the web.  With C output, I can probably get it working on iOS (my main work target platform, along with Android) - I had Boehm GC compiled and working sometime ago for iOS - but ran into deadlines at work to continue.  C might even be a better platform for Android then the current Java version - though would need to run some startup, cpu and memory benchmarks to determine that.   And it might might also be usable for WindowsPhone8 using MSVC++ - but I don't much experience with that platform.  

  However, thanks for your work on this!  In the next couple of days, I'll see if I can get it compiling and running under iOS.  

  

[Mark Probst]

On Mon, Jul 9, 2012 at 7:03 PM, Adam King <aking...@gmail.com> wrote:
> However, thanks for your work on this! In the next couple of days, I'll
> see if I can get it compiling and running under iOS.

Wonderful!

If you're interested in working on Objective-C interop, let me know -
I have a few ideas.

Mark

[Adam King]

Hi Mark:

  I've spent a couple of hours on this - and have run into a snag.  I've got libgc compiling/linking with arm iOS and the Xcode project compiles cljc.c fine. The problem is glib.  From what I've read, iOS can not use glib as it must be statically linked into the app - but the LGPL doesn't allow that (at least from what I've read).  Currently, there only seems to be about 7 methods that you're using that need it - 6 of which at utf8 related.  So, I'm thinking that I'll make a 'prepreamble.m' and use NSString's utf8/unichar support for simulating those methods.  But before I spend time on that, any comments or ideas?  For g_file_get_contents(), I can probably use [NSString stringWithContentsOfFile:encoding:error].

 

Here's the current output from Xcode when it gets to linking:

Undefined symbols for architecture armv7:

  "_g_unichar_to_utf8", referenced from:

      _make_string_from_unichar in cljc.o

  "_g_utf8_get_char", referenced from:

      _FN_G__36475 in cljc.o

  "_g_utf8_skip", referenced from:

      _FN_G__36582 in cljc.o

  "_g_utf8_strchr", referenced from:

      _strchr_offset in cljc.o

  "_g_utf8_strlen", referenced from:

      _FN_G__36477 in cljc.o

  "_g_utf8_offset_to_pointer", referenced from:

      _FN_G__36475 in cljc.o

  "_g_strdup", referenced from:

      _FN_G__36582 in cljc.o

  "_g_file_get_contents", referenced from:

      _slurp_file in cljc.o

  "_g_strndup", referenced from:

      _FN_G__36582 in cljc.o

  "_g_strdup_printf", referenced from:

      _FN_G__36452 in cljc.o

ld: symbol(s) not found for architecture armv7

clang: error: linker command failed with exit code 1 (use -v to see invocation)

Mark

[Timothy Baldridge]

> Please don't bypass C! Almost all platforms have some sort of C compiler - it's like javascript for native platforms, instead of the web. With C output, I can probably get it working on iOS

The same is true for LLVM. There's no reason why LLVM can't target iOS.

Timothy

[Adam King]

On Tue, Jul 10, 2012 at 2:09 PM, Timothy Baldridge <tbald...@gmail.com> wrote:

> Please don't bypass C!  Almost all platforms have some sort of C compiler - it's like javascript for native platforms, instead of the web.  With C output, I can probably get it working on iOS

The same is true for LLVM. There's no reason why LLVM can't target iOS.

  Right - but what about Android NDK and Win8RT (mobile and 'Surface')?  I build for native Android almost every day and NDK-r8 is currently at gcc 4.4.3.  After a quick google search, there doesn't appear to be a LLVM compiler for Android (other than "it's possible, but not easy").  And then what about Microsoft?  ClojureCLR on Win8RT might be a possibility - but can LLVM compile to that platform?  And other, less common embedded systems might not have great support for the latest llvm.   

  I also don't know much about LLVM, other than using clang++ for iOS work - so maybe it is all possible with enough time and money..  *shrug*

 I'll keep at getting iOS/AndroidNDK working as that is what interests me.  LLVM does look cool though :)

   Adam

   

Timothy

[Mark Probst]

Hey Adam,

> I've spent a couple of hours on this - and have run into a snag. I've got
> libgc compiling/linking with arm iOS and the Xcode project compiles cljc.c
> fine. The problem is glib. From what I've read, iOS can not use glib as it
> must be statically linked into the app - but the LGPL doesn't allow that (at
> least from what I've read). Currently, there only seems to be about 7
> methods that you're using that need it - 6 of which at utf8 related. So,
> I'm thinking that I'll make a 'prepreamble.m' and use NSString's
> utf8/unichar support for simulating those methods. But before I spend time
> on that, any comments or ideas? For g_file_get_contents(), I can probably
> use [NSString stringWithContentsOfFile:encoding:error].

I see the problem.

My guess is that on iOS we'd use NSString anyway, even if we use glib2
or whatever else for command-line programs. One problem here is that
you cannot allocate NSString's from Boehm, so you'd have to use
finalizers to make sure the NSString's are released when the
corresponding Clojure objects are collected. See
GC_register_finalizer in Boehm's gc.h.

Late on we'll have proper Objective-C bindings, so the glueing will
become much easier.

Mark

[kovas boguta]

Looks like a lot of people are excited about this.

For the specific purpose of cocoa apps, can someone explain how this
is better/different than the other approaches?

There are at least 3 other projects that promise some kind of
c/objective-c interop,
https://github.com/takeoutweight/clojure-scheme,
https://github.com/raph-amiard/clojurescript-lua,
https://github.com/jspahrsummers/cocoa-clojure

So it would be interesting to know what the pros and cons are (again,
for the case making cocoa apps)

[John Szakmeister]

On Mon, Jul 9, 2012 at 7:03 AM, Mark Probst <mark....@gmail.com> wrote:
> Dear Clojurians,
>
> I'm excited to announce ClojureC, an effort to produce a Clojure
> implementation that targets C:
>
> https://github.com/schani/clojurec
>
> My personal goals with this are to be able to write self-contained
> (command-line) Clojure programs that have (essentially) zero start-up time,
> as well as to have a Clojure implementation that plays well with iOS (no
> work on Objective-C bindings has been done so far, but I have plans...).

Nice work! Are you looking to support multiple threads, STM, agents,
etc.? I've been debating on working on a native Clojure
implementation for a while, and that was one of my goals... I'm
interested if it's one of yours.

Thanks!

-John

[Mark Probst]

On Wed, Jul 11, 2012 at 11:57 AM, John Szakmeister <jo...@szakmeister.net> wrote:
> Nice work! Are you looking to support multiple threads, STM, agents,
> etc.? I've been debating on working on a native Clojure
> implementation for a while, and that was one of my goals... I'm
> interested if it's one of yours.

Yes, it's definitely a goal, albeit not an immediate one. Which
doesn't mean that it can't be worked on right away - let me know if
you'd like to help :-)

Mark

[Timothy Baldridge]

>> I've been debating on working on a native Clojure

I've also been debating this for over two years now,y but I keep
coming back to the same problems over and over again:

1) A from-scratch native Clojure either needs to be just a compiler
(as this project is), or you're going to need a decent GC, JIT and
object system
2) The Boehm GC is easy to use, but it's not fast or accurate. The
JVMs can implement generational concurrent compacting GCs (and do
depending on your implementation). These are all very useful features,
and things that Boehm doesn't support. In addition, Boehm isn't
accurate, so memory leaks can still occur.
3) Unless you plan on staying strictly AOT, and igoring the REPL, then
we're going to need a JIT, that's not exactly something you can hack
together in an afternoon
4) Arithmetic must auto-unbox or it will be quite slow. This also
requires a JIT.
5) Clojure is made up of hundreds of single line functions, a JIT
should be able to inline this, one more reason why its needed


All these reasons have brought me to the conclusion that the simplest
way to get all this, is to base a native version of Clojure on PyPy.
No, I didn't say write it in Python and have it run on the Python VM.
PyPy is technically speaking a "language agnostic JIT generator
written in Python".

A good overview of this process is here:
http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html

What you get with a JIT written in PyPy:

1) Your interpreter/JIT is written in a dynamic language with tons of
meta-programming abilities
2) You get at least 3 GCs (reference counting, mark-and-sweep, Boehm)
3) You get a tracing JIT, that for dynamic languages blows method
level JITs away
4) You get STM abilities (as of the most recent work in PyPy)
5) Automatic unboxing, inlining and constant propagation of JIT'ed
code (comes free with a tracing JIT).
6) And you still have great interop with C and with some work C++ (via Reflex)

I don't really want to hi-jack this thread, I just wanted to make
people aware that writing a fast native Clojure takes a bit more work
than simply slapping LLVM and Boehm together. Do that and the JVM will
beat you every single time.

Timothy

[Jules]

You don't really need a JIT for those things, smart compile time analysis can get as good or better results. See Stalin scheme:

http://en.wikipedia.org/wiki/Stalin_(Scheme_implementation) 

[Timothy Baldridge]

> You don't really need a JIT for those things, smart compile time analysis
> can get as good or better results. See Stalin scheme:
>
> http://en.wikipedia.org/wiki/Stalin_(Scheme_implementation)

True if you don't want a repl or eval, then you need a JIT. Notice it
is also a "full program optimizer" which limits interop and removes
your ability to debug the resulting code. From your link: "The
compiler itself runs slowly, and there is little or no support for
debugging or other niceties. "

That's the nice thing about PyPy's JITs. At their core they are just
interpreters. So when something goes wrong, you have total debugging
support.

Timothy

[Mark Probst]

On Wed, Jul 11, 2012 at 3:05 PM, Timothy Baldridge <tbald...@gmail.com> wrote:
>>> I've been debating on working on a native Clojure
>
> I've also been debating this for over two years now,y but I keep
> coming back to the same problems over and over again:
>
> 1) A from-scratch native Clojure either needs to be just a compiler
> (as this project is), or you're going to need a decent GC, JIT and
> object system

You need a decent GC and object system in either case.

> 2) The Boehm GC is easy to use, but it's not fast or accurate. The
> JVMs can implement generational concurrent compacting GCs (and do
> depending on your implementation). These are all very useful features,
> and things that Boehm doesn't support. In addition, Boehm isn't
> accurate, so memory leaks can still occur.

Boehm is just a starting point to get things off the ground. We'll
eventually have a precise, generational GC as well. In particular,
there is one I have a bit of experience with:

http://schani.wordpress.com/tag/sgen/

> 3) Unless you plan on staying strictly AOT, and igoring the REPL, then
> we're going to need a JIT, that's not exactly something you can hack
> together in an afternoon

I don't follow.

> 4) Arithmetic must auto-unbox or it will be quite slow. This also
> requires a JIT.

No. JITs can help there in some ways, with run-time feedback
optimizations, but type declarations and some analysis also go a long
way.

> 5) Clojure is made up of hundreds of single line functions, a JIT
> should be able to inline this, one more reason why its needed

Static compilers can also inline. Again, JITs can do more, but this
is not a show-stopper by a long shot.

> All these reasons have brought me to the conclusion that the simplest
> way to get all this, is to base a native version of Clojure on PyPy.
> No, I didn't say write it in Python and have it run on the Python VM.
> PyPy is technically speaking a "language agnostic JIT generator
> written in Python".

We already have two first class JITting Clojure implementations. This
is a different project.

Mark

[Timothy Baldridge]

>> 3) Unless you plan on staying strictly AOT, and igoring the REPL, then
>> we're going to need a JIT, that's not exactly something you can hack
>> together in an afternoon
>
> I don't follow.

With a AOT compiler like this project, you don't have the ability to
play at a repl. I guess if you don't want that, then that's not a
problem.

> We already have two first class JITting Clojure implementations. This
> is a different project.

We have two JITing versions of Clojure bolted onto a OOP VMs that are
optimized for imperative programming.

I agree that this is a different project. And I hope it succeeds.


Timothy

[Mark Probst]

>>> 3) Unless you plan on staying strictly AOT, and igoring the REPL, then
>>> we're going to need a JIT, that's not exactly something you can hack
>>> together in an afternoon
>>
>> I don't follow.
>
> With a AOT compiler like this project, you don't have the ability to
> play at a repl. I guess if you don't want that, then that's not a
> problem.

There are two REPL options, both of which I consider possibilities for
debugging purposes, but not for production:

* A Clojure interpreter (in Clojure). This would be useful in and of itself.

* Dynamic code loading. This is kinda how ClojureScript does it.

>> We already have two first class JITting Clojure implementations. This
>> is a different project.
>
> We have two JITing versions of Clojure bolted onto a OOP VMs that are
> optimized for imperative programming.
>
> I agree that this is a different project. And I hope it succeeds.

I shall do my best :-)

Mark

[Steve Tickle]

Llvm does target iOS already, plenty of other platforms aren't so lucky tho.
There is a lot of value in having a c target, it's effectively the
most portable assembly language in existence.

SteveT

[Steve Tickle]

I would agree with those not wanting to bypass C, every platform on the planet has a C compiler, only a few are targeted by LLVM.

SteveT

Mark

[Adam King]

Hi Mark:

  Just a quick iOS update.  I grabbed the latest from your git repo and from my initial work at implementing the required glib methods that you're using, I now have it running on both my iPad and iPhone4S.  I don't look at the glib source - I just google the missing function name and implement it based on what the docs describe.   It spits out (to the debug console) the same output as OSX shows when running "./cljc" from the Terminal, which would be: "(4 5)"

  I took a snap of xcode running (with the "(4 5)" ) and dumped it here: http://antimass.org/clojure/cljc_xcode.png  

  As you can see, there are some "Unused variable 'env'" and "Expression result unused" warnings which I'll probably just disable. But the main item I was trying to determine from all this, was that it would be possible to use your clojurec work under iOS, which now looks very doable!  The only change I really had to make to the generated cljc.c file was change main() -> main_ios() which I call from [AppDelegate application:didFinishLaunchingWithOptions].  

  I'll clean it up and commit to my github clojurec repo branch and then I'll do a pull request later this weekend/Monday.  I'll also pop my signed Clojure Contribution Agreement in the mail tomorrow incase that's needed.  Again, thanks for your work on clojurec!  And to David Nolen for ClojureScript - I do most of my play in ClojureScript and it's REPL (mainly playing with WebGL as it ties into my day job. :)   I just want to have the option to compile it out to C and then release in the App Store(s).  Cheers,

  Adam

[Mark Probst]

Adam,

> Just a quick iOS update. I grabbed the latest from your git repo and from
> my initial work at implementing the required glib methods that you're using,
> I now have it running on both my iPad and iPhone4S. I don't look at the
> glib source - I just google the missing function name and implement it based
> on what the docs describe. It spits out (to the debug console) the same
> output as OSX shows when running "./cljc" from the Terminal, which would be:
> "(4 5)"

That's fantastic news!

Reimplementing the glib string functions is not the way to go for iOS,
but I understand why you did it in the first iteration. As I said,
we'll have proper Objective-C bindings which will make those things
much better.

> As you can see, there are some "Unused variable 'env'" and "Expression
> result unused" warnings which I'll probably just disable.

Yes, they're just a result of the compiler being very simple-minded,
and they do no harm.

> I'll clean it up and commit to my github clojurec repo branch and then
> I'll do a pull request later this weekend/Monday. I'll also pop my signed
> Clojure Contribution Agreement in the mail tomorrow incase that's needed.

I'll have to figure out how to handle this, too. Thanks for reminding!

Mark