A plan for the future

[Timothy Baldridge]

Well, Clojure-Py is almost a year old! And before I continue, I'd like to say a big thanks to everyone for the work that's been done on the project. I know I haven't been keeping up on this list as of late, and I'd like that to change.

In the past year, ClojureScript has made some major, major developments. To name two, the analyzer and the compiler have been almost completely decomplected. In addition, great strides are being made lately to make CLJs not require the JVM platform. The combination of these two means than I think the time may be here to do a re-write of Clojure-Py on CLJS.

But! Before we start down that path, I think it's quite important to discuss the goals of Clojure-Py, and also in what areas it currently out-performs Clojure on the JVM:

1) Simple fly-weight web services.

2) Numpy programs

3) GUI apps (QT/wxWidgets vs Swing)

4) It's not the JVM (people who hate the bloat of the JVM)

These are all things I want to keep. Therefore it is not my goal to keep Clojure-Py2 as a JVM compiled Python language (yuck). We will have to start there, but that's not the end goal. To get to the goal of a fully Python hosted Clojure-Py2, we'll need the following:

1) a "python.clj" file. See closure.clj in the CLJS repo. This file co-ordinates between the analyzer and the compiler. Its primary goal is to provide library resolution. If someone says "require foo.bar". Is that a python or a cljs file? This is the job of closure.clj. My suggestion would be to write our own version of this file that from the JVM shells out to python, imports the lib and then does some introspection on the library, pumping the data back as JSON.

2) The compiler.clj needs to be re-written. However, the analyzer now does most of the heavy lifting. So the compiler.clj file is really more like a bytecode writer. This is the project I'm going to be starting on as of today.

3) Our new compiler.clj will not be able to produce .pyc files. Or more correctly it won't be able to properly write .pyc files to disk. This is because .pycs are basically marshaled Python data-structures, and the marshall format can change with each python version. Instead, the plan is to output a .pyc file as JSON. We will then create a .py file that takes a .json and outputs a .pyc. 

4) At this point we'll have Clojure-Py2....on the JVM. So the plan is to take all the above files (analyzer, python.clj, compiler.clj, JSON->pyc writer) and compile them to .pyc files via Clojure-Py2. Now we have a boot-strapped environment and we can toss the JVM.


So, as I mentioned, I plan on working on #2 (above) as of right now. I could use lots of help on the other steps, however. If someone is interested, let me know on this list, and we'll get started. 

Now...I'm also here to announce something that I should have done awhile ago, but didn't. We need a better CA process. I really, really don't want to run into the situation in the future where I want to do something with Clojure-Py (like make it part of the main Clojure github repo), and then I find out that I have to track down everyone who committed to this project. Therefore, Clojure-Py2 is going to require that committers sign some sort of paperwork. If you're happy with a Clojure CA (or have one already) I'll accept that. If not...we'll have to figure something out. 

As far as Clojure-Py1 goes. I really don't want to see that die. If someone wants to maintain it, please let me know. I think Clojure-Py on CLJs is the future, but that's no reason to kill existing code. 

Why is it the future? Well the answer is simple, because we'd get 90% of bug fixes free. For instance, think of reducers. Someone ported those to CLJs. If we were running off of CLJS we suddenly would have reducers as well. Instead we have to wait to see them ported. Overall we'll see much faster development with a Clojure-Py based off of CLJS.

Thanks,

Timothy Baldridge (halgari)

[Konrad Hinsen]

--On 10 décembre 2012 06:42:19 -0700 Timothy Baldridge

<tbald...@gmail.com> wrote:

> These are all things I want to keep. Therefore it is not my goal to keep
> Clojure-Py2 as a JVM compiled Python language (yuck). We will have to
> start there, but that's not the end goal. To get to the goal of a fully
> Python hosted Clojure-Py2, we'll need the following:

...

I haven't ever looked at ClojureScript, so maybe I am just showing my
ignorance, but I don't see any reference to a run-time library in your
list. So where do things like a Python-compatible implementation of the
Clojure data structures come from? From the current ClojurePy code base?

> project. Therefore, Clojure-Py2 is going to require that committers sign
> some sort of paperwork. If you're happy with a Clojure CA (or have one
> already) I'll accept that. If not...we'll have to figure something out. 

The Clojure CA looks like the best option, since that makes it possible to
contribute code that will be used all over the Clojure universe. But maybe
I am just biased because I already signed one.

Konrad.

[Timothy Baldridge]

Good question about the runtime. 

The CLJS compiler is actually quite a bit different than the CLJ compiler. The CLJS compiler starts you off with the following:

defn, defprotocol, deftype, basic math (+, -, /, * etc.), and some sort of interop system (in CLJS this is simply a way to inject JS strings into the code). 

This means that yes...in CLJS all the data structures are written in ClojureScript. For instance, the hashmap implementation is here: https://github.com/clojure/clojurescript/blob/master/src/cljs/cljs/core.cljs#L4673


So the runtime is actually written in ClojureScript. That's the beauty of CLJS, if you can explain defn, defprotocol, etc. on your planform, you get everything else for "free". 

We will have to provide some sort of interop in addition to this. For instance, hashmaps should implement python __iter__ methods, but that should be easy to layer on top of the existing code. 

Timothy 

--
“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[John Gabriele]

On Mon, Dec 10, 2012 at 8:42 AM, Timothy Baldridge <tbald...@gmail.com> wrote:
>
> 3) Our new compiler.clj will not be able to produce .pyc files. Or more
> correctly it won't be able to properly write .pyc files to disk. This is
> because .pycs are basically marshaled Python data-structures, and the
> marshall format can change with each python version. Instead, the plan is to
> output a .pyc file as JSON. We will then create a .py file that takes a
> .json and outputs a .pyc.

Just curious: is this the sort of thing which [edn] might be useful for?

[edn]: https://github.com/edn-format/edn

---John

[Timothy Baldridge]

That would be awesome, but we'd need a EDN reader for python. It looks like there are a few: https://github.com/edn-format/edn/wiki/Implementations 

It'd be nice if one of those supported reader literals (for dealing with binary blobs), but I suppose that's not completely necessary. 

Timothy

[Antony Lee]

Hi all,

Here are my thoughts:

1/ I don't see why we need yet another library resolution mechanism when the PEP302 import hook works well.  As it is right now, the priority is 1/ Python package, 2/ Python module, 3/ clj module.

2/ I have slowly come to the idea that compiling to Python bytecode may not be the best idea after all.  Helpers such as byteplay make this reasonably easy; however supporting many different versions of Python is awkward due to variation in the bytecode set between versions (it took me some time to realize, for example, that MAKE_FUNCTION pops an extra argument since function objects gained a __qualname__).  Timothy has started a branch (treadle) that targets a custom-made AST structure instead, and emits bytecode from there, which may be a solution (if the AST compiler follows the changes in bytecode).  However, perhaps an even simpler solution would be to compile to Python's AST itself and let builtins.compile() do the hard work.  Of course Python's AST changes too, but the subset common e.g. to Python2 and Python3 is significant (it is not that hard to write code that works for Python2 and Python3 without any modifications).  Also, the fact that the compiler for the "try" special form, for closures (!) or for (letfn) is still incomplete (see the issues I recently posted on the repo) shows that writing bytecode is not THAT trivial.

It is true that some clojure constructs do not map well to Python AST, e.g. lambda functions, but there is a way around that: everytime we need an expression where Python only provides a statement, create an intermediate variable and assign to it.  For example,

(let [x 1] ((fn [y] (try (foo) (except E e (bar)))) x))

would compile into (the AST of)

x# = 1 (where #=unique identifier)
def _#(y):
  try: _## = foo()
  except E as e: _## = bar()
  return _##
_#(x)

On a more technical side, each special-form compiler would now take a new argument, the variable name to which the result of the form should be assigned.

Also, a quick look at Python-dev shows me that there is some work currently done on AST optimization passes as well.  Compiling to AST rather than bytecode would allow us to take advantage of these.

On an unrelated note, I think that saying "we can just take data structures implemented in clojurescript and run these on the top of Python" is a slight oversimplification.  Look, e.g. at the implementation of PersistentHashMap.  I haven't read all the details but it just seems like a direct translation of the Java one (e.g. with the has-null? field, which only makes sense in a world where NULL is a special object), just like the current persistenthashmap.py is.  I have sent an email to this mailing list some time ago observing that is implementation performs extremely poorly, every so often, when run on CPython, probably due to hash collision (has anyone been able to reproduce this behaviour?).  Instead, a naïve full-copy-on-write implementation showed significantly improved startup time!...

Antony

2012/12/10 Timothy Baldridge <tbald...@gmail.com>

[Timothy Baldridge]

Ast compilation...good point. Originally I had tossed this out, but I think you're right, we'll save a ton of time by doing what ClojureScript does, and abusing locals/closures to get the results we're looking for. 

And this also opens a super, super awesome possibility. One of the problems with Clojure-Py2 will be testing. Before we completely boot-strap the system it'd be really nice to have some sanity checks. Well, if we use the AST we can. The cycle will look like this:

Clojure form -> analyzer -> AST -> compiler -> Python AST -> Jython -> execute -> result. 

Once we're happy with the result, we simply dump the Python AST out as a json object, then write a python script that reads JSON and spits out a CPython AST. The pipeline is then:

Clojure form -> analyzer -> AST -> compiler -> Python AST -> JSON -> Python Script -> Python AST -> CPython -> Execute. 

Since python can write out ASTs as .pyc files, we should never have to touch marshal, bycodes, or anything. I like it!

And I agree, I think that the ClojureScript structures will work out-of-the box, but they may not be fast. To get there, we may actually need to do some work, but we'll see. 

Personally, this has be psyched. There's no reason why we can't run "lein test" and have it run the entire clojure-py suite.

Timothy

[Adam Feuer]

On Mon, Dec 10, 2012 at 12:05 PM, Timothy Baldridge <tbald...@gmail.com> wrote:

Personally, this has be psyched. There's no reason why we can't run "lein test" and have it run the entire clojure-py suite.

This has me psyched too. I like the idea of being able to reuse a lot of what CLJs has implemented.

-adam 

--
Adam Feuer <ad...@pobox.com>

[Adam Feuer]

On Mon, Dec 10, 2012 at 5:42 AM, Timothy Baldridge <tbald...@gmail.com> wrote:

Now...I'm also here to announce something that I should have done awhile ago, but didn't. We need a better CA process.

Timothy,

By CA, do you mean Contributor Agreement?

 

I really, really don't want to run into the situation in the future where I want to do something with Clojure-Py (like make it part of the main Clojure github repo), and then I find out that I have to track down everyone who committed to this project. Therefore, Clojure-Py2 is going to require that committers sign some sort of paperwork. If you're happy with a Clojure CA (or have one already) I'll accept that.

Here's the Clojure Contributor Agreement:

http://clojure.org/file/view/ca.pdf

I'm fine with that and would sign one for Clojure-Py

cheers

adam

--
Adam Feuer <ad...@pobox.com>

[Dax Fohl]

Excuse my inexperience, but I've been wondering for a while, would there be any value in creating a clojure-py interpreter in RPython and letting its tracing JIT do the hard work?  Would that kind of implementation likely end up faster or slower, easier or harder than the current proposal (and why)?

(I'm talking about this)

http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html

http://morepypy.blogspot.com/2011/04/tutorial-part-2-adding-jit.html

[Antony Lee]

Hi,
As far as I can tell this approach won't be useful for us, because clojure-py is not (in its current form) a clojure interpreter written in python.  Rather, it is a clojure compiler that happens to be written in python, and more importantly emits python (byte)code (rather than, say, java bytecode), that can then be run on a python VM.
Best,
Antony

2012/12/10 Dax Fohl <dax....@gmail.com>

[Konrad Hinsen]

--On 10 décembre 2012 11:28:41 -0800 Antony Lee <anton...@berkeley.edu>
wrote:

> 2/ I have slowly come to the idea that compiling to Python bytecode may
> not be the best idea after all.  Helpers such as byteplay make this

Since we are at the "crazy ideas are allowed" stage, let me propose mine:

How about compiling to LLVM and use py-llvm to run the code? We'd then have
a native-code system that uses the Python API to interface with the Python
ecosystem. A bit like "Unladen Swallow" wanted to with Python itself, and a
bit like Numba does for a subset of Python.

Konrad.

[Timothy Baldridge]

Well I've spent a few days playing with the Python ASTs and they are basically an utter mess. The problems seem to come from the fact that Python ASTs are actually parser trees. For instance, in Jython the AST nodes are wrapped ATLR nodes, and on CPython they are wrappers around the c parser nodes. I was able to generate an AST on almost every platform, but was only able to get it to compile on a few. 

For instance, Jython has a major bug, where doing this;

Module(body=[foo]) results in Module(body=[None])

Run the exact same code on Python3 and it works as expected. 

Several AST trees I wrote, won't compile in Python 3 because they don't have a lineno set for every node...even though they do, and ast.dump(node, True, True) shows that every node has a lineno. Run the exact same code on Python 2.7 and it works just fine. 

Also, the more I looked into it, the Python ASTs are going to be horribly ineficient. Python makes distinctions between statements and expressions, having to deal with them means we would almost have to drop down to a single assignment system and completely ignore the Python stack. That means creating tons and tons of temp variables. This is forced on us since in Python there is no way to put a statement inside an expression. defs are statements...bam! problem. 

When talking to the PyPy devs about this idea originally (about a year ago) their response was: "I used to think translating to a different language's AST was a good idea...i've since discovered that it doesn't work out well".

So, sadly I'm back at the drawing board here. I'm probably going to spend the rest of the week prototyping a python bytecode assembler written in core.logic. It's an idea I've had in the back of my head for awhile, and I'll probably give it a few days and see how clean the implementation feels. The basic idea is that many many aspects of treadle or byteplay can be simplified by using a constraint solver like core.logic. In this style, instead of saying "walk every bytecode and get me the stack size", you can declaratively describe what a bytecode does, and then all the parameters of a python code object are resolved by the logic engine. 

---

Now, on to the other questions:

PyPy - Yes it's possible, Clojure-Py is a prerequisite to this. If you look at my github project: http://github.com/halgari/cljvm This is an implementation of a lisp on PyPy. The problem is, it's not too fast. Making pypy interpreters fast takes time, lots of time. And I'd rather write such an interpreter in RClojure (whatever that is) instead of RPython. So my idea is, we get Clojure compiling to python bytecode, then write a clojure interpreter in clojure using PyPy to generate the JIT. Or at least, that's what I'd do if I had the time ;-)

LLVM - Possible. But also a ton of work. I'm not sure what the answer is here. I've done some work on this in the past, and have a super basic sexpr compiler using LLVM, JNA and core.logic (for type inference). But the problem is, LLVM isn't going to be that fast. Java and PyPy will beat the pants off of it because they both optimize over the runtime of the program. To compare, Mono uses LLVM and even the best C# programs are often 1/2 to 1/4 the speed of a similar Java program. So all LLVM will get you is two things: a native binary, and reduced memory usage. The latter we can get with Python. I'm not sure about the former yet. 

[Antony Lee]

2012/12/12 Timothy Baldridge <tbald...@gmail.com>

Well I've spent a few days playing with the Python ASTs and they are basically an utter mess. The problems seem to come from the fact that Python ASTs are actually parser trees. For instance, in Jython the AST nodes are wrapped ATLR nodes, and on CPython they are wrappers around the c parser nodes. I was able to generate an AST on almost every platform, but was only able to get it to compile on a few. 

For instance, Jython has a major bug, where doing this;

Module(body=[foo]) results in Module(body=[None])

What is the exact bug? Obviously  ast.Module(body=[foo]) will fail because foo is not defined, but something like

>>> mod = ast.Module(body=[ast.Assign([ast.Name("x", ast.Store())], ast.Num(1))])
>>> ast.fix_missing_locations(mod)
Module
>>> exec(compile(mod, "", "exec"))
>>> x
1

works both on Jython 2.5.3 and CPython 2.7.3/3.3.0 for me.

Run the exact same code on Python3 and it works as expected. 

Several AST trees I wrote, won't compile in Python 3 because they don't have a lineno set for every node...even though they do, and ast.dump(node, True, True) shows that every node has a lineno. Run the exact same code on Python 2.7 and it works just fine. 

Again, can you give me an example?  ast.fix_missing_locations wasn't enough?

Also, the more I looked into it, the Python ASTs are going to be horribly ineficient. Python makes distinctions between statements and expressions, having to deal with them means we would almost have to drop down to a single assignment system and completely ignore the Python stack. That means creating tons and tons of temp variables. This is forced on us since in Python there is no way to put a statement inside an expression. defs are statements...bam! problem. 

Yes, I mentioned in my previous email that my approach would be indeed to use a single assignment system and, as you say, create tons of temporaries.  My opinion on that is that I have little interest in generating the most efficient code possible, but correct code instead (which is I think easier by compiling to AST rather than to bytecode), and that it is the job of the Python interpreter (whether CPython, PyPy or anything else) to optimize that away.  And actually by generating an AST we can probably take advantage of more optimizers than if we were generating bytecode.

[John Gabriele]

On Wed, Dec 12, 2012 at 10:31 AM, Timothy Baldridge
<tbald...@gmail.com> wrote:
>
> So, sadly I'm back at the drawing board here.

Probably a bit off-the-wall, but:

* What are your thoughts on creating a C-implementation of Clojure
and embedding Python into it? (Possibly making use of [clojurec]?)

* What about a Clojure implementation as a Python extension?

[clojurec]: https://github.com/schani/clojurec

For reference: http://en.wikipedia.org/wiki/Python_%28programming_language%29#Embedding_and_extending_Python

---John

[Timothy Baldridge]

Here's what I'm hitting (ast copy and pasted from the output of the Clojure-Py2 compiler). 

Python 3.3.0 (default, Dec  5 2012, 11:46:31) 

[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> from python import *

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

ImportError: No module named 'python'

>>> from ast import *

>>> z = Module(body=[Assign(targets=[Name(id='tmp_1304', ctx=Store(), lineno=0, col_offset=0)], value=BinOp(left=Num(n=1, lineno=0, col_offset=0), op=Add(), right=Num(n=2, lineno=0, col_offset=0), lineno=0, col_offset=0), lineno=0, col_offset=0), Expr(value=Call(func=Name(id='print', ctx=Store(), lineno=0, col_offset=0), args=[Name(id='tmp_1304', ctx=Load(), lineno=0, col_offset=0)], keywords=[], starargs=[], kwargs=[], lineno=0, col_offset=0), lineno=0, col_offset=0)])

>>> compile(z, "<string>", "exec")

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

TypeError: required field "lineno" missing from expr

>>> fix_missing_locations(z)

<_ast.Module object at 0x102259f50>

>>> r = fix_missing_locations(z)

>>> compile(r, "<string>", "exec")

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

TypeError: required field "lineno" missing from expr

>>> 

Not sure what's happening here. 

Timothy

[Timothy Baldridge]

And I agree on the speed thing. All the extra tmps will be optimized away via PyPy, so if you want speed there's stil that option.

Timothy

[Antony Lee]

This works:

z = Module([Assign(targets=[Name(id="tmp", ctx=Store())], value=BinOp(Num(1), Add(), Num(2))), Expr(Call(Name("print", Load()), [Name("tmp", Load())], [], None, None))])

"print"'s context is Load(), not Store(), and starargs and kwargs default to None.

2012/12/12 Timothy Baldridge <tbald...@gmail.com>

[Timothy Baldridge]

Yep, that did it for me. I found the bug with the Load vs. Store, but that didn't fix it, so it must have been the [] vs None. Well, I'll keep working on this tonight and try to get my changes uploaded as soon as I get the code cleaned up a bit. 

Timothy

[John Gabriele]

On Wed, Dec 12, 2012 at 12:02 PM, John Gabriele <jmg...@gmail.com> wrote:
>
> * What about a Clojure implementation as a Python extension?
>

Was thinking of [Cython](http://cython.org/) there.

[John Gabriele]

On Wed, Dec 12, 2012 at 10:31 AM, Timothy Baldridge
<tbald...@gmail.com> wrote:
>
> LLVM - Possible. But also a ton of work.

Can anyone provide a rough overview of how Python, LLVM, and
clojure-py-2 would all fit together?

---John

[Timothy Baldridge]

We should clarify something first. Right now, there are no plans for doing any integration with clojure-py2 and LLVM. There are a few reasons why:

1) no library support. Compared to Python or Java, C just doesn't have that many libraries
2) no defined object system or GC
3) no JIT as of yet. PyPy and the JVM have extremely advanced JITs. Comparatively, even the CLR struggles to out perform either. 
4) Clojure-py has no support for typing. I'd like to keep it that way. In order to properly target LLVM we'd have to support some sort of type inference. That's part of the reason why the clojure-py compiler is about 1/10th the size of the Clojure JVM compiler. 

So yeah, it's technically possible, but I think a well designed, self-hosting Clojure-Py is going to get us 90% of what we'd get with LLVM. 

Timothy

[John Gabriele]

On Mon, Dec 10, 2012 at 8:42 AM, Timothy Baldridge <tbald...@gmail.com> wrote:
>
> But! Before we start down that path, I think it's quite important to discuss
> the goals of Clojure-Py, and also in what areas it currently out-performs
> Clojure on the JVM:
>
> 1) Simple fly-weight web services.

Hm. "fly-weight" makes me think of Lua. My understanding is that the
Lua VM has a JIT.

Further, I remember reading about work this past summer on a Lua
backend for ClojureScript.

Then a quick search turned up http://pypi.python.org/pypi/lupa .

---John

[Konrad Hinsen]

--On 13 décembre 2012 16:50:41 -0700 Timothy Baldridge <tbald...@gmail.com> wrote:

> We should clarify something first. Right now, there are no plans for
> doing any integration with clojure-py2 and LLVM. There are a few reasons
> why:
>
> 1) no library support. Compared to Python or Java, C just doesn't have
> that many libraries
> 2) no defined object system or GC

That's exactly why my idea was to use the Python platform as a runtime
system, providing an object system, a library, and a GC.

> 3) no JIT as of yet. PyPy and the JVM have extremely advanced JITs.
> Comparatively, even the CLR struggles to out perform either. 

The problem with PyPy is that is doesn't fit into CPython's ecosystem,
which depends a lot on the large number of extension modules interfacing to
C libraries. For me, PyPy is no more useful than the JVM at the moment.

> 4) Clojure-py has no support for typing. I'd like to keep it that way. In
> order to properly target LLVM we'd have to support some sort of type
> inference. That's part of the reason why the clojure-py compiler is about
> 1/10th the size of the Clojure JVM compiler. 

That's indeed a problem if efficiency is a main goal. My idea was to use
LLVM initially only as a more portable alternative to Python bytecode,
keeping all data in dynamically typed Python objects. Something more
optimizing could follow later on.

Konrad.

[Timothy Baldridge]

ooh...I see. That's an interesting thought. Let me research this one for a few days. LLVM does have some officially supported Python bindings, so that part will be really nice to have. 

Now that would mean that we couldn't run clojure-py on pypy, But I'm not sure it would be as needed if we dropped to LLVM. https://github.com/numba/numba is an example of what we're talking about here. 

We'd need to do some research on how to implement polymorphic functions, but perhaps we could dig into how polymorphism works in Python right now, and leverage that. 

Does anyone know how many changes there were to the C API between 2.7 and 3? My goal is that we need to support both 2.7 and 3.2. I'm going to suggest that we cut 2.6 support. There are just too many changes between 2.6 and 2.7 to make it worth the hassle. 

Timothy

[Konrad Hinsen]

--On 15 décembre 2012 08:38:46 -0700 Timothy Baldridge

<tbald...@gmail.com> wrote:

> Now that would mean that we couldn't run clojure-py on pypy, But I'm not
> sure it would be as needed if we dropped to
> LLVM. https://github.com/numba/numba is an example of what we're talking
> about here. 

That was the kind of things I had in mind for future optimizations.

> Does anyone know how many changes there were to the C API between 2.7 and
> 3? My goal is that we need to support both 2.7 and 3.2. I'm going to
> suggest that we cut 2.6 support. There are just too many changes between
> 2.6 and 2.7 to make it worth the hassle. 

Python 3 mostly drops obsolete features of the Python 2 C API. Supporting
both in newly written code should not be too hard. Cython supports both, so
it could serve as a source for inspiration.

Konrad.