Life-cycle of a Clojure program under clojure-py?

69 views
Skip to first unread message

John Gabriele

unread,
Mar 17, 2012, 12:57:53 AM3/17/12
to clojure-py-dev
Hi,

I'm curious how a Clojure (err... clojure-py) program gets run by
clojure-py. (Is that terminology correct?)

A couple of observations first though: I see that clojure/core.clj is
written in clojure-py, and that it makes use of the various functions
and classes in the .py files in clojure/lang/ (and that any clojure-py
program I write will make use of those functions implemented in
clojure/core.clj).

Does a given clojure-py program I write get translated to Python and
then handed over to the regular python binary (which would import what
it needs from clojure/lang)?

If there's any bytecode involved, it's provided by the python binary
rather than clojure-py, correct?

How exactly is my clojure-py program getting translated/compiled into
python? What steps are involved?

Finally, just a practical question: how do I run a clojure-py program
from a github checkout? (I see that I can get a repl via `python
clojure.py`, but I want to run a standalone foo.clj program.)

Thanks so much!
---John

Timothy Baldridge

unread,
Mar 17, 2012, 11:38:16 AM3/17/12
to clojure...@googlegroups.com
> Does a given clojure-py program I write get translated to Python and
> then handed over to the regular python binary (which would import what
> it needs from clojure/lang)?

> How exactly is my clojure-py program getting translated/compiled into


> python? What steps are involved?

I'll try to put this in a wiki soon. But till then:

1) the file is opened
2) A Compiler instance from clojure/lang/compiler.py is created
3) lispreader.py is passed the open file. It returns a single form.
This could be a persistentlist, a vector, etc.
4) This form is passed to compilerinstance.compile(form)
5) The compiler walks the form and generates Python bytecodes for the
form and returns a list of byteplay bytecodes (byteplay is in
clojure/util/byteplay.py
6) The bytecodes are passed back into the compiler's execute methods.
The bytecodes are reduced to a binary stream
7) exec() is called on the resulting bytecodes
8) jump to 3 until lispreader.py returns EOF

What you'll find is that Clojure-py functions are exactly like python
functions. For instance:

user=> (defn squared [a] (* a a))
#'user/squared
user=> (dis/dis squared)
0 0 LOAD_CONST 1 (#'clojure.core/*)
3 LOAD_ATTR 0 (deref)
6 CALL_FUNCTION 0
9 LOAD_FAST 0 (a)
12 LOAD_FAST 0 (a)
15 CALL_FUNCTION 2
18 RETURN_VALUE
None
user=>

If you're interested in how the compiler works, I'd jump to
Compiler.compile inside clojure/lang/compiler.py and just start
reading through it all. It's not super complex, it's just a ton of
"if/then/else" logic to handle all the side cases.


>
> Finally, just a practical question: how do I run a clojure-py program
> from a github checkout? (I see that I can get a repl via `python
> clojure.py`, but I want to run a standalone foo.clj program.)

Any argument ending in .clj added after clojure.py is considered a
file to be loaded and run.

e.g.:

python clojure.py tests/core-tests.clj

If you do:

python setup.py installl

then you can simply do:

clojurepy foo.clj

from any directory


I hope this helps, and feel free to e-mail back with any questions.

Timothy

--
“One of the main causes of the fall of the Roman Empire was
that–lacking zero–they had no way to indicate successful termination
of their C programs.”
(Robert Firth)

Daniel Gagnon

unread,
Mar 17, 2012, 12:01:29 PM3/17/12
to clojure...@googlegroups.com

> Finally, just a practical question: how do I run a clojure-py program
> from a github checkout? (I see that I can get a repl via `python
> clojure.py`, but I want to run a standalone foo.clj program.)

Any argument ending in .clj added after clojure.py is considered a
file to be loaded and run.

e.g.:

python clojure.py tests/core-tests.clj

If you do:

python setup.py installl

then you can simply do:

clojurepy foo.clj

from any directory

Would it be possible through pypy to get a clojurepy executable? 

Timothy Baldridge

unread,
Mar 17, 2012, 12:08:18 PM3/17/12
to clojure...@googlegroups.com
>
> Would it be possible through pypy to get a clojurepy executable?

Yes, it technically is, and I've done it. However all code translated
through pypy (notice translated....not run through the pypy
interpreter), must be RPython compatible. This basically means
everything must support static type inference. In the examples folder
there's a rpython example that actually uses pypy to compile
clojure-py code to a binary executable. But the restrictions are
fairly severe. One of these days I should just do a screencast about
how the pypy/rpython/clojure-py stuff would work, and what the
problems are.

TL;DR -- It's possible. It's not a simple task, but it's possible.

Timothy

Daniel Gagnon

unread,
Mar 17, 2012, 12:12:45 PM3/17/12
to clojure...@googlegroups.com
I meant the clojurepy executable itself, not clojurepy code.

Timothy Baldridge

unread,
Mar 17, 2012, 12:16:53 PM3/17/12
to clojure...@googlegroups.com
> I meant the clojurepy executable itself, not clojurepy code.

Oh yeah, sorry, I misunderstood. Currently clojurepy references
whatever the default python install is. We didn't want to default it
to using pypy because that would break on systems where pypy wasn't
installed. What we could do however, is have two versions: clojurepy
and clojurepypy then usrs could fire off whichever they wanted.

Daniel Gagnon

unread,
Mar 17, 2012, 12:25:21 PM3/17/12
to clojure...@googlegroups.com
On Sat, Mar 17, 2012 at 12:16 PM, Timothy Baldridge <tbald...@gmail.com> wrote:
> I meant the clojurepy executable itself, not clojurepy code.

Oh yeah, sorry, I misunderstood. Currently clojurepy references
whatever the default python install is. We didn't want to default it
to using pypy because that would break on systems where pypy wasn't
installed. What we could do however, is have two versions: clojurepy
and clojurepypy then usrs could fire off whichever they wanted.


I seem to remember you had plans to be able to convert clojurepy code to pyc files. Is that correct or I'm just confusing it with the plan to load the core functions quickly? Would pyc files work equally well under Python and Pypy? 

Timothy Baldridge

unread,
Mar 17, 2012, 12:59:07 PM3/17/12
to clojure...@googlegroups.com
> I seem to remember you had plans to be able to convert clojurepy code to pyc
> files. Is that correct or I'm just confusing it with the plan to load the
> core functions quickly? Would pyc files work equally well under Python and
> Pypy?

If we can get pyc files generated from clojure code then yes, they'd
work just as well for pypy as they would for cpython. However,
actually pulling this off is going to be a bit more complex than I
thought. pyc files use the marshal module and this module can't
serialize user classes. So either we have to use something different
(like pickle) or we try to generate code differently. For example:

(:foo mymap)

compiles to:

LOAD_CONST :foo
LOAD_FAST mymap
CALL_FUNCTION 1

The issue is that marshal can't serialize :foo, and pickle can't
serialize functions at all. Now I could do this:

LOAD_CONST keyword
LOAD_CONST "foo"
CALL_FUNCTION 1
LOAD_FAST mymap
CALL_FUNCTION 1

but then we have the insane overhead of interning a keyword every time
we simply want get something from a map.

So all that to say, I'm still figuring out the best way to do this.

John Gabriele

unread,
Mar 17, 2012, 1:00:04 PM3/17/12
to clojure-py-dev
On Mar 17, 11:38 am, Timothy Baldridge <tbaldri...@gmail.com> wrote:
>
> 1) the file is opened
> 2) A Compiler instance from clojure/lang/compiler.py is created
> 3) lispreader.py is passed the open file. It returns a single form.
> This could be a persistentlist, a vector, etc.

So, this form would be a list of all the forms in my program? For
example, for a trivial program:

(defn foo [x]
(* x x))

(def a (foo 3))

(println a)

the first item in the list would be `(defn ...)`, the 2nd `(def
a ...)`, and the 3rd would be `(println a)`, correct?

> 4) This form is passed to compilerinstance.compile(form)
> 5) The compiler walks the form and generates Python bytecodes for the
> form and returns a list of byteplay bytecodes (byteplay is in
> clojure/util/byteplay.py

Oh, I see. It can compile to bytecode all by itself, without the help
of the python binary. :) Somehow I got the impression that clojure-py
was translating to actual Python code (not *bytecode*).

> 6) The bytecodes are passed back into the compiler's execute methods.
> The bytecodes are reduced to a binary stream
> 7) exec() is called on the resulting bytecodes

Is this exec() from the python binary?

Ignoring PyPy for the moment, is step 7 the only step where the python
binary is required?

> 8) jump to 3 until lispreader.py returns EOF
>
> What you'll find is that Clojure-py functions are exactly like python
> functions. For instance:
>
> user=> (defn squared [a] (* a a))
> #'user/squared
> user=> (dis/dis squared)
>   0           0 LOAD_CONST               1 (#'clojure.core/*)
>               3 LOAD_ATTR                0 (deref)
>               6 CALL_FUNCTION            0
>               9 LOAD_FAST                0 (a)
>              12 LOAD_FAST                0 (a)
>              15 CALL_FUNCTION            2
>              18 RETURN_VALUE
> None
> user=>

Is that an example of Python bytecode?

(Hm. I tried `(doc dis/dis)`, but got a Traceback.)

> If you're interested in how the compiler works, I'd jump to
> Compiler.compile inside clojure/lang/compiler.py and just start
> reading through it all. It's not super complex, it's just a ton of
> "if/then/else" logic to handle all the side cases.

Thanks. I'm curious, and will take a look.

>
>
> > Finally, just a practical question: how do I run a clojure-py program
> > from a github checkout? (I see that I can get a repl via `python
> > clojure.py`, but I want to run a standalone foo.clj program.)
>
> Any argument ending in .clj added after clojure.py is considered a
> file to be loaded and run.
>
> e.g.:
>
> python clojure.py tests/core-tests.clj

Just tried this with my above trivial example program:

python clojure.py foo.clj

(file is just sitting there in the top-level clojure-py dir), but it
failed, yielding:

~~~
john@localhost:~/temp/cl-py/clojure-py$ python ./clojure.py foo.clj
clojure-py 0.1.0
Compiling (def foo (clojure.core/fn ([x], (* x x))))
Compiling (defn foo [x] (* x x))
(defn foo [x] (* x x)) foo.clj
Traceback (most recent call last):
File "./clojure.py", line 4, in <module>
clojure.main.main()
File "/home/john/temp/cl-py/clojure-py/clojure/main.py", line 169,
in main
requireClj(x)
File "/home/john/temp/cl-py/clojure-py/clojure/main.py", line 86, in
requireClj
res = comp.compile(s)
File "/home/john/temp/cl-py/clojure-py/clojure/lang/compiler.py",
line 1103, in compile
c.extend(self.compileForm(itm))
File "/home/john/temp/cl-py/clojure-py/clojure/lang/compiler.py",
line 1003, in compileForm
form, ret = macroexpand(form, self)
File "/home/john/temp/cl-py/clojure-py/clojure/lang/compiler.py",
line 909, in macroexpand
mresult = comp.compile(mresult)
File "/home/john/temp/cl-py/clojure-py/clojure/lang/compiler.py",
line 1103, in compile
c.extend(self.compileForm(itm))
File "/home/john/temp/cl-py/clojure-py/clojure/lang/compiler.py",
line 1002, in compileForm
return builtins[form.first()](self, form)
File "/home/john/temp/cl-py/clojure-py/clojure/lang/compiler.py",
line 103, in compileDef
v = internVar(comp.getNS(), sym)
File "/home/john/temp/cl-py/clojure-py/clojure/lang/var.py", line
180, in intern
ns = findOrCreate(symbol(ns))
File "/home/john/temp/cl-py/clojure-py/clojure/lang/symbol.py", line
61, in symbol
idx = a.rfind("/")
AttributeError: 'NoneType' object has no attribute 'rfind'
~~~

This is with Python 2.6.5 on a slightly aging Ubuntu. :)

Thanks!
---John

Timothy Baldridge

unread,
Mar 17, 2012, 1:15:23 PM3/17/12
to clojure...@googlegroups.com
>
>    idx = a.rfind("/")
> AttributeError: 'NoneType' object has no attribute 'rfind'
> ~~~

put this at the top of foo.clj

(ns foo)

That's a really bad error message. But it's complaining that you
haven't set the namespace for the file yet.

> Oh, I see. It can compile to bytecode all by itself, without the help
> of the python binary. :) Somehow I got the impression that clojure-py
> was translating to actual Python code (not *bytecode*).

I started with that idea, but for instance, clojure supports
multi-statement lambdas and Python does not. So the compiler has to
drop to the bytecode level to support all the features it needs.

>
> Is this exec() from the python binary?

Yes


> Ignoring PyPy for the moment, is step 7 the only step where the python
> binary is required?

I'm not exactly sure what you're asking here...all of clojure-py is
python bytecode. From end-to-end it all runs on the python binary.

>> user=> (dis/dis squared)
>>   0           0 LOAD_CONST               1 (#'clojure.core/*)
>>               3 LOAD_ATTR                0 (deref)
>>               6 CALL_FUNCTION            0
>>               9 LOAD_FAST                0 (a)
>>              12 LOAD_FAST                0 (a)
>>              15 CALL_FUNCTION            2
>>              18 RETURN_VALUE
>> None
>> user=>

>
> Is that an example of Python bytecode?

Yes, we're invoking dis from the python dis function.

John Gabriele

unread,
Mar 18, 2012, 1:11:18 AM3/18/12
to clojure-py-dev
Timothy,

Thanks for all the info. I've written a short "clojure-py overview"
that I think would be make a good clojure-py wiki page. Here's a
draft: http://www.unexpected-vortices.com/temp/clojure-py-overview.html

Feedback would be appreciated.

Thanks,
---John

John Gabriele

unread,
Mar 18, 2012, 1:16:13 AM3/18/12
to clojure-py-dev
>
> Thanks for all the info. I've written a short "clojure-py overview"
> that I think would be make a good clojure-py wiki page. Here's a

s/would/might/ :)

---J

Timothy Baldridge

unread,
Mar 19, 2012, 7:02:00 AM3/19/12
to clojure...@googlegroups.com
Looks pretty good. And yes, currently there are no generated pyc files
for core.clj. Hopefully that will change .

John Gabriele

unread,
Mar 19, 2012, 2:18:12 PM3/19/12
to clojure...@googlegroups.com
On Mon, Mar 19, 2012 at 7:02 AM, Timothy Baldridge <tbald...@gmail.com> wrote:
> Looks pretty good. And yes, currently there are no generated pyc files
> for core.clj. Hopefully that will change .

Ok, thanks for the info. Added the wiki page.

---J

Reply all
Reply to author
Forward
0 new messages