> How exactly is my clojure-py program getting translated/compiled into
> python? What steps are involved?
I'll try to put this in a wiki soon. But till then:
1) the file is opened
2) A Compiler instance from clojure/lang/compiler.py is created
3) lispreader.py is passed the open file. It returns a single form.
This could be a persistentlist, a vector, etc.
4) This form is passed to compilerinstance.compile(form)
5) The compiler walks the form and generates Python bytecodes for the
form and returns a list of byteplay bytecodes (byteplay is in
clojure/util/byteplay.py
6) The bytecodes are passed back into the compiler's execute methods.
The bytecodes are reduced to a binary stream
7) exec() is called on the resulting bytecodes
8) jump to 3 until lispreader.py returns EOF
What you'll find is that Clojure-py functions are exactly like python
functions. For instance:
user=> (defn squared [a] (* a a))
#'user/squared
user=> (dis/dis squared)
0 0 LOAD_CONST 1 (#'clojure.core/*)
3 LOAD_ATTR 0 (deref)
6 CALL_FUNCTION 0
9 LOAD_FAST 0 (a)
12 LOAD_FAST 0 (a)
15 CALL_FUNCTION 2
18 RETURN_VALUE
None
user=>
If you're interested in how the compiler works, I'd jump to
Compiler.compile inside clojure/lang/compiler.py and just start
reading through it all. It's not super complex, it's just a ton of
"if/then/else" logic to handle all the side cases.
>
> Finally, just a practical question: how do I run a clojure-py program
> from a github checkout? (I see that I can get a repl via `python
> clojure.py`, but I want to run a standalone foo.clj program.)
Any argument ending in .clj added after clojure.py is considered a
file to be loaded and run.
e.g.:
python clojure.py tests/core-tests.clj
If you do:
python setup.py installl
then you can simply do:
clojurepy foo.clj
from any directory
I hope this helps, and feel free to e-mail back with any questions.
Timothy
--
“One of the main causes of the fall of the Roman Empire was
that–lacking zero–they had no way to indicate successful termination
of their C programs.”
(Robert Firth)
Any argument ending in .clj added after clojure.py is considered a
> Finally, just a practical question: how do I run a clojure-py program
> from a github checkout? (I see that I can get a repl via `python
> clojure.py`, but I want to run a standalone foo.clj program.)
file to be loaded and run.
e.g.:
python clojure.py tests/core-tests.clj
If you do:
python setup.py installl
then you can simply do:
clojurepy foo.clj
from any directory
Yes, it technically is, and I've done it. However all code translated
through pypy (notice translated....not run through the pypy
interpreter), must be RPython compatible. This basically means
everything must support static type inference. In the examples folder
there's a rpython example that actually uses pypy to compile
clojure-py code to a binary executable. But the restrictions are
fairly severe. One of these days I should just do a screencast about
how the pypy/rpython/clojure-py stuff would work, and what the
problems are.
TL;DR -- It's possible. It's not a simple task, but it's possible.
Timothy
Oh yeah, sorry, I misunderstood. Currently clojurepy references
whatever the default python install is. We didn't want to default it
to using pypy because that would break on systems where pypy wasn't
installed. What we could do however, is have two versions: clojurepy
and clojurepypy then usrs could fire off whichever they wanted.
> I meant the clojurepy executable itself, not clojurepy code.Oh yeah, sorry, I misunderstood. Currently clojurepy references
whatever the default python install is. We didn't want to default it
to using pypy because that would break on systems where pypy wasn't
installed. What we could do however, is have two versions: clojurepy
and clojurepypy then usrs could fire off whichever they wanted.
If we can get pyc files generated from clojure code then yes, they'd
work just as well for pypy as they would for cpython. However,
actually pulling this off is going to be a bit more complex than I
thought. pyc files use the marshal module and this module can't
serialize user classes. So either we have to use something different
(like pickle) or we try to generate code differently. For example:
(:foo mymap)
compiles to:
LOAD_CONST :foo
LOAD_FAST mymap
CALL_FUNCTION 1
The issue is that marshal can't serialize :foo, and pickle can't
serialize functions at all. Now I could do this:
LOAD_CONST keyword
LOAD_CONST "foo"
CALL_FUNCTION 1
LOAD_FAST mymap
CALL_FUNCTION 1
but then we have the insane overhead of interning a keyword every time
we simply want get something from a map.
So all that to say, I'm still figuring out the best way to do this.
put this at the top of foo.clj
(ns foo)
That's a really bad error message. But it's complaining that you
haven't set the namespace for the file yet.
> Oh, I see. It can compile to bytecode all by itself, without the help
> of the python binary. :) Somehow I got the impression that clojure-py
> was translating to actual Python code (not *bytecode*).
I started with that idea, but for instance, clojure supports
multi-statement lambdas and Python does not. So the compiler has to
drop to the bytecode level to support all the features it needs.
>
> Is this exec() from the python binary?
Yes
> Ignoring PyPy for the moment, is step 7 the only step where the python
> binary is required?
I'm not exactly sure what you're asking here...all of clojure-py is
python bytecode. From end-to-end it all runs on the python binary.
>> user=> (dis/dis squared)
>> 0 0 LOAD_CONST 1 (#'clojure.core/*)
>> 3 LOAD_ATTR 0 (deref)
>> 6 CALL_FUNCTION 0
>> 9 LOAD_FAST 0 (a)
>> 12 LOAD_FAST 0 (a)
>> 15 CALL_FUNCTION 2
>> 18 RETURN_VALUE
>> None
>> user=>
>
> Is that an example of Python bytecode?
Yes, we're invoking dis from the python dis function.
Ok, thanks for the info. Added the wiki page.
---J