Runtime Compilation of Clojure from Android

[George Jahad]

Remco van't Veer has done a lot of great working porting Clojure to
Android, but one thing remains missing, runtime compilation which
would allow a fully functional Repl. The problem is that the Android
VM doesn't use standard Java class files but rather Android specific
ones. There is a host-side tool that converts Java byte codes to
Android ones. But that tool doesn't normally run on Android.

So, being an Android systems hacker, I was wondering how hard it would
be to get runtime compilation working.

Turns out, it doesn't seem to be too hard. The trick is basically to
run the Java to Android/Dalvik byte code converter through itself, so
that the convertor can be run on Android; if you load that converted
convertor into Android, you can then invoke it after each class is
compiled, essentially like this:

if(RT.booleanCast(COMPILE_FILES.deref()))
{
writeClassFile(internalName, bytecode);
String strings[] = {"--dex", "--output=/data/gbj/classes.dex", "/
data/gbj/classes"};
com.android.dx.command.Main.main(strings);
}

(Note that *compile-path* is set elsewhere to "/data/gbj/classes")

Loading the generated class file is also a bit tricky, but not too
bad.


It isn't too fast, partly because of Android's slow GC and partly
because I'm using the filesystem as intermediate storage before
loading the classes, (because I haven't figured out how to do it all
in memory yet.)

I'll send out a full set of patches if there is interest, but for now
here is an Android app with the repl built in so you can play with
it. (It is version of Remco's simple calculator app with a socket
based repl added.)

Much thanks to Remco upon whose Android/Clojure work this is all
built.

App install instructions below.

g


(Note: I've only tested these install instructions out on a Linux host
with this sdk: android-sdk-linux_x86-1.5_r2.zip)

Download the prebuilt apk file here:
http://georgejahad.com/clojure/calc-debug.apk



To configure the emulator:
emulator -avd <your avd name>
adb install -r calc-debug.apk
adb shell mkdir /data/gbj
adb shell mkdir /data/gbj/classes
adb shell chmod 777 /data/dalvik-cache
adb shell chmod 777 /data/gbj
adb shell chmod 777 /data/gbj/classes
adb forward tcp:8030 tcp:8030



Then start up the calc app from the emulator gui.

Then access the repl like so:
telnet localhost 8030

Allow compilation like so:
(def *android* true) (def *compile-files* true)

Now you can compile from the repl. For example, this adds a simple
exponent operator to the calculator:

(in-ns 'examples.calc)
(defn exp [ b e] (reduce * (repeat b e)))
(def calc-opers {\+ #'+
\- #'-
\/ #'/
\* #'* \e #'exp})
(def calc-allowed-chars (.toCharArray "e0123456789."))

You can examine Remco's source code to see how the calculator works
here:
http://github.com/remvee/clj-android/blob/d1e96d33487ddcdc04b403e97f80fcf1b31bb9c2/examples/calc/src/examples/calc.clj

[rb]

Is your source code available somewhere? (I'm interested to know how
you added the socket based REPL and how it interacts with the app )

Raphaël

> To configure the emulator:
> emulator -avd <your avd name>
> adb  install -r    calc-debug.apk
> adb shell mkdir /data/gbj
> adb shell mkdir /data/gbj/classes
> adb shell chmod 777 /data/dalvik-cache
> adb shell chmod 777 /data/gbj
> adb shell chmod 777 /data/gbj/classes
> adb forward tcp:8030 tcp:8030
>
> Then start up the calc app from the emulator gui.
>
> Then access the repl like so:
> telnet localhost 8030
>
> Allow compilation like so:
> (def *android* true) (def *compile-files* true)
>
> Now you can compile from the repl.  For example, this adds a simple
> exponent operator to the calculator:
>
> (in-ns 'examples.calc)
> (defn exp [ b e] (reduce * (repeat b e)))
> (def calc-opers {\+ #'+
>                  \- #'-
>                  \/ #'/
>                  \* #'* \e #'exp})
> (def calc-allowed-chars (.toCharArray "e0123456789."))
>
> You can examine Remco's source code to see how the calculator works

> here:http://github.com/remvee/clj-android/blob/d1e96d33487ddcdc04b403e97f8...

[Remco van 't Veer]

Cool! Please share the code on github or whatever.

I've been looking at reducing the memory footprint so I tend to
consider including dex into your app as a bad thing. But it would be
really cool to do the slime dance during development.

I guess a more efficient approach lures in the bsh implementation at
android-scripting[1]. I haven't investigated yet but browsing the
code reveals it includes a (possibly modified) version of
org.objectweb.asm.

Remco


[1] http://code.google.com/p/android-scripting/

[George Jahad]

On Jun 16, 3:15 am, "Remco van 't Veer" <rwvtv...@gmail.com> wrote:
> Cool!  Please share the code on github or whatever.
>

working on it.

> I've been looking at reducing the memory footprint so I tend to
> consider including dex into your app as a bad thing.  But it would be
> really cool to do the slime dance during development.
>

yup. I never intended this as a long term solution, just an easy
first step that proves the concept.

> browsing the
> code reveals it includes a (possibly modified) version of
> org.objectweb.asm.
>

Now that is interesting. So Android does have a port of ASM! That
really will improve things.

g

[George Jahad]

I'm still cleaning up my changes. But with regards to the repl, I'm
just using clojure.contrib.server-socket and invoking it like so
from the :create routine:

(create-repl-server 8030)
(repl)

[George Jahad]

On Jun 16, 3:15 am, "Remco van 't Veer" <rwvtv...@gmail.com> wrote:

> Cool!  Please share the code on github or whatever.
>

ok, I've forked your clojure tree and added my patches here:
http://github.com/GeorgeJahad/clojure/tree/master
The main changes are in Compiler.java, with a few in build.xml and
main.clj.

Note that you'll probably need to set the sdk-location in build.xml


I've cleaned things up a bit and set out a new prebuilt calc apk here:
http://georgejahad.com/clojure/calc-debug.apk

(Note: I've only tested these install instructions out on a Linux host
with this sdk: android-sdk-linux_x86-1.5_r2.zip)

To configure the emulator:
emulator -avd <your avd name>
adb install -r calc-debug.apk

adb shell mkdir /data/clojure
adb shell mkdir /data/clojure/classes
adb shell chmod 777 /data/clojure
adb shell chmod 777 /data/clojure/classes
adb forward tcp:8032 tcp:8032

Then start up the calc app from the emulator gui.

Then access the repl like so:

telnet localhost 8032

Now you can compile from the repl.

My only change to examples.calc.clj was to add clojure.contrib.server-
socket and invoke it like so from the :create routine:

(create-repl-server 8032)
(repl)
My fork of your calc source is here:
http://github.com/GeorgeJahad/clj-android/blob/7f0497ee1aa155075d97333b73e6c3a7fed64cc3/examples/calc/src/examples/calc.clj

[Marklar]

Very nice, thanks for sharing.

I tried the first version you posted on my rooted G1. It could
evaluate strings and numbers, but unfortunately when I tried to create
a function it didn't do anything. It just stopped, though the activity
was still responding (the menu worked, etc). I looked at logcat but
couldn't find anything.

I will try to look at again in more detail, if I figure it out I will
definitely let you know.

> My fork of your calc source is here:http://github.com/GeorgeJahad/clj-android/blob/7f0497ee1aa155075d9733...

[George Jahad]

Odd, it does work on my rooted G1, (an Android Developer Phone.)

Remco, does it work for you, either on the emulator or the G1?

What you are seeing indicates that run-time compilation isn't working
at all for you. (Self-evalution of numbers and strings from the repl
don't require compilation, and worked even before I made my changes.)
If you get a chance, try the current version of the apk, along with
the current instructions from my 2nd post:
http://groups.google.com/group/clojure/msg/0f378ff4a352e151


If that doesn't work, send me your logcat output as well as an "ls -l"
on:
/data/clojure
/data/clojure/classes

> It just stopped, though the activity was still responding (the menu worked, etc).

I don't believe there is a menu in calc app. Which menu are you
referring to?

Also, if you have time, please try it on the emulator and see if it
works for you there.

Thanks much for the beta test.

g

[George Jahad]

Odd, it does work on my rooted G1, (an Android Developer Phone.)

Remco, does it work for you, either on the emulator or the G1?

What you are seeing indicates that run-time compilation isn't working
at all for you. (Self-evalution of numbers and strings from the repl
don't require compilation, and worked even before I made my changes.)
If you get a chance, try the current version of the apk, along with
the current instructions from my 2nd post:
http://groups.google.com/group/clojure/msg/0f378ff4a352e151


If that doesn't work, send me your logcat output as well as an "ls -l"
on:
/data/clojure
/data/clojure/classes

> It just stopped, though the activity was still responding (the menu worked, etc).

I don't believe there is a menu in calc app. Which menu are you
referring to?

Also, if you have time, please try it on the emulator and see if it
works for you there.

Thanks much for the beta test.

g


On Jun 17, 5:32 am, Marklar <ddil...@gmail.com> wrote:

[George Jahad]

Odd, it does work on my rooted G1, (an Android Developer Phone.)

Remco, does it work for you, either on the emulator or the G1?

What you are seeing indicates that run-time compilation isn't working
at all for you. (Self-evalution of numbers and strings from the repl
don't require compilation, and worked even before I made my changes.)
If you get a chance, try the current version of the apk, along with
the current instructions from my 2nd post:
http://groups.google.com/group/clojure/msg/0f378ff4a352e151


If that doesn't work, send me your logcat output as well as an "ls -l"
on:
/data/clojure
/data/clojure/classes

> It just stopped, though the activity was still responding (the menu worked, etc).

I don't believe there is a menu in calc app. Which menu are you
referring to?

Also, if you have time, please try it on the emulator and see if it
works for you there.

Thanks much for the beta test.

g


On Jun 17, 5:32 am, Marklar <ddil...@gmail.com> wrote:

[Marklar]

I tried the second apk and it works perfectly. Nice job!

My phone is a retail G1 with the JesusFreak 1.51 image. The menu I was
referring to was the menu of the terminal application from which I was
using telnet. Let me know if you'd like to investigate why the first
one wasn't working, I'd be able to look at it this weekend.

Thanks again!

[George Jahad]

On Jun 18, 7:05 am, Marklar <ddil...@gmail.com> wrote:
> I tried the second apk and it works perfectly. Nice job!
>

Cool, thanks.

> My phone is a retail G1 with the JesusFreak 1.51 image. The menu I was
> referring to was the menu of the terminal application from which I was
> using telnet.

You mean you are using the repl from the G1 keyboard itself? How
cool. I hadn't thought of trying that.

> Let me know if you'd like to investigate why the first
> one wasn't working, I'd be able to look at it this weekend.

Don't bother about the other apk; it is already a dim memory from many
sleepless nights ago. The cause of my insomnia? hacking Android with a
repl is totally addictive. ;)

Thanks again,
g