Run-time compilation

[Jon Harrop]

Are there any examples of run-time compilation from the Java language
anywhere?

--
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?u

[Arne Vajhøj]

Jon Harrop wrote:
> Are there any examples of run-time compilation from the Java language
> anywhere?

You want to compile Java source to byte code runtime ?

Various third party packages existed before 1.6, but
it was added as part of standard Java in 1.6.

What do you need ? file->file ? file->memory ?
memory->file ? memory->memory ?

Arne

[Jon Harrop]

Arne Vajhøj wrote:
> Jon Harrop wrote:
>> Are there any examples of run-time compilation from the Java language
>> anywhere?
>
> You want to compile Java source to byte code runtime ?

I want to compile and execute ML source code on the JVM at run time.

> What do you need ? file->file ? file->memory ?
> memory->file ? memory->memory ?

Just memory, no files.

[Jon Harrop]

Stefan Ram wrote:

> Jon Harrop <j...@ffconsultancy.com> writes:
>>Are there any examples of run-time compilation from the Java
>>language anywhere?
>

> http://www.purl.org/stefan_ram/pub/evaluating-expressions-with-java
>
> (I am using files there, which is not necessary,
> it all could also have been compiled in memory.)

Ok, so you're generating Java source code in a file and then invoking the
Java compiler using the new API.

I see there is also a ClassLoader. Is there an API to generate classes
directly, i.e. without going through Java source code as text?

[Mark Space]

Jon Harrop wrote:

> I see there is also a ClassLoader. Is there an API to generate classes
> directly, i.e. without going through Java source code as text?
>

I haven't tried any of these libraries myself, but is this along the
lines of what you're looking for?

<http://java-source.net/open-source/bytecode-libraries>

[Arne Vajhøj]

Jon Harrop wrote:

> Arne Vajhøj wrote:
>> What do you need ? file->file ? file->memory ?
>> memory->file ? memory->memory ?
>
> Just memory, no files.

Below are an example of memory->memory with the Java 1.6 builtin
compiler.

Arne

================================================

package test;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintStream;
import java.net.URI;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;

import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;
import javax.tools.FileObject;
import javax.tools.ForwardingJavaFileManager;
import javax.tools.JavaCompiler;
import javax.tools.JavaFileObject;
import javax.tools.SimpleJavaFileObject;
import javax.tools.StandardJavaFileManager;
import javax.tools.ToolProvider;
import javax.tools.JavaCompiler.CompilationTask;

public class JC {
public static void dynamicCall(String clznam, String src, String
cp, String metnam) throws Exception {
SpecialClassLoader xcl = new SpecialClassLoader();
compileMemoryMemory(src, cp, clznam, xcl, System.err);
Class<?> c = Class.forName(clznam, true, xcl);
Object o = c.newInstance();
c.getMethod(metnam, new Class[] { }).invoke(o, new Object[] { });
}
public static void compileMemoryMemory(String src, String cp,
String name, SpecialClassLoader xcl, PrintStream err) {
JavaCompiler javac = ToolProvider.getSystemJavaCompiler();
DiagnosticCollector<JavaFileObject> diacol = new
DiagnosticCollector<JavaFileObject>();
StandardJavaFileManager sjfm =
javac.getStandardFileManager(diacol, null, null);
SpecialJavaFileManager xfm = new SpecialJavaFileManager(sjfm, xcl);
CompilationTask compile = javac.getTask(null, xfm, diacol,
Arrays.asList(new String[] { "-classpath", cp }), null,
Arrays.asList(new
JavaFileObject[] { new MemorySource(name, src) }));
boolean status = compile.call();
if(err != null) {
err.println("Compile status: " + status);
for(Diagnostic<? extends JavaFileObject> dia :
diacol.getDiagnostics()) {
err.println(dia);
}
}
}
}

class MemorySource extends SimpleJavaFileObject {
private String src;
public MemorySource(String name, String src) {
super(URI.create("string:///" + name.replace(".", "/") +
".java"), Kind.SOURCE);
this.src = src;
}
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
return src;
}
public OutputStream openOutputStream() {
throw new IllegalStateException();
}
public InputStream openInputStream() {
return new ByteArrayInputStream(src.getBytes());
}
}

class MemoryByteCode extends SimpleJavaFileObject {
private ByteArrayOutputStream baos;
public MemoryByteCode(String name) {
super(URI.create("byte:///" + name.replace(".", "/") +
".class"), Kind.CLASS);
}
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
throw new IllegalStateException();
}
public OutputStream openOutputStream() {
baos = new ByteArrayOutputStream();
return baos;
}
public InputStream openInputStream() {
throw new IllegalStateException();
}
public byte[] getBytes() {
return baos.toByteArray();
}
}

class SpecialJavaFileManager extends
ForwardingJavaFileManager<StandardJavaFileManager> {
private SpecialClassLoader xcl;
public SpecialJavaFileManager(StandardJavaFileManager sjfm,
SpecialClassLoader xcl) {
super(sjfm);
this.xcl = xcl;
}
public JavaFileObject getJavaFileForOutput(Location location,
String name, JavaFileObject.Kind kind, FileObject sibling) throws
IOException {
name = sibling.getName().substring(1,
sibling.getName().length() - 5).replace("/", ".");
MemoryByteCode mbc = new MemoryByteCode(name);
xcl.addClass(name, mbc);
return mbc;
}
}

class SpecialClassLoader extends ClassLoader {
private Map<String,MemoryByteCode> m;
public SpecialClassLoader() {
super(SpecialClassLoader.class.getClassLoader());
m = new HashMap<String, MemoryByteCode>();
}
protected Class<?> findClass(String name) throws
ClassNotFoundException {
MemoryByteCode mbc = m.get(name);
if(mbc != null) {
return defineClass(name, mbc.getBytes(), 0,
mbc.getBytes().length);
} else {
throw new ClassNotFoundException(name);
}
}
public void addClass(String name, MemoryByteCode mbc) {
m.put(name, mbc);
}
}

[Arne Vajhøj]

Jon Harrop wrote:
> Arne Vajhøj wrote:
>> Jon Harrop wrote:
>>> Are there any examples of run-time compilation from the Java language
>>> anywhere?
>> You want to compile Java source to byte code runtime ?
>
> I want to compile and execute ML source code on the JVM at run time.

I can not help with the ML part.

There are several Java byte code generating libraries available.

BCEL, CGLIB, ASM, JavaAssist etc..

The only one I have troed is BCEL. See example attached below.

Arne

======================================================

import java.util.*;

import org.apache.bcel.*;
import org.apache.bcel.generic.*;

public class CodeGenAndLoad {
private static Object get(Object o, String field) throws Exception {
return o.getClass().getMethod("get" + field.substring(0,
1).toUpperCase() + field.substring(1), new Class[0]).invoke(o, new
Object[0]);
}
private static void set(Object o, String field, int value) throws
Exception {
o.getClass().getMethod("set" + field.substring(0,
1).toUpperCase() + field.substring(1), new Class[] { int.class
}).invoke(o, new Object[] { new Integer(value) });
}
private static void set(Object o, String field, double value)
throws Exception {
o.getClass().getMethod("set" + field.substring(0,
1).toUpperCase() + field.substring(1), new Class[] { double.class
}).invoke(o, new Object[] { new Double(value) });
}
private static void set(Object o, String field, Object value)
throws Exception {
o.getClass().getMethod("set" + field.substring(0,
1).toUpperCase() + field.substring(1), new Class[] { value.getClass()
}).invoke(o, new Object[] { value });
}
public static void main(String[] args) throws Exception {
Map m = new HashMap();
List d1 = new ArrayList();
d1.add(new FieldDescrip("a", FieldDescrip.FIELD_INT));
d1.add(new FieldDescrip("b", FieldDescrip.FIELD_DOUBLE));
m.put("Bean1", d1);
List d2 = new ArrayList();
d2.add(new FieldDescrip("c", FieldDescrip.FIELD_INT));
d2.add(new FieldDescrip("d", FieldDescrip.FIELD_STRING));
m.put("Bean2", d2);
SpecialClassLoader scl = new SpecialClassLoader(m);
Class c1 = Class.forName("Bean1", true, scl);
Object o1 = c1.newInstance();
System.out.println(get(o1, "a"));
System.out.println(get(o1, "b"));
set(o1, "a", 123);
set(o1, "b", 123.456);
System.out.println(get(o1, "a"));
System.out.println(get(o1, "b"));
Class c2 = Class.forName("Bean2", true, scl);
Object o2 = c2.newInstance();
System.out.println(get(o2, "c"));
System.out.println(get(o2, "d"));
set(o2, "c", 123);
set(o2, "d", "abc");
System.out.println(get(o2, "c"));
System.out.println(get(o2, "d"));
}
}

class FieldDescrip {
public final static int FIELD_INT = 1;
public final static int FIELD_DOUBLE = 2;
public final static int FIELD_STRING = 3;
private String name;
private int typ;
public FieldDescrip(String name, int typ) {
this.name = name;
this.typ = typ;
}
public String getName() {
return name;
}
public int getTyp() {
return typ;
}
}

class SpecialClassLoader extends ClassLoader {
private Map m;
public SpecialClassLoader(Map m) {
this.m = m;
}
protected Class findClass(String name) {
byte[] bc = generateClass(name);
return defineClass(name, bc, 0, bc.length);
}
private Type typeToType(int typ) {
switch(typ) {
case FieldDescrip.FIELD_INT:
return Type.INT;
case FieldDescrip.FIELD_DOUBLE:
return Type.DOUBLE;
case FieldDescrip.FIELD_STRING:
return Type.STRING;
default:
throw new RuntimeException("Bad field typ");
}
}
private byte[] generateClass(String name) {
ClassGen cg = new ClassGen(name, "java.lang.Object", "",
Constants.ACC_PUBLIC | Constants.ACC_SUPER, new String[] { });
ConstantPoolGen cpg = cg.getConstantPool();
InstructionFactory ifact = new InstructionFactory(cg, cpg);
List d = (List)m.get(name);
for(int i = 0; i < d.size(); i++) {
FieldDescrip fd = (FieldDescrip)d.get(i);
FieldGen field = new FieldGen(Constants.ACC_PRIVATE,
typeToType(fd.getTyp()), fd.getName(), cpg);
cg.addField(field.getField());
}
InstructionList il = new InstructionList();
MethodGen constructor = new MethodGen(Constants.ACC_PUBLIC,
Type.VOID, Type.NO_ARGS, new String[] { }, "<init>", name, il, cpg);
il.append(InstructionFactory.createLoad(Type.OBJECT, 0));
il.append(ifact.createInvoke("java.lang.Object", "<init>",
Type.VOID, Type.NO_ARGS, Constants.INVOKESPECIAL));
il.append(InstructionFactory.createReturn(Type.VOID));
constructor.setMaxStack();
constructor.setMaxLocals();
cg.addMethod(constructor.getMethod());
il.dispose();
for(int i = 0; i < d.size(); i++) {
FieldDescrip fd = (FieldDescrip)d.get(i);
MethodGen getter = new MethodGen(Constants.ACC_PUBLIC,
typeToType(fd.getTyp()), Type.NO_ARGS, new String[] { }, "get" +
fd.getName().substring(0, 1).toUpperCase() + fd.getName().substring(1),
name, il, cpg);
il.append(InstructionFactory.createLoad(Type.OBJECT, 0));
il.append(ifact.createFieldAccess(name, fd.getName(),
typeToType(fd.getTyp()), Constants.GETFIELD));

il.append(InstructionFactory.createReturn(typeToType(fd.getTyp())));
getter.setMaxStack();
getter.setMaxLocals();
cg.addMethod(getter.getMethod());
il.dispose();
MethodGen setter = new MethodGen(Constants.ACC_PUBLIC,
Type.VOID, new Type[] { typeToType(fd.getTyp()) }, new String[] {
"value" }, "set" + fd.getName().substring(0, 1).toUpperCase() +
fd.getName().substring(1), name, il, cpg);
il.append(InstructionFactory.createLoad(Type.OBJECT, 0));

il.append(InstructionFactory.createLoad(typeToType(fd.getTyp()), 1));
il.append(ifact.createFieldAccess(name, fd.getName(),
typeToType(fd.getTyp()), Constants.PUTFIELD));
il.append(InstructionFactory.createReturn(Type.VOID));
setter.setMaxStack();
setter.setMaxLocals();
cg.addMethod(setter.getMethod());
il.dispose();
}
return cg.getJavaClass().getBytes();
}
}

[Eric Sosman]

Jon Harrop wrote:
> Stefan Ram wrote:
>
>> Jon Harrop <j...@ffconsultancy.com> writes:
>>> Are there any examples of run-time compilation from the Java
>>> language anywhere?
>> http://www.purl.org/stefan_ram/pub/evaluating-expressions-with-java
>>
>> (I am using files there, which is not necessary,
>> it all could also have been compiled in memory.)
>
> Ok, so you're generating Java source code in a file and then invoking the
> Java compiler using the new API.
>
> I see there is also a ClassLoader. Is there an API to generate classes
> directly, i.e. without going through Java source code as text?

The class file format is documented in the Java Virtual
Machine Specification. Before friendlier tools came on the
scene, I used that documentation to write a compiler that
turned a String like "cos(x) * sin(y)" into an object of a
freshly-loaded class with an evaluate() method that performed
the calculation. So I can testify that the task is do-able.
A bit tedious, but do-able. (The fact that evaluate() would
always be straight-line code without branches, try blocks,
loops, and so on helped minimize the tedium; there were big
chunks of the class file format I could simply ignore.)

--
Eric Sosman
eso...@ieee-dot-org.invalid

[Jon Harrop]

Thanks to everyone for their responses!

Yes. I was really hoping for easy metaprogramming as in F#, Lisp or
MetaOCaml. Looks like this is a lot harder in Java but I may give it a go
if I can find the time.