Weaving bunch of classes in memory

39 views
Skip to first unread message

Kalle Kärkkäinen

unread,
Apr 15, 2011, 1:54:47 PM4/15/11
to kilimt...@googlegroups.com
Hi,

I've got most of this down already, but there seemed to be various problems with my approach (I got odd classloading exceptions). In order to get this somewhat documented, I'll post here my approach, and we can then discuss the issues encountered.

The goal is to do in memory compilation, and subsequent weaving. I'm getting stuff compiled, and I am getting them to start the weaving.

problem 1.

kilim.KilimException: Check classpath. Method foo/bar.yield()V could not be located

at kilim.analysis.MethodFlow.visitMethodInsn(MethodFlow.java:204)

at org.objectweb.asm.ClassReader.accept(Unknown Source)

at org.objectweb.asm.ClassReader.accept(Unknown Source)

at kilim.analysis.ClassFlow.analyze(ClassFlow.java:81)

at kilim.analysis.ClassWeaver.weave(ClassWeaver.java:50)

at kilim.analysis.ClassWeaver.<init>(ClassWeaver.java:41)

at org.demos.Demo.compile(Demo.java:114)

In this problem I've got a task that calls yield. Notice that the same runtime has been able to compile the class, and now I'm weaving it (in memory). It has kilim.Task loaded and all should be set.

Since I'm doing dynamic compilation, I've got the pre-weave bytecode in different classloader (lets call it A). I'm trying to get it to generate weaved classes, and I'd load those to classloader B (and then subsequently forget about A). This way there should be no problem with overwriting the bytecode, or any such.

What gives?

-- 
Kalle Kärkkäinen

Sriram Srinivasan

unread,
Apr 16, 2011, 6:50:09 AM4/16/11
to kilimt...@googlegroups.com
What's the classloader hierarchy?

Is it possible to show the code of your "Demo" program, or is it
inextricably linked with other non-related portions of your work? A
simplified test case would be much appreciated.

--ram.

Kalle Kärkkäinen

unread,
Apr 16, 2011, 7:10:00 AM4/16/11
to kilimt...@googlegroups.com
Basically it's like this: the system cl, with all the kilim classes and dependencies (and bunch of others). Then I do:

//node -> java

Map<String, String> codes = create_runner( 1, 1, "", nodes );


where I turn a graph into a set of tasks and a runner (that sets the tasks up in certain order and runs them). It's not really important for this case. Next I do:

//java -> class

DiagnosticCollector<JavaFileObject> diags = new DiagnosticCollector<JavaFileObject>();

CharSequenceCompiler csc = new CharSequenceCompiler(Demo.class.getClassLoader(), null);

Map<String, Class> classes = csc.compile(codes, diags);


Where the CharSequenceCompiler is from here: http://www.ibm.com/developerworks/java/library/j-jcomp/index.html. Thing to note here is that CSC uses it's internal classloader, with the Demo.class.CL being the parent loader. Next I do:

// class -> weaved

for (Map.Entry<String, Class> e : classes.entrySet()) {

InputStream bcode = e.getValue().getClassLoader()

.getResourceAsStream(e.getKey() + ".class");

ClassWeaver weaver = new ClassWeaver(bcode, Detector.DEFAULT);

List<ClassInfo> cis = weaver.getClassInfos();

if (cis.size() > 0) {

for (ClassInfo ci : cis) {

/*

* OutputStream o = jfo.openOutputStream();

* o.write(ci.bytes); o.close();

*/

}

}

}


To turn them into weaved counterparts. At this stage i use the CSC classloader to get the bytecode (bcode) form the classes. This then ends up being weaved, but it fails at the creation of the classweaver  (since it calls weave straight of the bat internally).

If further info is needed, I can try to make a compilable test, but I think all the info is here. I think that one might get the same result with code that does:

public class foo extends task
public void execute throws pausable
yield();

I don't think anything else is needed. It just can't find the parent class. I suspect that weaving does not search the parent chain of the given classloader.

-- 
Kalle Kärkkäinen

Sriram Srinivasan

unread,
Apr 16, 2011, 11:33:24 AM4/16/11
to kilimt...@googlegroups.com
Quickfix solution: In your generated code, make static calls fully qualified. 

That is, instead of calling  yield(),  make it Task.yield(). 

-------------------

This is a bug in the weaver. When you say an unqualified "yield", it translates to MyTask.yield() in the bytecode. When the weaver looks for such a method, it finds that it can't locate MyTask itself. That is because it uses its own classloader to load MyTask, which looks for it on disk. 

I wouldn't be surprised if this is the source of other such problems. I'll use you as a guinea pig to ferret them out. Sorry! 
Meanwhile, I'll look into it in a little more detail and fix this asap.

--sriram.

----------------------------------------------------------------------

The test case is as follows: 

import java.io.*;
import java.util.*;

import javax.tools.*;
import javaxtools.compiler.*;

import kilim.analysis.*;

public class Kcl {
    public static void main(String[] args) throws Exception {
        String src =
            "import kilim.*;"
            "class MyTask extends Task {" +
            "   public void execute() throws Pausable{" +
            "        Task.yield();" +
            "    }" +
            "}";

        InputStream is = compile(src);
        weave(is);
    }

    private static void weave(InputStream is) throws IOException {
        ClassWeaver cw = new ClassWeaver(is, Detector.DEFAULT);
        for (ClassInfo ci: cw.getClassInfos()) {
            System.out.println("Woven: " + ci.className);
        }
    }

    

    private static InputStream compile(String src) throws CharSequenceCompilerException {
        DiagnosticCollector<JavaFileObject> errs = new DiagnosticCollector<JavaFileObject>();
        CharSequenceCompiler<Object> csc = 
            new CharSequenceCompiler<Object>(Thread.currentThread().getContextClassLoader(), 
                    Arrays.asList(new String[] { "-target", "1.5" }));
        Class<Object> clss = csc.compile("MyTask", src, errs, new Class<?>[] {Object.class});
        InputStream is = clss.getClassLoader().getResourceAsStream("MyTask.class");
        return is;
    }
}



Sriram Srinivasan

unread,
Apr 16, 2011, 12:41:17 PM4/16/11
to kilimt...@googlegroups.com

I wrote earlier:

> Quickfix solution: In your generated code, make static calls fully
> qualified.
>
> That is, instead of calling yield(), make it Task.yield().
>

While this is correct, the solution is useless. If, instead of
yield(), you were to call another method foo() defined in the same
class, you'd have the same error because the weaver's classloader
depends on the reflection framework to load the pre-weave class and to
provide signatures for all the methods.

Clearly, the correct solution is to define a classloader for the
weaver, then load the pre-woven classes into the classloader. That
way, it can find a signature for MyTask.foo() while it is weaving
MyTask's bytes.

Watch this space for the solution ..

---sriram.

Kalle Kärkkäinen

unread,
Apr 18, 2011, 2:51:15 AM4/18/11
to kilimt...@googlegroups.com
Ok, to give an update:

java.io.IOException: Class not found

at org.objectweb.asm.ClassReader.a(Unknown Source)

at org.objectweb.asm.ClassReader.<init>(Unknown Source)

at kilim.analysis.ClassFlow.<init>(ClassFlow.java:41)

at kilim.analysis.ClassWeaver.<init>(ClassWeaver.java:40)

at org.demos.Demo.compile(Demo.java:114)


Basically this is my task, line to line

package analysis;

import java.util.*;

import org.uncommons.maths.random.*;

import org.joda.time.*;

import kilim.*;


public class random extends kilim.TaskWrapper {

public FwdMailBox<TSPoint> output = new FwdMailBox<TSPoint>();


public void execute() throws Pausable{

Random r = new XORShiftRNG(new byte[]{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20});

double stddev = 2;

double last = 50;

long t = new DateMidnight().getMillis();

int i = 100000;

int c = 0;

while(i-- > 0){

double v = last += r.nextDouble()*stddev;

last= v;

t += 1000*60*60*24l;

output.put(new TSPoint(t, v));

c++;

if(i%10 == 0)

Task.yield();

}

output.put(TSPoint.END);

calculated=true;

}

}


TaskWrapper is an extension for Task, it contains some scheduling analysis critical data, for instance the calculated flag. FwdMailbox is an outbound mailbox (basically a list of mailboxes).

I can't really fathom the problem here. I took a shot to update to asm 3.3.1, and that seemed to go down fairly easily (although there are problematic bits like the intermixed instructions and line number / frame nodes). This did not fix the problem though. Is this the same issue as the one we uncovered last time?

-- 
Kalle Kärkkäinen

Kalle Kärkkäinen

unread,
Apr 18, 2011, 5:28:14 AM4/18/11
to kilimt...@googlegroups.com
Consider that one solved. It was my bad, basically I inadvertently had one version of one class loaded via eclipse, and that had a different classloader, resulting in bytecode not found via resource loading, and thus empty stream -> class not found.

But I'm now definitely facing the class loading problem:

kilim.KilimException: Check classpath. Method analysis/FwdMailBox.put(Ljava/lang/Object;)V could not be located

at kilim.analysis.MethodFlow.visitMethodInsn(MethodFlow.java:204)

at org.objectweb.asm.ClassReader.accept(Unknown Source)

at org.objectweb.asm.ClassReader.accept(Unknown Source)

at kilim.analysis.ClassFlow.analyze(ClassFlow.java:81)

at kilim.analysis.ClassWeaver.weave(ClassWeaver.java:50)

at kilim.analysis.ClassWeaver.<init>(ClassWeaver.java:41)

at org.demos.Demo.compile(Demo.java:115)



Can't put messages to my forwarding messagebox. :( 

I guess I can move forward by doing it 'c-style'. ie:

static put(box, msg);


-- 
Kalle Kärkkäinen

Sriram Srinivasan

unread,
Apr 18, 2011, 11:12:06 PM4/18/11
to kilimt...@googlegroups.com

I believe that there's a reasonably easy and correct way to enable run-
time weaving _without_ requiring classloading. .

Here's the current situation, and the reason for the problems. When
presented with a method invocation, the weaver calls Class.forName
(via RuntimeMirror) to get the method's signature. Since the weaver
is in the classpath, it is loaded by the system classloader, which
means that it relies on the system classloader to load other classes,
which in turn means that the system relies on the classpath to locate
all the files that need to be woven. In your case, it won't find the
new classes, hence the problems.

The solution is to not rely on the classloader at all. Clearly, the
weaver has access to the bytecode, so it can figure out the method
signatures itself. The complication is that the weaver cannot process
one file at a time. For example, if a class relies on an interface,
one must make sure that the interface is processed (and its method
signatures made available) before the class is processed. A slightly
more complex case is that of circular class references. This means
that the weaver requires two passes, one to absorb all the signatures
of all the classfiles presented to it, then to do the validation and
weaving step. The good thing is that it does not require any
classloader.

Given this, if you can infer what to do from the description above,
feel free to volunteer :) I'll get to it myself in the next couple of
days, but can't promise urgency.

--sriram.

Kalle Kärkkäinen

unread,
Apr 19, 2011, 7:39:28 AM4/19/11
to kilimt...@googlegroups.com
On tiistaina 19. huhtikuuta 2011 at 6.12, Sriram Srinivasan wrote:
Here's the current situation, and the reason for the problems. When
presented with a method invocation, the weaver calls Class.forName
(via RuntimeMirror) to get the method's signature. Since the weaver
is in the classpath, it is loaded by the system classloader, which
means that it relies on the system classloader to load other classes,
which in turn means that the system relies on the classpath to locate
all the files that need to be woven. In your case, it won't find the
new classes, hence the problems.
So it uses the cl that loaded the lib? So as an interim solution I could just load the weaver to a cl under dynamic content..? Or that I could make the weaver use what even cl I give it?
The solution is to not rely on the classloader at all. Clearly, the
weaver has access to the bytecode, so it can figure out the method
signatures itself. The complication is that the weaver cannot process
one file at a time. For example, if a class relies on an interface,
one must make sure that the interface is processed (and its method
signatures made available) before the class is processed. A slightly
more complex case is that of circular class references. This means
that the weaver requires two passes, one to absorb all the signatures
of all the classfiles presented to it, then to do the validation and
weaving step. The good thing is that it does not require any
classloader.
This is a Good IIdea. Much better than the above interim ideas.
Given this, if you can infer what to do from the description above,
feel free to volunteer :) I'll get to it myself in the next couple of
days, but can't promise urgency.
Ok, I'll race you. :) I'll probably get on it to night, so if you've already started, you have an advantage. :D

--
Kalle.

Kalle Kärkkäinen

unread,
Apr 19, 2011, 7:58:19 AM4/19/11
to kilimt...@googlegroups.com
Having a definable ClassLoader did the trick in the mirroring code. It's not pretty, and does not cover the multiple classloader situation, so it's not of the shut-down-everything solution. I'll dig deeper.

-- 
Kalle Kärkkäinen

Kresten Krab Thorup

unread,
Apr 19, 2011, 12:08:06 PM4/19/11
to kilimt...@googlegroups.com, kilimt...@googlegroups.com
I handle these cases in erjang (multiple class loaders, runtime weaving). Basically you need to introduce a level of indirection so that the weaver does not try to classload it's dependent classes, but acces the meta information through a "class info handler".

Kalle Kärkkäinen

unread,
Apr 20, 2011, 3:48:11 AM4/20/11
to kilimt...@googlegroups.com

On tiistaina 19. huhtikuuta 2011 at 19.08, Kresten Krab Thorup wrote:

I handle these cases in erjang (multiple class loaders, runtime weaving). Basically you need to introduce a level of indirection so that the weaver does not try to classload it's dependent classes, but acces the meta information through a "class info handler".
So would you place the indirection layer as a Detector or as one of the runtimemirrors? Personally I think this boils down to having a classname->classloader map.

My case is such that I compile task graphs through generated code into weaved tasks. Now this process produces a single classloadable segment. If I down the line decide to re-use the stuff I compiled earlier and currently have stashed in certain classloader, I'd have a map saying these-classes -> use that loader, those-classes -> use this one. This map is fairly simple to do in runtime, since it's über straight forward to extend the classloader to support this.

As such this operation happens naturally in the classForName method of RuntimeClassMirrors. It should be simple to do a runtimemirror for the detector, such that it is able to work with a classloader map.

Is there some special gain in the more complex approaches?

nice qcon video btw. Your video got me to kilim.

--
Kalle
Reply all
Reply to author
Forward
0 new messages