GXP Dynamic Mode

[wcrosby]

I have complied a GXP in dynamic mode using the ant task and deployed
them to a tomcat server. When I first load the gxp everything renders
properly. However, when I change the GXP tempate and it attempts to
recompile I get:

inmem:///com/.../web/gxp/TestTemplate$Impl1.java:14: package
com.google.gxp.base does not exist.

I get this for all of the gxp packages:
com.google.gxp.css
com.google.gxp.html
com.google.gxp.js
com.google.gxp.text
com.google.gxp.base.dynamic
com.google.gxp.base

The gxp.jar is in the lib directory of my webapp. Why is the dynamic
compiler unable to locate it?

[Laurence Gonsalves]

I haven't tried the dynamic mode with Tomcat, so I'm not sure exactly
why it isn't working.

In dynamic compilation mode, the GXP compiler is invoked, and then the
Java compiler (javax.tools.JavaCompiler) is invoked on its output. The
error you're seeing looks like it's coming from the Java compiler, so
my guess is that something is preventing it from "seeing" the GXP jar.

The code that gets and uses the JavaCompiler instance is in
com.google.gxp.base.dynamic.StubGxpTemplate. We don't really do a huge
amount of configuration, except to use a different JavaFileManager.
Even then, our JavaFileManager forwards to the StandardFileMangaer
(returned by javaCompiler.getStandardFileManager()), but that seems
like the most logical place to start. Do you think you could try and
debug the problem?

[Wayne Crosby]

You are correct that it's the JavaCompiler that's throwing these errors.  The GXP is compliled without any issue.  My best guess is that the JavaCompiler is using a different class loader than the one that loaded the Gxp.  Tomcat has lots of classloaders and I've run into simalar problems of not being able to find jars because of it in the past.  I've looked at this code a bit and done some searching around.  There is a post here: http://forums.sun.com/thread.jspa?threadID=5295526 that does some extra classpath work.  I'm going to try and modify the JavaFileManager and explicitly add the gxp jar to see if that works.  If so, I'll look at resolving how to use the right class loader -- it shouldn't be too hard (famous last words).  Any other thoughts on how to debug this one?

-Wayne

[Musachy Barroso]

There are 2 ways to tell the compiler about the classpath, one is
passing an "options" (-cp foo.jar) parameter to getTask(...), which is
not used in GXP:

javaCompiler.getTask(null, javaFileManager, diagnosticCollector,
null, null, compilationUnits).call();

the other one is to implement a file manager, and that's how GXP does
it in JavaFileManagerImpl. I can't say for sure what the problem is
because I haven't debug it, but I have hat lots of troubles using the
second option with my own code, and the first one works rather easily,
like:

....
List<String> optionList = new ArrayList<String>();
Set<String> classPath = new HashSet<String>()

//this jar
classPath.add(getJarUrl(EmbeddedJSPResult.class));
//servlet api
classPath.add(getJarUrl(Servlet.class));
//jsp api
classPath.add(getJarUrl(JspPage.class));

optionList.addAll(Arrays.asList("-classpath",
StringUtils.join(classPath, ";")));


//compile
JavaCompiler.CompilationTask task = compiler.getTask(
null, jfm, diagnostics, optionList, null,
Arrays.asList(sourceCodeObject));
....


protected String getJarUrl(Class clazz) {
ProtectionDomain protectionDomain = clazz.getProtectionDomain();
CodeSource codeSource = protectionDomain.getCodeSource();
URL loc = codeSource.getLocation();
File file = FileUtils.toFile(loc);
return file.getAbsolutePath();
}

I did something similar to compile JSP to java, using Jasper, and then
use the compiler API to compile to bytecode in memory, so I thought I
would share the code:

http://svn.apache.org/viewvc/struts/sandbox/trunk/struts2-jsp-plugin/src/main/java/org/apache/struts2/

musachy

--
"Hey you! Would you help me to carry the stone?" Pink Floyd

[Wayne Crosby]

Here is what I discovered...  It's failing to compile the GXP at this line in StubGxpTemplate:

javaCompiler.getTask(null, javaFileManager, diagnosticCollector,
                           null, null, compilationUnits).call();

After setting a few breakpoints in JavaFileManagerImpl it looked like the call to

public ClassLoader getClassLoader(Location location)

was returning the top level tomcat class loader (all the jars in the $TOMCAT_HOME/lib directory) and not the webapp.  So I added the following to JavaFileManagerImpl:

public ClassLoader getClassLoader(Location location) {
  if (location.equals(StandardLocation.CLASS_OUTPUT)) {
    return JavaFileManagerImpl.class.getClassLoader();
  }
  return super.getClassLoader(location);
}

Setting breakpoints here confirmed it is returning the ClassLoader with all the jars I would expect.
However, I'm still getting the same error messages that it can't find the gxp packages.

The javaCompiler task it's attempting to load is com.sun.tools.javac.api.JavacTool.  Unfortnately, I the only source I can find for this is the open jdk 6 version wich doesn't seem to sync up with Mac java 1.6.0, so it's next to impossible to debug what is actually happening.  I think next steps are to build and install open jdk on my mac to get the byte code and source to line up.  Any other suggestions?

-Wayne

[Wayne Crosby]

After much poking and debugging, I have a hacked solution that works and I need some advice on how to get to the general solution.

First in StubGxpTemplate,

javaCompiler.getTask(null, javaFileManager, diagnosticCollector,
Arrays.asList(new String[] {"-verbose", "-classpath", "/Developer/apache-tomcat-6.0.16/webapps/ROOT/WEB-INF/lib/gxp.jar"}), null, compilationUnits).call();

By adding the jar explicitly to the classpath of the compiler, it will find the GXP classes correctly.  Obviously we will need several other jars on the classpath for a real template.  So, given a ClassLoader is there an easy way to reconstruct the classpath?  As it turns out for Tomcat there is a WebappClassLoader that provides getters to get at all the required jars.  But, that's a very specific solution and I'd rather come up with something more general.  Or, is there an some way to tell JavaCompiler to use a classpath instead of having to specify the classpath as a String argument?

Second, the defineClass(byte[] classFile) method had to change to install the new class in the right ClassLoader.

private static Class defineClass(byte[] classFile)
      throws Throwable {
  Object[] args = new Object[]{ null, classFile,
      new Integer(0), new Integer(classFile.length),
      PROTECTION_DOMAIN };
  try {
    // Used to be ClassLoader.getSystemClassLoader(), webapps are unable to find the newly created class if it is defined here.  Instead, just use the ClassLoader that loaded this class.
    return (Class) DEFINE_CLASS.invoke(StubGxpTemplate.class.getClassLoader(), args);
  } catch (InvocationTargetException e) {
    throw e.getCause();
  }
}

-Wayne

[Musachy Barroso]

There isn't any easy way to reconstruct the classpath (or better, I
haven't found one). You can get the system classpath, but that will
fail in some cases, like if you run tests from maven for example. If
the classes that are referenced in the compiled gxp are always the
same, you can find the location of the jars like in the code that I
posted. If you really need to find all the jars you can do something
like:

private static URL getClassesURL(ClassLoaderInterface classLoader)
throws IOException {
List<URL> urls = Collections.list(classLoader.getResources(""));
for (URL url : urls) {
String externalForm =
StringUtils.removeEnd(url.toExternalForm(), "/");
if (externalForm.endsWith(".war/WEB-INF/classes")) {
//if it is inside a war file, get the url to the file
externalForm =
StringUtils.substringBefore(externalForm, CLASSES_DIR);
return URLUtil.normalizeToFileProtocol(new URL(externalForm));
} else if (externalForm.endsWith(CLASSES_DIR)) {
return URLUtil.normalizeToFileProtocol(url);
}
}

return null;
}

private static List<URL> getUrls(ClassLoaderInterface classLoader)
throws IOException {
List<URL> list = new ArrayList<URL>();

//find jars
ArrayList<URL> urls =
Collections.list(classLoader.getResources("META-INF"));

for (URL url : urls) {
if ("jar".equalsIgnoreCase(url.getProtocol())) {
String externalForm = url.toExternalForm();
//build a URL pointing to the jar, instead of the META-INF dir
url = new
URL(StringUtils.substringBefore(externalForm, "META-INF"));
list.add(url);
} else if (LOG.isDebugEnabled())
LOG.debug("Ignoring URL [#0] because it is not a jar",
url.toExternalForm());

}

//get the "classes" dir
URL classesURL = getClassesURL(classLoader);
if (classesURL != null)
list.add(classesURL);

return list;
}

And then convert the URLs to absolute paths. I would strongly
discourage this option, as I keep getting surprised by what is
returned by getResources(...) on different containers (yes JBoss 5, I
am looking at you). In the code above, ClassLoaderInterface is just an
interface that abstract a class loader.

musachy

[Laurence Gonsalves]

That's exactly the type of change I was thinking might work based both
on reading the JavaFilManager docs and from that forums.sun.com thread
you'd linked to the other day. It's very puzzling to me that it
doesn't work. Debugging the javac code is the best thing I can think
of too.

On Thu, Aug 13, 2009 at 12:27 AM, Wayne Crosby<wcr...@gmail.com> wrote:

[Laurence Gonsalves]

On Thu, Aug 13, 2009 at 1:35 AM, Wayne Crosby<wcr...@gmail.com> wrote:
> After much poking and debugging, I have a hacked solution that works and I
> need some advice on how to get to the general solution.
>
> First in StubGxpTemplate,
>
> javaCompiler.getTask(null, javaFileManager, diagnosticCollector,
> Arrays.asList(new String[] {"-verbose", "-classpath",
> "/Developer/apache-tomcat-6.0.16/webapps/ROOT/WEB-INF/lib/gxp.jar"}), null,
> compilationUnits).call();
>
> By adding the jar explicitly to the classpath of the compiler, it will find
> the GXP classes correctly.  Obviously we will need several other jars on the
> classpath for a real template.

Agreed.

> So, given a ClassLoader is there an easy way
> to reconstruct the classpath?

I'm pretty sure the answer to this is "no". A ClassLoader doesn't even
need to have a concept of a "classpath". To a large extendt the public
interface of a ClassLoader is just a map from (fully-qualified) class
names to Class objects.

> As it turns out for Tomcat there is a
> WebappClassLoader that provides getters to get at all the required jars.
> But, that's a very specific solution and I'd rather come up with something
> more general.  Or, is there an some way to tell JavaCompiler to use a
> classpath instead of having to specify the classpath as a String argument?

Do you mean "use a ClassLoader instead"...? If so, that's what I was
hoping the JavaFileManagerImpl.getClassLoader would let you do. Did
you have any luck debugging the open jdk JavacTool to see why it was
failing despite giving it the right classloader?

> Second, the defineClass(byte[] classFile) method had to change to install
> the new class in the right ClassLoader.
>
> private static Class defineClass(byte[] classFile)
>       throws Throwable {
>   Object[] args = new Object[]{ null, classFile,
>       new Integer(0), new Integer(classFile.length),
>       PROTECTION_DOMAIN };
>   try {
>     // Used to be ClassLoader.getSystemClassLoader(), webapps are unable to
> find the newly created class if it is defined here.  Instead, just use the
> ClassLoader that loaded this class.
>     return (Class)
> DEFINE_CLASS.invoke(StubGxpTemplate.class.getClassLoader(), args);
>   } catch (InvocationTargetException e) {
>     throw e.getCause();
>   }
> }

Oh, of course. :-Q

Technically, instead of StubGxpTemplate's ClassLoader, that should
probably be the ClassLoader that loaded the stub template, though I
can't think of a situation where they'd actually be different. I'm
guessing that before you made this change you got a very different
error from the javac error that started this thread. Is that right?

[Laurence Gonsalves]

Thanks for the pointers, Musachy!

GXPs can reference arbitrary Java classes (actually, the same is true
of JSP too, isn't it?) so unless we have the same "complete" class
path as used to load the stub template, the template implementation
will always have the chance of behaving in a way that's inconsistent
with a statically compiled GXP.

I keep wondering why the JavaFileManager.getClassLoader() approach
isn't working. Just based on the docs it seems like it's the "right"
way to do what we're trying to do, but it sounds like theory and
practice may not be in agreement.

[Musachy Barroso]

On Thu, Aug 13, 2009 at 11:34 AM, Laurence Gonsalves<laur...@google.com> wrote:
> GXPs can reference arbitrary Java classes (actually, the same is true
> of JSP too, isn't it?) so unless we have the same "complete" class
> path as used to load the stub template, the template implementation
> will always have the chance of behaving in a way that's inconsistent
> with a statically compiled GXP.
>

Yes JSPs can reference other classes, but it has to do so using tag
libraries, so I make a list of the tag libraries used and find the
jars they were loaded from and add it to the classpath. This does not
include scriptlets, but I didn't care about those.

> I keep wondering why the JavaFileManager.getClassLoader() approach
> isn't working. Just based on the docs it seems like it's the "right"
> way to do what we're trying to do, but it sounds like theory and
> practice may not be in agreement.

I also tried that way but I had a hard time understanding it, and the
documentation is very little.

musachy