Creating a runtime mirror based on Jar file

[Mads Hartmann Jensen]

Hi,

I'm trying to create a runtime mirror based on a jar file.

It should be simply a matter of passing a URLClassLoader to scala.reflect.runtime.universe.runtimeMirror but

it doesn't appear to do the trick.

I've attached a small program that tries to do it. Run it using `scala JarReflection.scala path_to.jar` e.g. 

`scala JarReflection.scala /usr/local/Cellar/scala/2.10.2/libexec/lib/scala-library.jar`.

In my case it simply outputs 

Creating classloader based on /usr/local/Cellar/scala/2.10.2/libexec/lib/scala-library.jar

Scope{

  final package _root_;

  final package <empty>

}

I would've expected the scala package to be there.

If you don't want to download the file, I've copy-pasted it below:

---- JarReflection.scala ----

import java.net.JarURLConnection

import java.net.URL

import java.net.URLClassLoader

import scala.reflect.runtime.universe

object JarReflection {

  def main(args: Array[String]) {

     val path = args.head

     val url = new URL(s"jar:file:$path!/")

     // This would fail if the URL wasn't right. Otherwise we don't need it.

     val jarConnection = url.openConnection.asInstanceOf[JarURLConnection]

     val manifest = jarConnection.getManifest()

    println(s"Creating classloader based on $path")

    val urlcl = URLClassLoader.newInstance(Array(url))

    val mirror = universe.runtimeMirror(urlcl)

    println(mirror.RootClass.toType.members)

  }

}

[Jason Zaugg]

On Wed, Jul 3, 2013 at 12:31 PM, Mads Hartmann Jensen <mad...@gmail.com> wrote:

Hi,

I'm trying to create a runtime mirror based on a jar file.

It should be simply a matter of passing a URLClassLoader to scala.reflect.runtime.universe.runtimeMirror but

it doesn't appear to do the trick.

I've attached a small program that tries to do it. Run it using `scala JarReflection.scala path_to.jar` e.g. 

`scala JarReflection.scala /usr/local/Cellar/scala/2.10.2/libexec/lib/scala-library.jar`.

In my case it simply outputs 

Creating classloader based on /usr/local/Cellar/scala/2.10.2/libexec/lib/scala-library.jar

Scope{

  final package _root_;

  final package <empty>

}

I would've expected the scala package to be there.

Runtime reflection never tries to enumerate the classloader, so you can't browse packages. They are populated as a side effect of loading specific classes within that package.

See: https://issues.scala-lang.org/browse/SI-6393

An alternative is to use the real compiler, typically through IMain. (Example [1])

-jason

[1] https://github.com/twitter/util/blob/master/util-eval/src/main/scala/com/twitter/util/Eval.scala

[Mads Hartmann Jensen]

Hi Jason and Eugene,

Okay I see. Makes sense that I doesn't work then ;)

What I'm trying to achieve is to find all the classes/traits/ etc. that are defined

in a given jar.

I could just read the bytecode but I would prefer to not have to de-mangle names

and reverse the bytecode back to the proper Scala representation :)

Jason, from the example you linked it seems that they're concerned mostly

with evaluating Scala code on the fly. Will the 'real' compiler allow me to traverse

all the classes that were loaded from a given jar?

Cheers,

Mads

[Mads Hartmann Jensen]

Okay, so compiler.RootClass.tpe.declarations makes it possible to traverse all of the

the symbols that are on the class path. Great :)

I can't seem to find a method that does this just for the jars that were passed as the

classpath to the compiler though.

If I run println(compiler.RootClass.tpe.declarations) it prints

Scope{
  final package scala;
  final package doc;
  final package sun;
  final package com;
  final package java;
  final package javax;
  final package org;
  final package apple;
  final package sunw;
  final package oracle;
  final package quicktime;
  final package <empty>;
  final package _root_
}

which is considerable more packages than what exists in the small jar I added to the class path.

I've attached the program so you can see how the compiler instance is created.

Cheers,

Mads

[Jason Zaugg]

On Wed, Jul 3, 2013 at 2:20 PM, Mads Hartmann Jensen <mad...@gmail.com> wrote:

Okay, so compiler.RootClass.tpe.declarations makes it possible to traverse all of the

the symbols that are on the class path. Great :)

I can't seem to find a method that does this just for the jars that were passed as the

classpath to the compiler though.

If I run println(compiler.RootClass.tpe.declarations) it prints

Scope{
  final package scala;
  final package doc;
  final package sun;
  final package com;
  final package java;
  final package javax;
  final package org;
  final package apple;
  final package sunw;
  final package oracle;
  final package quicktime;
  final package <empty>;
  final package _root_
}

This is a pretty tricky topic. Let me point out a few pitfalls.

First of all, the symbols you see are sometimes speculative: the compiler doesn't eagerly parse the bytecode, instead, it initialially populates the symbols based on the names of the classfiles alone. A single Java originated class might result in both a Class- and Module-Symbol, so it always creates both. See "Why I got stuck on package objects" [1] for a more thorough explanation.

I don't think you can exclude the Java/Scala standard libraries. But you can ask each symbol for the classfile from which it was loaded.

scala> typeOf[org.apache.commons.lang3.time.DateUtils].typeSymbol

res26: $r.intp.global.Symbol = class DateUtils

scala> res26.associatedFile

res27: scala.reflect.io.AbstractFile = /Users/jason/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.1.jar(org/apache/commons/lang3/time/DateUtils.class)

Problem is, this is not populated until the symbol is initialized:

scala> typeOf[org.apache.commons.lang3.time.DateUtils].typeSymbol.owner.info.members.map(_.associatedFile)

res32: Iterable[scala.reflect.io.AbstractFile] = List(, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , /Users/jason/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.1.jar(org/apache/commons/lang3/time/DateUtils.class), , , , , , , , , , , , , , )

Worse, you can't just go and call .initialize on all of the symbols as you might trigger cycles.

You can get the associated file out *without* initializing the symbol with some hackery:

scala>  println(pack.info.members.map{s => (s.fullName, s.rawInfo match { case cfl: loaders.ClassfileLoader => cfl.classfile; case _ => s.associatedFile })}.toList.mkString("\n"))

(org.apache.commons.lang3.time.FastDateFormat$TimeZoneNumberRule,/Users/jason/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.1.jar(org/apache/commons/lang3/time/FastDateFormat$TimeZoneNumberRule.class))

(org.apache.commons.lang3.time.FastDateFormat$TimeZoneNumberRule,/Users/jason/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.1.jar(org/apache/commons/lang3/time/FastDateFormat$TimeZoneNumberRule.class))

(org.apache.commons.lang3.time.FastDateFormat$1,/Users/jason/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.1.jar(org/apache/commons/lang3/time/FastDateFormat$1.class))

(org.apache.commons.lang3.time.FastDateFormat$1,/Users/jason/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.1.jar(org/apache/commons/lang3/time/FastDateFormat$1.class))

(org.apache.commons.lang3.time.FastDateFormat$PaddedNumberField,/Users/jason/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.1.jar(org/apache/commons/lang3/time/FastDateFormat$PaddedNumberField.class))

Here, we inspect the LazyType `rawinfo` to find that it is a `ClassfileLoader`, which has the associated field as a member.

What is your higher level use case, BTW? Maybe there is an easier way.

-jason

[1] https://groups.google.com/forum/#!topic/scala-internals/8bwPH8_65XE

[Mads Hartmann Jensen]

Hi Jason,

Thanks a lot, I'm pretty sure you've saved me a couple of days of research here :) 

The use-case is pretty simple: For each type (class/trait/object, etc.) that is defined

in a jar I want the fully qualified identifier and the super-types.

I need this information to find all sub-types of a given type. 

It would be nice if we could get additional information such as all public fields in

case we want to implement a Hoogle like clone inside of the IDE but that's just

a pet project of mine so it's not that important atm :) 

Cheers,

Mads

[Eugene Burmako]

Why can't you call initialize everywhere? Imagine a Scala program that makes use of all the classes (top-level classes, because inner ones get unpickled together with the top-level ones) in a jar. It will eventually have called initialize on every symbol, no?

[Mads Hartmann Jensen]

I ended up taking a slightly different route:

- Get all the classfiles from the jar using http://docs.oracle.com/javase/6/docs/api/java/util/jar/JarFile.html#entries()

- Demangle the names

-  use a runtime mirror based on the class loader of the jar to load all of the entities

This approach seems to work. 

Cheers,

Mads

[Peter Jones]

On Friday, 5 July 2013 04:57:32 UTC+1, Mads Hartmann Jensen wrote:

I ended up taking a slightly different route:

- Get all the classfiles from the jar using http://docs.oracle.com/javase/6/docs/api/java/util/jar/JarFile.html#entries()

- Demangle the names

-  use a runtime mirror based on the class loader of the jar to load all of the entities

This approach seems to work. 

Cheers,

Mads

Hi,

Sorry for reviving an old thread.

I've been going through much the same pain (for a different use case: dynamically load all modules implementing a trait).  I've passed a bunch of class (and potentially JAR) files to a URLClassLoader (well, the nicer scala.tools.nsc.util.ScalaClassLoader), read the class names from files on disk, looked up the class names in the runtime mirror to find those with a type signature that have the required trait and retrieved the relevant module mirror.  All is working to that point.

The critical next step is to get a reference to the module object instance itself with moduleMirror.instance.asInstanceOf[Sims3ResourceHandler].  This is  failing with

java.lang.ClassNotFoundException: info.drealm.s3pi.defaultWrapper.DefaultSims3ResourceHandler$
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)

where "info.drealm.s3pi.defaultWrapper.DefaultSims3ResourceHandler$" is the correct name.

Mads, you said you had an approach that seems to be working for you.  Could you expand on this?

You can see my approach here:
https://github.com/pljones/Reflection/blob/master/src/scala/info/drealm/s3pi/WrapperDealer.scala

(I'd also appreciate any improvements to the scala code in general -- I've only been learning it a few weeks.)

Thanks,

-- Peter

[Peter Jones]

On Thursday, 6 February 2014 20:55:47 UTC, I wrote:

Hi,

Sorry for reviving an old thread.

I've been going through much the same pain (for a different use case: dynamically load all modules implementing a trait).  I've passed a bunch of class (and potentially JAR) files to a URLClassLoader (well, the nicer scala.tools.nsc.util.ScalaClassLoader), read the class names from files on disk, looked up the class names in the runtime mirror to find those with a type signature that have the required trait and retrieved the relevant module mirror.  All is working to that point.

The critical next step is to get a reference to the module object instance itself with moduleMirror.instance.asInstanceOf[Sims3ResourceHandler].  This is  failing with

java.lang.ClassNotFoundException: info.drealm.s3pi.defaultWrapper.DefaultSims3ResourceHandler$
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)

where "info.drealm.s3pi.defaultWrapper.DefaultSims3ResourceHandler$" is the correct name.

Mads, you said you had an approach that seems to be working for you.  Could you expand on this?

You can see my approach here:
https://github.com/pljones/Reflection/blob/master/src/scala/info/drealm/s3pi/WrapperDealer.scala

(I'd also appreciate any improvements to the scala code in general -- I've only been learning it a few weeks.)

Thanks,

-- Peter

I have now pieced together a bit of a hack using a method from stackoverflow to instantiate a companion class by name:
https://github.com/pljones/Reflection/commit/03bc8d6a257ce85b999694842c061e4507172c44

I'm not sure whether it's "good Scala 2.10" or not but it definetely works.  "Test.scala" uses the companion objects to instantiate new classes using the apply method on the companion class.

I would still very much appreciate guidance on any suggestions for a "better Scala 2.10" approach.

Thanks,

-- Peter