Multi-JAR Scanning

[Yunior Betancourt]

Hi!

I ran into an interesting issue while trying to scan several jars at the same time. I have three maven projects. The first projects is a library that contains an annotation called "AnntoationWIthClassValue". The second project is an app which uses this annotation and assigns the "Foo.class" value to the clazz property of the annotation in a test class. The third projects is the scanner project which tries to find the type of the annotation. In order for the scan to work both the compiled application jar and test jar are added to a classloader and then passed in to FCS, then the scan is ran. What happens though is that the class matcher throws a class not found exception when trying to load Foo because that class in not found in the test jar, it is in the compiled app jar. I have a github project with a test replicating the issue here. My question is, should this class loading be supported across jars, and is there a way to get around it if not?

Regards,

Yunior

[Luke Hutchison]

Hi Yunior,

This should actually work just fine whenever the two classes from different jars are handled by the same ClassLoader. For every class found during scan, FCS records which ClassLoader provided the URL to the jar that contains the class. The problem in this case is that (based on your previous bug report) one of your jars is a Spring-Boot jar, and the scanner is not running from within that jar. Since Spring-Boot jars do not have the package root at the root of the jar (instead, it is at "BOOT-INF/classes"), URLClassLoader cannot load classes from Spring-Boot jars, so in order to load classes from that jar, FCS has to extract the contents of the jar to disk, and then create a new URLClassLoader for that jar.

So here are your options to get all the classes visible by the same ClassLoader:

(1) Get a class reference from the Spring-Boot jar, then call .getClassLoader() on that class, to get the custom URLClassLoader that FCS created, and cast it to URLClassLoader, then get the URLs from that URLClassLoader, and add to that list the URL to the second class that can't currently see the Spring-Boot class. Then create your own URLClassLoader from that augmented list of URLs, and use that ClassLoader using Class.forName(className, false, augmentedClassLoader). One important thing though: make sure that whatever class you load through FCS, in order to get a reference to FCS' custom URLClassLoader, is not one of the classes you care about -- because once that class is loaded, you can't unload it, since it is cached by the classloading system. That means that you will end up with that class definition loaded into FCS' ClassLoader, and the other class definition loaded into your own custom ClassLoader. The two classes won't be able to link to each other. That will cause problems.

(2) Put your scanner class inside the Spring-Boot jar, and start it up by running the Spring-Boot jar, so that the Spring-Boot ClassLoader is run (this will cause Spring-Boot to extract all the contents of the jarfile itself, just as FCS does, and set up its own URLClassLoader). Then when you run this jar, add the second jar on the commandline using the "-classpath" option. The Spring-Boot ClassLoader will delegate to the parent ClassLoader before doing its own ClassLoading from the "exploded" (extracted) Spring-Boot jar, and the parent ClassLoader will handle the second jar.

Let me know if that works for you.

Luke

--
You received this message because you are subscribed to the Google Groups "FastClasspathScanner-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fastclasspathscanne...@googlegroups.com.
To post to this group, send email to fastclasspath...@googlegroups.com.
Visit this group at https://groups.google.com/group/fastclasspathscanner-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/fastclasspathscanner-users/b56b06f7-16d7-48a1-aadb-040354242061%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luke Hutchison]

Hi Yunior,

I built option (1) into FastClasspathScanner, released in version 3.1.10. Try calling FastClasspathScanner#createClassLoaderForMatchingClasses() before starting the scan. Let me know if this works for you. Note that you should not try to load any classes from any of the relevant jars before starting the scan, otherwise some of the classes will be cached in the environment classloader, whereas new classes will be loaded by the new classloader. (All classes you care about need to be loaded by the same classloader.)

https://github.com/lukehutch/fast-classpath-scanner/releases/tag/fast-classpath-scanner-3.1.10

Luke

[Luke Hutchison]

Hi Yunior,

I dropped back to JDK 8 (indeed your code does not work on JDK 10), and I was able to reproduce the problem.

Here is the issue: in your code, you do:

.matchClassesWithAnnotation(AnnotationWithClassValue.class, classAnnotationMatchProcessor)

However, using the class reference in red means that this annotation class is already loaded into the context classloader for the "scanner" project. In the ClassAnnotationMatchProcessor, you do:

AnnotationWithClassValue a = classWithAnnotation.getAnnotation(AnnotationWithClassValue.class);

This is the line that throws the exception. Note that the .getAnnotation(clazz) method does not take a ClassLoader parameter, the way that Class.forName() does in one of its forms. That means that .getAnnotation(clazz) can only find annotations that are loaded into the current classloader.

At that point though, classWithAnnotation has already been loaded by FCS' own custom classloader -- a custom classloader had to be created using .createClassLoaderForMatchingClasses(), since you are adding a Spring-Boot jar to the classpath, and the scanner is not running within that jar. So now you are dealing with two different classloaders: the annotated class was loaded by the custom FCS classloader, and the annotation class was already loaded (and cached) by the context classloader. You cannot have classes linked to each other that span more than one classloader.

The solution here is to get FCS to do all your classloading. That means you should never reference any of the classes in your scanned packages using a class reference of the form AnnotationWithClassValue.class, but instead, only ever reference this class by name, until scanning is complete. Once scanning is complete, you can use the FCS classloader to get class references using methods like classInfo.getClassRef(), or annotationClassRef.getType().

Here is how you would do the scanning without doing any classloading until you can do so safely through the FCS custom classloader. (This is FCS' own API that sort of mimics the Java reflection API.)

        ScanResult scanResult = new FastClasspathScanner("com.foo").overrideClassLoaders(loader)
                .createClassLoaderForMatchingClasses()
                .ignoreParentClassLoaders()
                .scan();
        List<String> annotatedClasses = scanResult
                .getNamesOfClassesWithAnnotation("com.lib.externalLib.AnnotationWithClassValue");
        for (String annotatedClass : annotatedClasses) {
            ClassInfo annotatedClassInfo = scanResult.getClassNameToClassInfo().get(annotatedClass);
            // You can safely load the annotated class here using: Class<?> annotatedClass = annotatedClassInfo.getClassRef();
            for (AnnotationInfo annotationInfo : annotatedClassInfo.getAnnotationInfo()) {
                if (annotationInfo.getAnnotationName().equals("com.lib.externalLib.AnnotationWithClassValue")) {
                    AnnotationClassRef clazz = null;
                    String name = null;
                    List<AnnotationParamValue> annotationParamValues = annotationInfo.getAnnotationParamValues();
                    for (AnnotationParamValue annotationParamValue : annotationParamValues) {
                        String paramName = annotationParamValue.getParamName();
                        if (paramName.equals("clazz")) {
                            clazz = (AnnotationClassRef) annotationParamValue.getParamValue();
                        } else if (paramName.equals("name")) {
                            name = (String) annotationParamValue.getParamValue();
                        }
                    }
                    System.out.println(annotatedClassInfo.getClassName() + " has annotation "
                            + annotationInfo.getAnnotationName() + " with params: clazz = " + clazz.getType()
                            + " ; name = \"" + name + "\"");
                    // You can safely load the annotation class parameter here using: Class<?> annotationClassParam = clazz.getType();
                }
            }
        }

This should solve your problem. Basically avoid using MatchProcessors altogether, and never use class references to look up classes in the ScanResult. (In fact, I will probably remove both of those from a future release, they cause a lot of problems like this.)

Thanks for your patience while I figured out what was going on here!

Luke

[Luke Hutchison]

PS I just made a change to deprecate AnnotationClassRef#getType() for AnnotationClassRef#getClassRef(), for consistency with ClassInfo#getClassRef(). This will be in the next release. (Just mentioning this since the clazz.getType() call will show up as deprecated soon.)

[Marc Magon]

I'm going to try that config for pulling on JDK 10, I've noticed a few libraries getting missed and have been trying to pinpoint where/why