java.lang.NoClassDefFoundError: javax/validation/Payload - Glassfish 5.0 and Payara 5.201 produce same error

[Dustin Peters]

Sorry if this is the wrong place to post this question. I talked to the Payara support (since this exact same problem occurs in gf and payara) and they recommended this forum.

PROBLEM:

I have been trying to migrate my Java EE application to run in a Java 8 (from Java 7) and Glassfish 5.0/Payara 5.201 (from GF 3.1.1) environment. I am able to deploy the application to both GF and Payara 5 seemingly successfully, I can access the applications ui and navigate through it. The problem arises when I try to run one of the many appclients, this is when I get the error message below.

SYMPTOMS:

You can find the full stack trace here: https://stackoverflow.com/questions/62700849/upgrading-from-glassfish-3-1-1-to-5-0-0-java-7-to-java-8-java-lang-noclassdef

MultiException stack 1 of 2
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/validation/Payload
    at org.glassfish.hk2.utilities.cache.LRUHybridCache.compute(LRUHybridCache.java:315)
    at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperImpl.getAllFields(ClassReflectionHelperImpl.java:113)

MultiException stack 2 of 2
java.lang.IllegalArgumentException: Errors were discovered while reifying SystemDescriptor(
    implementation=com.sun.enterprise.config.modularity.ConfigModularityJustInTimeInjectionResolver
    contracts={com.sun.enterprise.config.modularity.ConfigModularityJustInTimeInjectionResolver,org.glassfish.hk2.api.JustInTimeInjectionResolver}

There is another symptom I spotted on deployment, the number of components loaded in glassfish/payara (application->my application) is much lower than a working environment (155 components in broken env vs 450 in working env). There are no deployment errors that I can see.

JAVA:

java version "1.8.0_152"

Java(TM) SE Runtime Environment (build 1.8.0_152-b16)

Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

(Tried a number of java version including jdk1.8.0_202 but gf/payara would not start, had to replace a grizzly-npn-bootstrap.jar with a different version)

APPLICATION SERVER:

Payara Server 5.201

Glassfish 5.0

APPCLIENT EXECUTION:

VMARGS="-Xmx2048m -Xms2048m -Dcom.sun.corba.ee.transport.ORBTCPTimeouts=100:90000:20:500 -Dcom.sun.corba.ee.transport.ORBWaitForResponseTimeout=3600000 $ZAMBER_DEBUG_ARGS $ZAMBER_JVM_ARGS $JMX_ARGS"

export VMARGS

/root/profiler/appclient/glassfish/bin/appclient -client $productclient -mainclass ca.avant.zamber.my_app.scheduler.Main $*

Which, in appclient, executes:

eval `"${ACCJava}" -Dorg.glassfish.appclient.shell $cygwinProp -classpath "${_AS_INSTALL}/lib/gf-client.jar" org.glassfish.appclient.client.CLIBootstrap "$@"`

PROJECT POM:

See attached xml file. Note, while debugging I have attempted to migrate java ee dependencies to java ee 8 (eg. javaee-api).

SOLUTIONS ATTEMPTED:

1. I tried a number of different ways to force the jar containing the Payload class to be loaded (/modules, domain-dir/lib, classpath, etc)

2. I tried updating the applications dependencies to follow the java ee 8 specification.

3. I tried a number of different java versions.

4. I tired to force glassfish/app to use a specific version (2.0.1.FINAL) of the validation-api jar.

5. I tried using payara server instead, behaved exactly the same.

I have spent a ton of time on this problem and have run out of things to try. Nothing I've done so far have had any effect. Any ideas?

Thanks.

[Ondro Mihályi]

Hi,

The exception is thrown from the appclient, which doesn't run inside the server. Therefore it doesn't help if you modify the classpath of the server, it doesn't have any effect. My guess is that some of your appclient dependencies collide with the classes brought in by the appclient JAR, which effectively brings most of the GlassFish / Payara Server dependencies. It's not possible to suppress some of those GF/Payara dependencies so you need to tune the part of the classpath brought by your application dependencies. Most probably you need to exclude some of your application dependencies that clash with the dependencies brought in by GF/Payara Server client JAR.

Classloader issues can be nasty and very difficult to resolve just based on the information on a Payara Forum or StackOverflow post.As a Payara Enterprise Customer, you can get more dedicated support via the customer hub which allows to share more information about your project and get more specific advice from Payara engineers to resolve your issues.

Ondrej

Dátum: utorok 14. júla 2020, čas: 17:13:30 UTC+2, odosielateľ: Dustin Peters

[Dustin Peters]

Thanks for the quick response. 

You are absolutely correct. Appears I misunderstood how this all works.

Coincidentally I figured this out just last night (after working on this for a while). I used strace to track what was happening when I started the appclient.

strace -f -v -o strace-sched.log ./start-my-scheduler.sh

When I looked at the output I checked for mentions of the validation-api.jar (jar that contains the payload class) and found it was trying to use it but more importantly it told me WHERE it was trying to get it from:

stat("/root/my_app/appclient/glassfish/modules/validation-api.jar", 0x7f86916a5fc0) = -1 ENOENT (No such file or directory)

So I manually copied in the jar to the directory above (not as a final solution, still need to figure out why this is happening) and sure enough it got rid of the java.lang.NoClassDefFoundError: javax/validation/Payload error. Now I am getting:

Lookup failed for 'jms/zamberTopicConnectionFactory' in SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.url.pkgs=com.sun.enterprise.naming, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl}

Failed to look up ConnectorDescriptor from JNDI

Failed to start resource adapter : Unable to create a Configuration, because no Bean Validation provider could be found. Add a provider like Hibernate Validator (RI) to your classpath.

Unable to create a Configuration, because no Bean Validation provider could be found. Add a provider like Hibernate Validator (RI) to your classpath.

But I am guessing it's because there are other jars not being found since I see the following lines in the strace output right next to the validation-api line:

stat("/root/my_app/appclient/glassfish/modules/tiger-types.jar", 0x7f86916a5fc0) = -1 ENOENT (No such file or directory)

stat("/root/my_app/appclient/glassfish/modules/hibernate-validator.jar", 0x7f86916a5fc0) = -1 ENOENT (No such file or directory)

Anyways, I now have something to go off of to continue debugging. Like you said I'll need to figure out why these dependencies are clashing with GF/payara.

Thanks for the advice.