Could not initialize class com.intergral.apm.transit.impl.WrappedTransaction

[Brad Wood]

I've got a client server running Lucee 4.5.018 and RHEL 7 and I'm seeing the servers occasionally begin throwing this error on all pages. A quick google doesn't reveal anything.  Has anyone seen this before?  The only thing I can think of is that the hard drive has filled up a couple of times with an out-of-control log issue and perhaps that could have corrupted something?  Another interesting thing is this stack trace isn't even from a .cfm file-- it's just Lucee spinning up the request and getting the client storage data source.

Lucee 4.5.2.018 Error (java.lang.NoClassDefFoundError)
Message	Could not initialize class com.intergral.apm.transit.impl.WrappedTransaction
Cause	java.lang.NoClassDefFoundError
Java Stacktrace	Could not initialize class com.intergral.apm.transit.impl.WrappedTransaction   at com.intergral.apm.transit.impl.TransitImpl.addToHistory(TransitImpl.java:659):659   at com.intergral.apm.transit.txn.BaseTransaction._addToTransitHistory(BaseTransaction.java:1854):1854   at com.intergral.apm.metro.api.TransactionSeries.post(TransactionSeries.java:104):104   at com.intergral.apm.transit.impl.TransitImpl._txnClose_TransitHistory(TransitImpl.java:1078):1078   at com.intergral.apm.transit.impl.TransitImpl._txnClose(TransitImpl.java:1036):1036   at com.intergral.apm.transit.impl.TransitImpl.txnClose(TransitImpl.java:626):626   at com.intergral.apm.transit.txn.BaseTransaction._closeTransaction(BaseTransaction.java:1419):1419   at com.intergral.apm.transit.txn.BaseTransaction.close(BaseTransaction.java:569):569   at com.intergral.apm.transit.txn.BaseTransaction.close(BaseTransaction.java:555):555   at com.intergral.fusionreactor.jdbc.MeasurableStatement.metricSignalExecutionEnd(MeasurableStatement.java:175):175   at com.intergral.fusionreactor.jdbc.MeasurableStatement.commonExecuteEnd(MeasurableStatement.java:374):374   at com.intergral.fusionreactor.jdbc.jdbc42.PreparedStatementSurrogate2.execute(PreparedStatementSurrogate2.java:215):215   at lucee.runtime.db.driver.state.StateUtil.execute(StateUtil.java:145):145   at lucee.runtime.db.driver.state.StatePreparedStatement.execute(StatePreparedStatement.java:135):135   at lucee.runtime.type.util.QueryUtil.execute(QueryUtil.java:240):240   at lucee.runtime.type.QueryImpl.<init>(QueryImpl.java:252):252   at lucee.runtime.type.QueryImpl.<init>(QueryImpl.java:204):204   at lucee.runtime.type.scope.storage.db.Ansi92.select(Ansi92.java:72):72   at lucee.runtime.type.scope.storage.StorageScopeDatasource._loadData(StorageScopeDatasource.java:112):112   at lucee.runtime.type.scope.client.ClientDatasource.getInstance(ClientDatasource.java:55):55   at lucee.runtime.type.scope.ScopeContext.getClientScope(ScopeContext.java:233):233   at lucee.runtime.type.scope.ScopeContext.getClientScopeEL(ScopeContext.java:261):261   at lucee.runtime.PageContextImpl.clientScopeEL(PageContextImpl.java:1371):1371   at lucee.runtime.type.scope.UndefinedImpl.reinitialize(UndefinedImpl.java:582):582   at lucee.runtime.PageContextImpl.setApplicationContext(PageContextImpl.java:2813):2813   at lucee.runtime.listener.ModernAppListener.initApplicationContext(ModernAppListener.java:425):425   at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:113):113   at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:35):35   at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2262):2262   at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2225):2225   at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:456):456   at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:47):47   at javax.servlet.http.HttpServlet.service(HttpServlet.java:729):729   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291):291   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206):206   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52):52   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239):239   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206):206   at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source):-1   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43):43   at java.lang.reflect.Method.invoke(Method.java:497):497   at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:97):97   at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:472):472   at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:499):499   at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36):36   at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source):-1   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43):43   at java.lang.reflect.Method.invoke(Method.java:497):497   at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:79):79   at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source):-1   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43):43   at java.lang.reflect.Method.invoke(Method.java:497):497   at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53):53   at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41):41   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java):-1   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219):219   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106):106   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502):502   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142):142   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79):79   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88):88   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518):518   at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091):1091   at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668):668   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527):1527   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484):1484   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142):1142   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617):617   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61):61   at java.lang.Thread.run(Thread.java:745):745
Timestamp	12/29/15 9:34:36 AM PST

Looking through the catalina.out log also reveals some entries like this that look related:

java.lang.NoClassDefFoundError: com/quest/pas/agent/bootstrap/callbacks/DynamicMethodInstrumentorCallbacks

at com.intergral.fusionreactor.plugin.probi.internal.samplers.memory.HeapSampler$MemoryLog.<clinit>(HeapSampler.java)

at com.intergral.fusionreactor.plugin.probi.internal.samplers.memory.HeapSampler.getLogContributions(HeapSampler.java:100)

at com.intergral.fusionreactor.plugin.probi.internal.ProbiComponent$SamplerTask.run(ProbiComponent.java:291)

at java.util.TimerThread.mainLoop(Timer.java:555)

at java.util.TimerThread.run(Timer.java:505)

Caused by: java.lang.ClassNotFoundException: com.quest.pas.agent.bootstrap.callbacks.DynamicMethodInstrumentorCallbacks not found by fr-probi-bundle [29]

at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:812)

at org.apache.felix.framework.ModuleImpl.access$400(ModuleImpl.java:72)

at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1807)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 5 more

Exception in thread "FusionReactor Enterprise Service - Local Metrics Cache" java.lang.NoClassDefFoundError: com/quest/pas/agent/bootstrap/callbacks/DynamicMethodInstrumentorCallbacks

at com.intergral.fusionreactor.metrics.LocalMetricsFactory.<clinit>(LocalMetricsFactory.java)

at com.intergral.fusionreactor.core.universalProcessor.ServiceCommandMetricsCache.run(ServiceCommandMetricsCache.java:89)

Caused by: java.lang.ClassNotFoundException: com.quest.pas.agent.bootstrap.callbacks.DynamicMethodInstrumentorCallbacks not found by fusionreactor-core [22]

at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:812)

at org.apache.felix.framework.ModuleImpl.access$400(ModuleImpl.java:72)

at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1807)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 2 more

java.lang.NoClassDefFoundError: Could not initialize class com.intergral.fusionreactor.plugin.probi.internal.samplers.memory.HeapSampler$MemoryLog

at com.intergral.fusionreactor.plugin.probi.internal.samplers.memory.HeapSampler.getLogContributions(HeapSampler.java:100)

at com.intergral.fusionreactor.plugin.probi.internal.ProbiComponent$SamplerTask.run(ProbiComponent.java:291)

at java.util.TimerThread.mainLoop(Timer.java:555)

at java.util.TimerThread.run(Timer.java:505)

[Charlie Arehart]

While you await a reply perhaps from an engineer, I’d ask a couple things in advance of that:

- what version of FR is this? Not just is it 5 or 6, but what point release? and if you may have updated it recently, did the problem start after that?
- as for the “out of control logging issue”, are you saying that was FR’s doing or something else (and you just then wonder if there may be some residual impact)? I wouldn’t think the error indicated here would be affected by any issues about free space, etc.
- you say the error “isn’t even from a CFM”, but by that I assume you mean only that you don’t see reference to any CFM file in the stack trace. Fair enough. I wonder if, instead, this is reflecting simply the end of page processing, when Lucee (like CF) might be closing whatever jdbc connections it had had open (which could include client variable processing, and I do see the reference earlier/further down in the stack trace to it having used client vars in a datasource)

But the FR guys are great, as you know, and may see something more from what you’ve shared. :-)

/charlie

--
You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fusionreacto...@googlegroups.com.
To post to this group, send email to fusion...@googlegroups.com.
Visit this group at https://groups.google.com/group/fusionreactor.
For more options, visit https://groups.google.com/d/optout.

[Brad Wood]

Thanks for the reply Charlie.

what version of FR is this?

Good call, I should have included that information in my first post.  This is what outputs in the standard out log when the server starts:

Revision:  5.2.7 fusionreactor.204.48715.branches/FR-5.x-MAINT 

as for the “out of control logging issue”, are you saying that was FR’s doing or something else

Something else.  Lucee is logging a large amount of information to the catalina.out file, which Tomcat declines to rotate and the drive has been filling up.  This is something I'm working separately with Lucee on, but I've seen odd things happen when drives fill up and your server goes to write a file, so I thought it was worth mentioning.  

you say the error “isn’t even from a CFM”, but by that I assume you mean only that you don’t see reference to any CFM file

That's correct.  What I meant was that I couldn't show anyone the CFML or SQL that was causing the error because it was coming from the under-the-covers part of the request that presumably executes before or after the CFML portion of the code.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

[Brad Wood]

and if you may have updated it recently

 

Oops, forgot that question.  No, FR has not been upgraded recently.  

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

[Charlie Arehart]

OK, hope that (and the next note) may help them help you. :-)

To be clear, 5.2.7 was indeed the last update to the FR 5 release. As you may know, FR 6 came out a couple of months ago, and is now on 6.0.3, if you may want to consider that.

But I realize you may prefer first to just find out if the FR guys may have any explanation for what you are seeing now. I’ll leave it to you all to take it forward from this point. :-)

/charlie

[Brad Wood]

Yes, we have notified our client of the availability of FR 6 and they are working on the upgrade path for that.  In the mean time, I would still like to know what is tripping up the current install :)

--

[Neil Wightman]

Hi Brad,

The problem isn't specifically a bug in FRs code (any OSGI based program would show this problem) but a problem with class loaders and multiple java agents.

It appears the customer has multiple java agents in use. The class com.quest.pas.agent.bootstrap.callbacks.DynamicMethodInstrumentorCallbacks isn't part of FR, but has been injected into FR base code.  Which then triggers the problem.

We saw a similar issue when using AppDynamics and FR.
http://www.fusion-reactor.com/support/kb/frs-357/

You could try following the same steps and add the com.quest.pas.* to the org.osgi.framework.bootdelegation argument as documented in the above link.

I cannot find more information about the com.quest code as it appears quest was bought by dell.
 
Regards,
Neil

[Brad Wood]

Thanks for the reply Neil.  My client did add Foglight to the server as a java agent for additional monitoring.  Perhaps you can do some testing with that as well.  I'll test the work around in your KB article and see if it fixes the issue.  

Thanks!

~Brad

[Brad Wood]

Just wanted to confirm that this fix did make Fusion Reactor begin working again.  I followed the steps in the knowledgebase article, but added ",com.quest.pas.*" to the "org.osgi.framework.bootdelegation" argument.

Is this something a future version of FR could add in by default for out-of-the-box compatibility with FogLight monitoring?

Thanks~

~Brad

[Neil Wightman]

Hi Brad.

Did you only add
",com.quest.pas.*"
if so I can merge that into the default file shipped with FR.

Thanks
Neil

--

[Brad Wood]

Yes, that's all I needed to add and it seems to work fine now.  Thanks for following up.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com