FusionReactor 6 to 7 upgrade - massively slow performance

43 views
Skip to first unread message

Tom Chiverton

unread,
Apr 16, 2018, 4:29:06 AM4/16/18
to FusionReactor
I replaced my v6 .jar with the latest v7, and suddenly most of our sites stopped working :-/ Only on the dev server :)

This is the latest version of Amazon Linux, Java 1.8, 64bit, Lucee v4.5 latest.

Replacing the v7 jar with v6.2.2 brings the server back to normal

The symptoms appear in multiple sites, but seem to be clustered around anywhere that attempts to use a java object via createObject().

An example follows (S3.cfc line 8):
variables.awsCreds = createObject("java","com.amazonaws.auth.InstanceProfileCredentialsProvider").init();

Full stack :
"ERROR","Thread-117","04/16/2018","08:05:45","","could not terminate the thread;java.lang.Throwable;java.lang.Throwable
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:103)
        at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930)
        at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
        at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
        at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
        at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841)
        at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
        at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
        at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
        at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
        at sun.misc.URLClassPath.findResource(URLClassPath.java:214)
        at java.net.URLClassLoader$2.run(URLClassLoader.java:569)
        at java.net.URLClassLoader$2.run(URLClassLoader.java:567)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findResource(URLClassLoader.java:566)
        at java.lang.ClassLoader.getResource(ClassLoader.java:1096)
        at com.intergral.fusionreactor.agent.transformer.ClassInfo.loadClassInfo(ClassInfo.java:88)
        at com.intergral.fusionreactor.agent.transformer.ClassInfo.loadClassInfo(ClassInfo.java:61)
        at com.intergral.fusionreactor.agent.utils.PointCutUtil.implementsInterface(PointCutUtil.java:75)
        at com.intergral.fusionreactor.agent.pointcuts.database.cassandra.server.CassandraCQLStatement.classMatches(CassandraCQLStatement.java:43)
        at com.intergral.fusionreactor.agent.transformer.PointCutTransformer.matchingClassPointCuts(PointCutTransformer.java:555)
        at com.intergral.fusionreactor.agent.transformer.PointCutTransformer.transform(PointCutTransformer.java:452)
        at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
        at java.lang.Class.getConstructors(Class.java:1651)
        at lucee.runtime.reflection.storage.WeakConstructorStorage.store(WeakConstructorStorage.java:60)
        at lucee.runtime.reflection.storage.WeakConstructorStorage.getConstructors(WeakConstructorStorage.java:45)
        at lucee.runtime.reflection.Reflector.getConstructorInstance(Reflector.java:426)
        at lucee.runtime.reflection.Reflector.getConstructorInstance(Reflector.java:419)
        at lucee.runtime.reflection.Reflector.callConstructor(Reflector.java:809)
        at lucee.runtime.java.JavaObject.init(JavaObject.java:295)
        at lucee.runtime.java.JavaObject.call(JavaObject.java:222)
        at lucee.runtime.java.JavaObject.call(JavaObject.java:259)
        at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:743)
        at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1599)
        at pages.edm.api.manager.s3_cfc$cf.initComponent(/pages/edm/api/Manager/S3.cfc:8)

Tom

Neil Wightman

unread,
Apr 16, 2018, 5:35:19 AM4/16/18
to FusionReactor

Hi Tom,


I'm very sorry this has caused a problem.  There may be a performance issue with the Cassandra tracking in FR, even if cassandra is not being used.


Can you try turning off this pointcut in your FR instance.


To do this you need to create an fusionreactoragent.properties file in the FR instance directory (next to the fusionreactor.jar).   Inside the file you need to have a single line :

com.intergral.fusionreactor.agent.pointcuts.database.cassandra.server.CassandraCQLStatement=false


This should disable the part of FR which may be affecting performance of class loading.


Sorry again,

Neil Wightman

Tom Chiverton

unread,
Apr 16, 2018, 6:53:47 AM4/16/18
to FusionReactor
That's worked; and no, we don't use Cassandra.

Will we need to add this to all our instances going forward, or is a permanent fix now in the works ?

Tom

Neil Wightman

unread,
Apr 16, 2018, 7:57:20 AM4/16/18
to fusion...@googlegroups.com

Hi Tom.  

Yes we have already started working on the fix.

Regards,

Neil

--
You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fusionreacto...@googlegroups.com.
To post to this group, send email to fusion...@googlegroups.com.
Visit this group at https://groups.google.com/group/fusionreactor.
For more options, visit https://groups.google.com/d/optout.

--
Neil Wightman
Senior Software Engineer
Email: neil_w...@intergral.com
Company Web http://www.intergral.com

This e-mail and any files transmitted with it are confidential and may be protected by legal privilege. If you are not the intended recipient, please notify the sender and delete the e-mail from your system. This e-mail has been scanned for malicious content but the internet is inherently insecure and Intergral GmbH cannot accept any liability for the integrity of this message or its attachments.

charlie arehart

unread,
Apr 17, 2018, 8:29:09 PM4/17/18
to fusion...@googlegroups.com

Great job sorting that out so quickly, Neil. Can you help us (following along) to know if that is a problem that could happen to pretty much anyone? Or was there something about Tom’s info that clued you in to it, which may be unique to his setup? And can you say when the issue arose (which FR update), for people who may wonder if their subject to it?

Also, Tom described it as “massively slow performance”, but he didn’t give metrics. Tom, are we talking seconds or minutes per page load? You say that the sites “stopped working”, so perhaps you mean pages never seemed to render.

Also, Tom shows an error having been tracked. Neil, would people see poor performance because of this while not seeing any such errors?

 

/charlie

Neil Wightman

unread,
Apr 18, 2018, 11:42:09 AM4/18/18
to FusionReactor
Hi Charlie,

The issue can exist for anyone and it would affect startup mostly and class loading times.   We already have a fix and this will be released in 7.2.2 soon, hopefully tomorrow.

This issue exists in all current 7 releases but we have only had 1 person see this issue to an extreme level.

It should not affect how the pages execute once they have been loaded / compiled so I doubt it would show as an error for the users.

Regards,
Neil

charlie arehart

unread,
Apr 18, 2018, 4:01:21 PM4/18/18
to fusion...@googlegroups.com

OK, thanks.

/charlie

From: fusion...@googlegroups.com <fusion...@googlegroups.com> On Behalf Of Neil Wightman
Sent: Wednesday, April 18, 2018 10:42 AM
To: FusionReactor <fusion...@googlegroups.com>
Subject: Re: [fusionreactor] Re: FusionReactor 6 to 7 upgrade - massively slow performance

 

Hi Charlie,

Tom Chiverton

unread,
Apr 19, 2018, 11:15:48 AM4/19/18
to FusionReactor
Mostly for others, but by "massive" I mean instead of a few hundred MS to return a page, all requests ran to our request time out limit (~ 1 hundred seconds).

No sure why we were so heavily impacted. Nothing special in our setup...

Tom

Tom Chiverton

unread,
Apr 23, 2018, 4:13:23 AM4/23/18
to FusionReactor
7.2.2 had the fix in the release notes, so I upgraded the server and removed the pointcut line from the config, and it's much happier now.

Thanks !

Tom

charlie arehart

unread,
Apr 24, 2018, 10:19:44 AM4/24/18
to fusion...@googlegroups.com

These guys are good! :-)

 

/charlie

 

From: fusion...@googlegroups.com <fusion...@googlegroups.com> On Behalf Of Tom Chiverton
Sent: Monday, April 23, 2018 03:13 AM
To: FusionReactor <fusion...@googlegroups.com>
Subject: Re: [fusionreactor] Re: FusionReactor 6 to 7 upgrade - massively slow performance

 

7.2.2 had the fix in the release notes, so I upgraded the server and removed the pointcut line from the config, and it's much happier now.



Thanks !

Tom

On Monday, April 16, 2018 at 12:57:20 PM UTC+1, Neil Wightman wrote:

Reply all
Reply to author
Forward
0 new messages