BLOCKED on org.apache.felix.framework.ServiceRegistry

[Jason Loo]

Good Morning -

Running FusionReactor ENT Edition v.5 and ColdFusion9. Performing some load testing, trying to get RPS above 500. However, there seems to be a cap: no matter how simple I make my request - it could be a CFM that does nothing - I am unable to exceed 400 RPS. I kept clicking the Stack Trace button within the FR UI until I was able to generate one. I noticed that there is blocking thread - see below. Is this the blocking code part of the FR framework? If I am looking for maximum throughput, do I need to disable FR? Thank you in advance.

"jrpp-299" Id=1687 BLOCKED on org.apache.felix.framework.ServiceRegistry@276fad23 owned by "jrpp-222" Id=1598
java.lang.Thread.State: BLOCKED
at org.apache.felix.framework.ServiceRegistry.getServiceReferences(ServiceRegistry.java:191)
- waiting to lock org.apache.felix.framework.ServiceRegistry@276fad23 owned by "jrpp-222"
at org.apache.felix.framework.Felix.getServiceReferences(Felix.java:3041)
at org.apache.felix.framework.Felix.getAllowedServiceReferences(Felix.java:3102)
at org.apache.felix.framework.BundleContextImpl.getServiceReferences(BundleContextImpl.java:386)
at org.apache.felix.framework.BundleContextImpl.getServiceReference(BundleContextImpl.java:327)
at com.intergral.fusionreactor.core.api.FRAPIImpl.(FRAPIImpl.java:101)
at com.intergral.fusionreactor.service.FusionReactorService.getFRAPIImpl(FusionReactorService.java:267)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intergral.fusionreactor.api.FRAPI.getInstance(FRAPI.java:156)
at com.intergral.fusionreactor.j2ee.core.FusionRequest.finished(FusionRequest.java:456)
at com.intergral.fusionreactor.j2ee.core.FusionReactor.finish(FusionReactor.java:150)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:428)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:507)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:79)
at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53)
at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)
at jrun.servlet.FilterChain.doFilter(FilterChain.java)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

[Darren Pywell]

Hi Jason,

There is no throttling to 400 or 500 requests and I can around 3000-4000 RPS on my Mac with an empty page. I can run into problems after a while with the connections on my mac getting somehow throttled and that will result in the requests dropping to about 300-400 for me. If I restart the load application it gets better though.

Are you logging the requests to disk and the disk IOPS are limited?

What version of FR 5 is it exactly?

Thanks,

Darren

[Brad Wood]

Sometimes when I try to run load tests, I'll saturate my network adapter or disk before hitting the actual limit of the server.  This all depends on the tool you're using, how much data is returned from each request, and the network latencies. 

Start by seeing of the network adapter is saturated on your server or the computer where you're running the tests from.  Also note, load testing software can eat a lot of CPU if you're running the tool on the server itself.

If CPU and network usage are low, but you think there should be higher throughput, grab some stack traces of the entire JVM to see what the running threads are doing.  It might be something like a query on the page that's slowing it down.  And finally, check your thread queue settings for your servlet container (Jrun I assume) and ColdFusion.  Depending on how many concurrent threads your load testing tool is using, you may be creating more threads on CF than it's configured to allow and getting requests queued.  

Thanks!

~Brad

Developer Advocate

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

--
You received this message because you are subscribed to the Google Groups "FusionReactor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fusionreactor+unsubscribe@googlegroups.com.
To post to this group, send email to fusion...@googlegroups.com.
Visit this group at https://groups.google.com/group/fusionreactor.
For more options, visit https://groups.google.com/d/optout.

[Neil Wightman]

Hi Jason,

You have found a minor bottleneck in FusionReactor 5, this was fixed in FusionReactor 6 (maybe even in a patch version of FR 5.x).  This issue caused FR 5 to take a few milliseconds more than it should (due to this service lookup).  Under a normal test case this overhead is not visible (as its 2ms of a whole page) but if you are running an empty page to see how quick your CF will run, you could see code in the stacktraces.

I cant remember exactly when I improved this code but if you can upgrade to the newest FR 6 this issue will not exist.

Thanks
Neil

This issue was fixed in FR 6 and if