Lucee 5 crashing (Exact issue happening in 4.5 also)

[Lyle Karstensen]

I could really use some help on this..

I have been plagued for a long time (years) with my services just stopping taking requests. It have gone as far as writing a service monitor that will restart my app when it become unresponsive. Recently I upgraded to Lucee 5 hoping things would be better as I cannot find anything in my code that could be causing this (but most likely it is the code). I have fusionreactor installed and once the service became unresponsive I started my investigation. It turned out that all the threads were in a "blocked" state. I have found that 5-7% of my requests get the error below.

Unlike other errors in my logs when it is a .cfm or .cfc file from my code, I can see that in the error log. In this case it appears to be something either in jetty or lucee getting concurrent modification errors then never releasing the thread block.

java.util.ConcurrentModificationException

at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)

at java.util.HashMap$KeyIterator.next(HashMap.java:1453)

at java.util.Collections$3.nextElement(Collections.java:5216)

at lucee.runtime.net.http.HTTPServletRequestWrap.disconnect(HTTPServletRequestWrap.java:364)

at lucee.runtime.PageContextImpl.release(PageContextImpl.java:550)

at lucee.runtime.CFMLFactoryImpl.releaseLuceePageContext(CFMLFactoryImpl.java:199)

at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:852)

at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:103)

at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:62)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:806)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:339)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:215)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:839)

at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:483)

at com.intergral.fusionreactor.agent.filter.FusionReactorStaticServlet.service(FusionReactorStaticServlet.java:50)

at com.intergral.fusionreactor.agent.pointcuts.ServletPointCut$1.invoke(ServletPointCut.java:41)

at javax.servlet.http.HttpServlet.service(HttpServlet.java)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

at org.eclipse.jetty.server.Server.handle(Server.java:370)

at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)

at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)

at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

at java.lang.Thread.run(Thread.java:745)

[Harry Klein]

Hi Lyle,

 

I have similar issues, see

https://luceeserver.atlassian.net/browse/LDEV-947

 

I added a comment with your stacktrace.

 

-Harry

--
Get 10% off of the regular price for this years CFCamp in Munich, Germany (Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead of 210€. Visit https://ti.to/cfcamp/cfcamp-2016/discount/Lucee@cfcamp
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lyle Karstensen]

This all starts to make sense. My framework uses <cfsavecontent><cfinclude></cfsavecontent> for every request.

I wonder if using <cflock> will be a temporary fix?

[Lyle Karstensen]

BTW.. THANKS.. 

On Monday, August 1, 2016 at 12:39:03 AM UTC-7, Harry Klein wrote:

[Harry Klein]

The problem occurs for me only when multiple requests occur in the same millisecond, e.g. for very fast includes.

I guess <cflock> could be a temporary (but very ugly) fix.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/d1489d56-a219-4624-98e5-f7ab538a87e0%40googlegroups.com.

[Lyle Karstensen]

I assume the same thing is my issue because it happens randomly. Usually when the Search Engines are hammering our sites. This explains why there are times it goes a day or two and other times the servers reboot 30 times a day.