Sitestats issue - Wicket page lock with 100% CPU usage

[Daniel Merino]

Dear community,

we have been affected by a critical issue related to Sitestats.

One teacher tried to generate a report of the use of Resources in his
site, the report didn't finish, an AJAX error was shown in the browser
console and one exception was written in the logs.

The browser console error was like this:

    GET: https://<server>/portal/site/<SITE>/tool/<TOOL>/<ASCII
CHUNK>?height=230&width=53& =<NUMBER> 500 (500)
    Wicket.Ajax: Wicket.Ajax.Call.failure: Error while parsing
response: 500

And I paste the stacktrace at the end of this email, but the more
relevant part was:

    24-may-2020 12:06:35.837 WARN [ajp-nio-18009-exec-152]
org.apache.wicket.page.PageAccessSynchronizer.lockPage Thread
'ajp-nio-18009-exec-152' failed to acquire lock to page with id '0',
attempted for 1 minute out of allowed 1 minute. The thread that holds
the lock has name 'ajp-nio-18009-exec-147'.
    24-may-2020 12:06:35.845 WARN [ajp-nio-18009-exec-152]
org.apache.wicket.page.PageAccessSynchronizer.dumpSingleThread
"ajp-nio-18009-exec-147" daemon prio=5 tid=93978 state=RUNNABLE
    org.apache.wicket.util.lang.Threads$ThreadDump
    (...)
    24-may-2020 12:07:35.852 ERROR [ajp-nio-18009-exec-152]
org.apache.wicket.request.RequestHandlerStack.detach Error detaching
RequestHandler
    org.apache.wicket.page.CouldNotLockPageException: Could not lock
page 0. Attempt lasted 1 minute

The worst thing of this was that one of the four CPU cores of the server
reached a 100% of usage and didn't recover. Pasting the URL of the
console again in the browser caused the same issue and another core
jumped to 100% of usage. With two more pastes we would have killed the
server!

We have found a similar issue described in the last comment of this
JIRA: https://jira.sakaiproject.org/browse/SAK-41751 but with no
solution AFAIK.

The site doesn't have a huge number of events or resources. It's a usual
site.

Does anybody know about this issue and how could we prevent or fix it?
Due to the current online evaluation we have been forced to disable
sitestats temporarily.

We are with Sakai 12.6. We have tried to update Wicket to version 6.30
just like it is in trunk, but the issue remained.

Please, any idea or help is welcome. I can provide more info if you need.

Thanks in advance.
Best regards.

Stacktrace:

24-may-2020 12:06:35.837 WARN [ajp-nio-18009-exec-152]
org.apache.wicket.page.PageAccessSynchronizer.lockPage Thread
'ajp-nio-18009-exec-152' failed to acquire lock to page with id '0',
attempted for 1 minute out of allowed 1 minute. The thread that holds
the lock has name 'ajp-nio-18009-exec-147'.
24-may-2020 12:06:35.845 WARN [ajp-nio-18009-exec-152]
org.apache.wicket.page.PageAccessSynchronizer.dumpSingleThread
"ajp-nio-18009-exec-147" daemon prio=5 tid=93978 state=RUNNABLE
org.apache.wicket.util.lang.Threads$ThreadDump
    at sun.java2d.loops.DrawGlyphListAA.DrawGlyphListAA(Native Method)
    at sun.java2d.pipe.AATextRenderer.drawGlyphList(AATextRenderer.java:41)
    at sun.java2d.pipe.GlyphListPipe.drawString(GlyphListPipe.java:72)
    at sun.java2d.SunGraphics2D.drawString(SunGraphics2D.java:2957)
    at
org.jfree.text.TextUtilities.drawAlignedString(TextUtilities.java:366)
    at org.jfree.chart.axis.PeriodAxis.drawTickLabels(PeriodAxis.java:870)
    at org.jfree.chart.axis.PeriodAxis.draw(PeriodAxis.java:632)
    at org.jfree.chart.plot.XYPlot.drawAxes(XYPlot.java:3007)
    at org.jfree.chart.plot.XYPlot.draw(XYPlot.java:2531)
    at org.jfree.chart.JFreeChart.draw(JFreeChart.java:1175)
    at org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1349)
    at org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1329)
    at org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1314)
    at
org.sakaiproject.sitestats.impl.chart.ChartServiceImpl.generateTimeSeriesChart(ChartServiceImpl.java:654)
    at
org.sakaiproject.sitestats.impl.chart.ChartServiceImpl.generateChart(ChartServiceImpl.java:316)
    at
org.sakaiproject.sitestats.tool.wicket.widget.WidgetTabTemplate$1.getChartImage(WidgetTabTemplate.java:164)
    at
org.sakaiproject.sitestats.tool.wicket.widget.WidgetTabTemplate$1.getImageData(WidgetTabTemplate.java:157)
    at
org.sakaiproject.sitestats.tool.wicket.components.AjaxLazyLoadImage.renderImage(AjaxLazyLoadImage.java:153)
    at
org.sakaiproject.sitestats.tool.wicket.components.AjaxLazyLoadImage$4.respond(AjaxLazyLoadImage.java:255)
    at
org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:654)
    at sun.reflect.GeneratedMethodAccessor6700.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at
org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258)
    at
org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241)
    at
org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:250)
    at
org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:236)
    at
org.apache.wicket.core.request.handler.RequestSettingRequestHandler.respond(RequestSettingRequestHandler.java:78)
    at
org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:890)
    at
org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
    at
org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:261)
    at
org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:218)
    at
org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:289)
    at
org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:259)
    at
org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:201)
    at
org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:137)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:418)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
    at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)
    at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:356)
    at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)
    at
org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:513)
    at
org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1490)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doSite(SiteHandler.java:469)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:266)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:154)
    at
org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:856)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:467)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479)
    at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
    at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

(...)

24-may-2020 12:07:35.849 WARN [ajp-nio-18009-exec-152]
org.apache.wicket.page.PageAccessSynchronizer.lockPage Thread
'ajp-nio-18009-exec-152' failed to acquire lock to page with id '0',
attempted for 1 minute out of allowed 1 minute. The thread that holds
the lock has name 'ajp-nio-18009-exec-147'.
24-may-2020 12:07:35.851 WARN [ajp-nio-18009-exec-152]
org.apache.wicket.page.PageAccessSynchronizer.dumpSingleThread
"ajp-nio-18009-exec-147" daemon prio=5 tid=93978 state=RUNNABLE
org.apache.wicket.util.lang.Threads$ThreadDump
    at sun.java2d.loops.DrawGlyphListAA.DrawGlyphListAA(Native Method)
    at sun.java2d.pipe.AATextRenderer.drawGlyphList(AATextRenderer.java:41)
    at sun.java2d.pipe.GlyphListPipe.drawString(GlyphListPipe.java:72)
    at sun.java2d.SunGraphics2D.drawString(SunGraphics2D.java:2957)
    at
org.jfree.text.TextUtilities.drawAlignedString(TextUtilities.java:366)
    at org.jfree.chart.axis.PeriodAxis.drawTickLabels(PeriodAxis.java:870)
    at org.jfree.chart.axis.PeriodAxis.draw(PeriodAxis.java:632)
    at org.jfree.chart.plot.XYPlot.drawAxes(XYPlot.java:3007)
    at org.jfree.chart.plot.XYPlot.draw(XYPlot.java:2531)
    at org.jfree.chart.JFreeChart.draw(JFreeChart.java:1175)
    at org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1349)
    at org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1329)
    at org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1314)
    at
org.sakaiproject.sitestats.impl.chart.ChartServiceImpl.generateTimeSeriesChart(ChartServiceImpl.java:654)
    at
org.sakaiproject.sitestats.impl.chart.ChartServiceImpl.generateChart(ChartServiceImpl.java:316)
    at
org.sakaiproject.sitestats.tool.wicket.widget.WidgetTabTemplate$1.getChartImage(WidgetTabTemplate.java:164)
    at
org.sakaiproject.sitestats.tool.wicket.widget.WidgetTabTemplate$1.getImageData(WidgetTabTemplate.java:157)
    at
org.sakaiproject.sitestats.tool.wicket.components.AjaxLazyLoadImage.renderImage(AjaxLazyLoadImage.java:153)
    at
org.sakaiproject.sitestats.tool.wicket.components.AjaxLazyLoadImage$4.respond(AjaxLazyLoadImage.java:255)
    at
org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:654)
    at sun.reflect.GeneratedMethodAccessor6700.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at
org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258)
    at
org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241)
    at
org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:250)
    at
org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:236)
    at
org.apache.wicket.core.request.handler.RequestSettingRequestHandler.respond(RequestSettingRequestHandler.java:78)
    at
org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:890)
    at
org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
    at
org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:261)
    at
org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:218)
    at
org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:289)
    at
org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:259)
    at
org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:201)
    at
org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:137)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:418)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
    at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)
    at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:356)
    at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)
    at
org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:513)
    at
org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1490)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doSite(SiteHandler.java:469)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:266)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:154)
    at
org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:856)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:467)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479)
    at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
    at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

24-may-2020 12:07:35.852 ERROR [ajp-nio-18009-exec-152]
org.apache.wicket.request.RequestHandlerStack.detach Error detaching
RequestHandler
org.apache.wicket.page.CouldNotLockPageException: Could not lock page 0.
Attempt lasted 1 minute
    at
org.apache.wicket.page.PageAccessSynchronizer.lockPage(PageAccessSynchronizer.java:167)
    at
org.apache.wicket.page.PageAccessSynchronizer$2.getPage(PageAccessSynchronizer.java:245)
    at
org.apache.wicket.DefaultMapperContext.getPageInstance(DefaultMapperContext.java:148)
    at
org.apache.wicket.core.request.handler.PageProvider.getStoredPage(PageProvider.java:296)
    at
org.apache.wicket.core.request.handler.PageProvider.isNewPageInstance(PageProvider.java:208)
    at
org.apache.wicket.core.request.handler.PageProvider.getPageParameters(PageProvider.java:186)
    at
org.apache.wicket.core.request.handler.logger.PageLogData.<init>(PageLogData.java:51)
    at
org.apache.wicket.core.request.handler.logger.ListenerInterfaceLogData.<init>(ListenerInterfaceLogData.java:56)
    at
org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.detach(ListenerInterfaceRequestHandler.java:125)
    at
org.apache.wicket.core.request.handler.RequestSettingRequestHandler.detach(RequestSettingRequestHandler.java:89)
    at
org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.detach(RequestCycle.java:901)
    at
org.apache.wicket.request.RequestHandlerStack.detach(RequestHandlerStack.java:180)
    at
org.apache.wicket.request.cycle.RequestCycle.onDetach(RequestCycle.java:636)
    at
org.apache.wicket.request.cycle.RequestCycle.detach(RequestCycle.java:589)
    at
org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:293)
    at
org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:259)
    at
org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:201)
    at
org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:418)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
    at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)
    at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:356)
    at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)
    at
org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:513)
    at
org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1490)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doSite(SiteHandler.java:469)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:266)
    at
org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:154)
    at
org.sakaiproject.portal.charon.handlers.WorksiteHandler.doPost(WorksiteHandler.java:69)
    at
org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1171)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:467)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479)
    at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
    at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)


--
Daniel Merino Echeverría
daniel...@unavarra.es
Gestor de E-learning - Centro Superior de Innovación Educativa.
Tfno: 948-168489 - Universidad Pública de Navarra.

[Paul Lukasewych]

We've seen this a couple of times in production over the years, but is
is very rare for us. We've tracked it down to something related to
drawing the charts as it appears to get stuck here:

sun.java2d.loops.MaskFill.FillAAPgram(Native Method)
sun.java2d.pipe.AlphaColorPipe.fillParallelogram(AlphaColorPipe.java:75
)
sun.java2d.pipe.PixelToParallelogramConverter.drawGeneralLine(PixelToPa
rallelogramConverter.java:287)
sun.java2d.pipe.PixelToParallelogramConverter.draw(PixelToParallelogram
Converter.java:139)
sun.java2d.SunGraphics2D.draw(SunGraphics2D.java:2497)
org.jfree.chart.axis.PeriodAxis.drawTickLabels(PeriodAxis.java:879)
org.jfree.chart.axis.PeriodAxis.draw(PeriodAxis.java:632)
org.jfree.chart.plot.XYPlot.drawAxes(XYPlot.java:3007)
org.jfree.chart.plot.XYPlot.draw(XYPlot.java:2531)
org.jfree.chart.JFreeChart.draw(JFreeChart.java:1175)
org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1349)
org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1329)
org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1314)
org.sakaiproject.sitestats.impl.chart.ChartServiceImpl.generateTimeSeri
esChart(ChartServiceImpl.java:654)
org.sakaiproject.sitestats.impl.chart.ChartServiceImpl.generateChart(Ch
artServiceImpl.java:316)

Unfortunately we don't know how to reproduce it.

--
Paul Lukasewych
Applications Development Team
Western Technology Services
Western University

> org.sakaiproject.portal.charon.handlers.WorksiteHandler.doPost(Worksi
> teHandler.java:69)
> at
> org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(Skinnable

> CharonPortal.java:1171)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
> icationFilterChain.java:231)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF

[Mark Triggs]

Hi Daniel and Paul,

NYU hit this issue as well. We were never able to reproduce it, but we upgraded JFreeChart and haven’t had the issue occur since:

https://github.com/NYUeServ/sakai11/commit/f1f5dba5b7dc2bb19aec2d5f60097731b67ceb21

The upgrade process was a little fiddly: some of the JFreeChart APIs had changed, and the option for 3D rendered charts no longer exists.

If your users are connoisseurs of sitestats it would probably be worth checking that the user-visible changes are acceptable. We were just pleased to not have servers falling out of the sky :)

Cheers,
Mark

Paul Lukasewych <pluk...@uwo.ca> writes:

We’ve seen this a couple of times in production over the years, but is is very rare for us. We’ve tracked it down to something related to drawing the charts as it appears to get stuck here:

 sun.java2d.loops.MaskFill.FillAAPgram(Native Method)
 sun.java2d.pipe.AlphaColorPipe.fillParallelogram(AlphaColorPipe.java:75
 )
 sun.java2d.pipe.PixelToParallelogramConverter.drawGeneralLine(PixelToPa
 rallelogramConverter.java:287)
 sun.java2d.pipe.PixelToParallelogramConverter.draw(PixelToParallelogram
 Converter.java:139)
 sun.java2d.SunGraphics2D.draw(SunGraphics2D.java:2497)
 org.jfree.chart.axis.PeriodAxis.drawTickLabels(PeriodAxis.java:879)
 org.jfree.chart.axis.PeriodAxis.draw(PeriodAxis.java:632)
 org.jfree.chart.plot.XYPlot.drawAxes(XYPlot.java:3007)
 org.jfree.chart.plot.XYPlot.draw(XYPlot.java:2531)
 org.jfree.chart.JFreeChart.draw(JFreeChart.java:1175)
 org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1349)
 org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1329)
 org.jfree.chart.JFreeChart.createBufferedImage(JFreeChart.java:1314)
 org.sakaiproject.sitestats.impl.chart.ChartServiceImpl.generateTimeSeri
 esChart(ChartServiceImpl.java:654)
 org.sakaiproject.sitestats.impl.chart.ChartServiceImpl.generateChart(Ch
 artServiceImpl.java:316)

Unfortunately we don’t know how to reproduce it.

On Wed, 2020-05-27 at 17:36 +0200, Daniel Merino wrote:

Dear community,

we have been affected by a critical issue related to Sitestats.

One teacher tried to generate a report of the use of Resources in his site, the report didn’t finish, an AJAX error was shown in the browser console and one exception was written in the logs.

[…]

The worst thing of this was that one of the four CPU cores of the server reached a 100% of usage and didn’t recover. Pasting the URL of the console again in the browser caused the same issue and another core jumped to 100% of usage. With two more pastes we would have killed the server!

-- 
Mark Triggs
<ma...@dishevelled.net>

[Jesus Maria Mendez Perez]

Thanks!

Why on trunk is only updated to 1.0.19?

https://github.com/sakaiproject/sakai/blame/master/sitestats/pom.xml#L22 

Should we updated to 1.5.0?

Regards Jesus. 

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/87imgh2ego.fsf%40dishevelled.net.

[Matthew Jones]

It looks like there was an update in Sakai 21 from 1.0.5 to 1.0.19:  https://jira.sakaiproject.org/browse/SAK-43292. It's possible that update alone may also solved this problem and there were a few minor code changes with that. Looking through the changelog there were a ton of fixes between 1.0.5 and 1.0.19 alone, many performance related and there weren't many between 1.0.19 and 1.5.0. 

We'll have to get this tested somehow and merged back into 20 and 19.

I created  https://jira.sakaiproject.org/browse/SAK-43708 to update to 1.5.0 and attached Mark's patch there. There are a few conflicts, possible some because of SAK-43292 so it'll have to be manually applied and fully tested. It sounds like a few graph types were dropped.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/CAKo8AAp32UFxgsGwB-%2BFeGXa6o4P0Uoe6PCb4wRncL%3D0ajOU8A%40mail.gmail.com.

[David Horwitz]

cherry-pick/ pull requested:

https://github.com/sakaiproject/sakai/pull/8252

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/CACqbowvuEK%2BMzSeCs3j1-s2ZEpiLj_w2D-yszrv5WPn%3DoDqhxw%40mail.gmail.com.

[Daniel Merino]

Thanks everybody for all this valuable information!

In this PR the version of JFreeChart replaced is 1.0.19. So must we assume that the issue still happens with JFreeChart 1.0.19 and it is only fixed by 1.5.0?

Best regards.

El 28/5/20 a las 15:48, David Horwitz escribió:

[Matthew Jones]

I don't think anyone knows if it's a problem in 1.0.19 since that fix is only in Sakai 21 and every previous version still is on 1.0.5.

Looking through the Changelog it seems likely fixed by 1.0.19 but looks like Dave got a PR up for the update to 1.5.0 that is under review. 1.5.0 was the version immediately after 1.0.19.

[David Horwitz]

FYI Evaluation System:

https://jira.sakaiproject.org/browse/EVALSYS-1593 

https://github.com/sakaicontrib/evaluation/pull/150

[Daniel Merino]

We have done testing of the 1.0.19/1.5.0 JFreeChart library and the graphics are not displaying well in 12.6.

For example, in the graph of visits and unique visitors in a test site, we can see this chart with 1.0.5 (without patch applied):



And with 1.5.0 it looks like this:



As you can see, only the two first bars are displayed in the graph. This happens in every bar graph, no matter how many values there are only the two first bars are painted.

Mark Triggs, you are the only one that has run this patch in production. Did you need to apply more changes after upgrading the JFreechart?

Thanks.
Best regards.

El 28/5/20 a las 17:04, Matthew Jones escribió:

[Mark Triggs]

Hi Daniel,

That’s the only patch we have applied, but NYU’s installation is 12.5 and has some modifications to sitestats from much earlier on that I’m not really familiar with, so our installation may not be representative of what you’d see in 12.6.

It’s definitely possible this is just a bug with the upgrade patch that I missed. Sitestats is pretty lightly used, and the priority at the time was to stop the servers crashing.

Mark

Daniel Merino <daniel...@unavarra.es> writes:

We have done testing of the 1.0.19/1.5.0 JFreeChart library and the graphics are not displaying well in 12.6.

For example, in the graph of visits and unique visitors in a test site, we can see this chart with 1.0.5 (without patch applied):

[…]

And with 1.5.0 it looks like this:

[…]

As you can see, only the two first bars are displayed in the graph. This happens in every bar graph, no matter how many values there are only the two first bars are painted.

Mark Triggs, you are the only one that has run this patch in production. Did you need to apply more changes after upgrading the JFreechart?

-- 
Mark Triggs
<ma...@dishevelled.net>

[Daniel Merino]

Thanks Mark,

it is not possible to test SAK-43708 in a nightly server because the issue affects to TimeSeries graph bars and nightly servers reset everyday.

So I don't know if this is happening only to us or it is a regression of SAK-43708. If somebody has applied SAK-43708 and can take a look, I would love to know if you can see correctly your Time Series bars in Sitestats.

Thanks in advance.

El 3/6/20 a las 1:13, Mark Triggs escribió:

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/87k10pvz7g.fsf%40dishevelled.net.

[Mark Triggs]

Hi Daniel,

Just doing some testing locally with some made up data, I saw the same issue you showed (just two bars visible in the chart).

I noticed that the Y axis was always set to 0..1, and found some code in ChartServiceImpl that was turning off auto range calculations. I commented that out:

https://tsp.nz/d/f6dd4a8f3cf83c2b6429a42eac1966e2d92980d1.txt

and now I see the full set of bars:

https://tsp.nz/s/940ebb52e3b4150674fd6e17d236b28362ef614e.jpg

Are you able to see if that change works for you too?

Cheers,

To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org <mailto:sakai-dev+...@apereo.org>. To view this discussion on the web visit
https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/87k10pvz7g.fsf%40dishevelled.net <https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/87k10pvz7g.fsf%40dishevelled.net?utm_medium=email&utm_source=footer>.

[Daniel Merino]

Mark, thanks a lot, it works here too.

Now we have Sitestats with 1.5.0 and everything is working fine.

Thanks again.
Best regards.

El 3/6/20 a las 12:06, Mark Triggs escribió:

[Sam Ottenhoff]

Great. Can you put in a PR Daniel?

To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/311bebe3-c4d9-d0ef-6e57-be6d81367164%40unavarra.es.

[Daniel Merino]

Sure. I have done a new JIRA SAK-43752 and submitted a PR.
Thanks again.

El 3/6/20 a las 20:04, Sam Ottenhoff escribió: