Problems with mysql transactions: Deadlock found when trying to get lock; try restarting transaction
832 views
Skip to first unread message
Miguel Escriva Gregori
Apr 12, 2018, 6:25:00 AM4/12/18
to us...@opencast.org
Hi,
I’m getting these errors in our Opencast 3.2 installation.
Any ideas to solve this?
Thanks,
Miguel
Logs from the presentation node:
2018-04-12 12:01:27,321 | WARN | (JavaLog:300) -
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.4.v20160829-44060b6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
Error Code: 1213
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:331)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicCommitTransaction(DatabaseAccessor.java:447)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.commitTransaction(DatasourceAccessor.java:405)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.commitTransaction(DatabaseAccessor.java:429)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicCommitTransaction(AbstractSession.java:814)
at org.eclipse.persistence.sessions.server.ClientSession.basicCommitTransaction(ClientSession.java:182)
at org.eclipse.persistence.internal.sessions.AbstractSession.commitTransaction(AbstractSession.java:1018)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransaction(UnitOfWorkImpl.java:1600)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitTransaction(RepeatableWriteUnitOfWork.java:650)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1615)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
at org.opencastproject.usertracking.impl.UserTrackingServiceImpl.addUserTrackingEvent(UserTrackingServiceImpl.java:203)
at org.opencastproject.usertracking.endpoint.UserTrackingRestService.addFootprint(UserTrackingRestService.java:323)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_162]
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)[137:org.apache.cxf.cxf-core:3.1.7]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)[137:org.apache.cxf.cxf-core:3.1.7]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189)[138:org.apache.cxf.cxf-rt-frontend-jaxrs:3.1.7]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)[138:org.apache.cxf.cxf-rt-frontend-jaxrs:3.1.7]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)[137:org.apache.cxf.cxf-core:3.1.7]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)[137:org.apache.cxf.cxf-core:3.1.7]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)[137:org.apache.cxf.cxf-core:3.1.7]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)[137:org.apache.cxf.cxf-core:3.1.7]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:254)[144:org.apache.cxf.cxf-rt-transports-http:3.1.7]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)[144:org.apache.cxf.cxf-rt-transports-http:3.1.7]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)[144:org.apache.cxf.cxf-rt-transports-http:3.1.7]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)[144:org.apache.cxf.cxf-rt-transports-http:3.1.7]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)[144:org.apache.cxf.cxf-rt-transports-http:3.1.7]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299)[144:org.apache.cxf.cxf-rt-transports-http:3.1.7]
at org.opencastproject.kernel.rest.RestPublisher$RestServlet.handleRequest(RestPublisher.java:535)[64:matterhorn-kernel:3.2.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:223)[144:org.apache.cxf.cxf-rt-transports-http:3.1.7]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)[30:javax.servlet-api:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274)[144:org.apache.cxf.cxf-rt-transports-http:3.1.7]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.opencastproject.kernel.rest.JsonpFilter.doFilter(JsonpFilter.java:113)[64:matterhorn-kernel:3.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.opencastproject.security.urlsigning.filter.UrlSigningFilter.doFilter(UrlSigningFilter.java:85)[96:matterhorn-urlsigning-verifier-service-impl:3.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.opencastproject.kernel.filter.proxy.TransparentProxyFilter.doFilter(TransparentProxyFilter.java:63)[64:matterhorn-kernel:3.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.opencastproject.kernel.security.RemoteUserAndOrganizationFilter.doFilter(RemoteUserAndOrganizationFilter.java:247)[64:matterhorn-kernel:3.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.opencastproject.kernel.rest.CurrentJobFilter.doFilter(CurrentJobFilter.java:87)[64:matterhorn-kernel:3.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:139)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:98)[257:org.springframework.web:3.1.4.RELEASE]
at org.springframework.security.oauth.provider.OAuthProviderProcessingFilter.doFilter(OAuthProviderProcessingFilter.java:173)[64:matterhorn-kernel:3.2.0]
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:102)[257:org.springframework.web:3.1.4.RELEASE]
at org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:115)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:102)[257:org.springframework.web:3.1.4.RELEASE]
at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:82)[257:org.springframework.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:65)[83:matterhorn-security-cas-client-wrapper:3.2.0]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)[255:org.springframework.security.web:3.1.4.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)[255:org.springframework.security.web:3.1.4.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)[255:org.springframework.security.web:3.1.4.RELEASE] at org.opencastproject.kernel.security.SecurityFilter.doFilter(SecurityFilter.java:124)[64:matterhorn-kernel:3.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908] at org.opencastproject.kernel.security.OrganizationFilter.doFilter(OrganizationFilter.java:135)[64:matterhorn-kernel:3.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908] at org.opencastproject.kernel.filter.https.HttpsFilter.doFilter(HttpsFilter.java:64)[64:matterhorn-kernel:3.2.0] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.opencastproject.kernel.rest.CleanSessionsFilter.doFilter(CleanSessionsFilter.java:89)[64:matterhorn-kernel:3.2.0] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)[213:org.eclipse.jetty.servlet:9.2.19.v20160908] at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)[235:org.ops4j.pax.web.pax-web-jetty:4.3.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)[212:org.eclipse.jetty.server:9.2.19.v20160908] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)[211:org.eclipse.jetty.security:9.2.19.v20160908]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)[212:org.eclipse.jetty.server:9.2.19.v20160908] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)[212:org.eclipse.jetty.server:9.2.19.v20160908]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:287)[235:org.ops4j.pax.web.pax-web-jetty:4.3.0] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)[213:org.eclipse.jetty.servlet:9.2.19.v20160908]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)[212:org.eclipse.jetty.server:9.2.19.v20160908] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)[212:org.eclipse.jetty.server:9.2.19.v20160908]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)[212:org.eclipse.jetty.server:9.2.19.v20160908] at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)[235:org.ops4j.pax.web.pax-web-jetty:4.3.0]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)[212:org.eclipse.jetty.server:9.2.19.v20160908] at org.eclipse.jetty.server.Server.handle(Server.java:499)[212:org.eclipse.jetty.server:9.2.19.v20160908] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)[212:org.eclipse.jetty.server:9.2.19.v20160908]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)[212:org.eclipse.jetty.server:9.2.19.v20160908] at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)[204:http://org.eclipse.jetty.io:9.2.19.v20160908]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)[215:org.eclipse.jetty.util:9.2.19.v20160908] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)[215:org.eclipse.jetty.util:9.2.19.v20160908]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_162]
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)[:1.8.0_162]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)[:1.8.0_162]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.8.0_162]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)[:1.8.0_162]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:951)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3970)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3906)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1614)
at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:981)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicCommitTransaction(DatabaseAccessor.java:439)
... 109 more
2018-04-12 12:01:27,389 | WARN | (WebApplicationExceptionMapper:72) - javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
at org.opencastproject.usertracking.endpoint.UserTrackingRestService.addFootprint(UserTrackingRestService.java:326)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:254)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299)
at org.opencastproject.kernel.rest.RestPublisher$RestServlet.handleRequest(RestPublisher.java:535)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:223)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at org.opencastproject.kernel.rest.JsonpFilter.doFilter(JsonpFilter.java:113)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.opencastproject.security.urlsigning.filter.UrlSigningFilter.doFilter(UrlSigningFilter.java:85)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.opencastproject.kernel.filter.proxy.TransparentProxyFilter.doFilter(TransparentProxyFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.opencastproject.kernel.security.RemoteUserAndOrganizationFilter.doFilter(RemoteUserAndOrganizationFilter.java:247)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.opencastproject.kernel.rest.CurrentJobFilter.doFilter(CurrentJobFilter.java:87)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:139)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:98)
at org.springframework.security.oauth.provider.OAuthProviderProcessingFilter.doFilter(OAuthProviderProcessingFilter.java:173)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:102)
at org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:115)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:102)
at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:82)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:65)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.opencastproject.kernel.security.SecurityFilter.doFilter(SecurityFilter.java:124)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.opencastproject.kernel.security.OrganizationFilter.doFilter(OrganizationFilter.java:135)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.opencastproject.kernel.filter.https.HttpsFilter.doFilter(HttpsFilter.java:64)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.opencastproject.kernel.rest.CleanSessionsFilter.doFilter(CleanSessionsFilter.java:89)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:287)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.opencastproject.usertracking.api.UserTrackingException: javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.4.v20160829-44060b6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
Error Code: 1213
at org.opencastproject.usertracking.impl.UserTrackingServiceImpl.addUserTrackingEvent(UserTrackingServiceImpl.java:209)
at org.opencastproject.usertracking.endpoint.UserTrackingRestService.addFootprint(UserTrackingRestService.java:323)
... 96 more
Caused by: javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.4.v20160829-44060b6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
Error Code: 1213
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159)
at org.opencastproject.usertracking.impl.UserTrackingServiceImpl.addUserTrackingEvent(UserTrackingServiceImpl.java:203)
... 97 more
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.4.v20160829-44060b6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
Error Code: 1213
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:331)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicCommitTransaction(DatabaseAccessor.java:447)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.commitTransaction(DatasourceAccessor.java:405)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.commitTransaction(DatabaseAccessor.java:429)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicCommitTransaction(AbstractSession.java:814)
at org.eclipse.persistence.sessions.server.ClientSession.basicCommitTransaction(ClientSession.java:182)
at org.eclipse.persistence.internal.sessions.AbstractSession.commitTransaction(AbstractSession.java:1018)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransaction(UnitOfWorkImpl.java:1600)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitTransaction(RepeatableWriteUnitOfWork.java:650)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1615)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
... 98 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:951)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3970)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3906)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1614)
at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:981)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicCommitTransaction(DatabaseAccessor.java:439)
... 109 more
Miguel
Greg Logan
Apr 17, 2018, 10:26:12 PM4/17/18
to Opencast Users
Hi Miguel,
I'm not terribly familiar with the url signing service since I don't use it, but what are you doing to trigger this? Is it repeatable? Does it happen on newer versions?
G
--
You received this message because you are subscribed to the Google Groups "Opencast Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.
Miguel Escriva Gregori
Apr 18, 2018, 6:56:42 AM4/18/18
to us...@opencast.org
Hi Greg,
I’m not using the url signing service..
It seems that the problem is with the UserTrackingService:
2018-04-12 12:01:27,321 | WARN | (JavaLog:300) -
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.4.v20160829-44060b6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
Error Code: 1213
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:331)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicCommitTransaction(DatabaseAccessor.java:447)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.commitTransaction(DatasourceAccessor.java:405)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.commitTransaction(DatabaseAccessor.java:429)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicCommitTransaction(AbstractSession.java:814)
at org.eclipse.persistence.sessions.server.ClientSession.basicCommitTransaction(ClientSession.java:182)
at org.eclipse.persistence.internal.sessions.AbstractSession.commitTransaction(AbstractSession.java:1018)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransaction(UnitOfWorkImpl.java:1600)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitTransaction(RepeatableWriteUnitOfWork.java:650)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1615)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
at org.opencastproject.usertracking.impl.UserTrackingServiceImpl.addUserTrackingEvent(UserTrackingServiceImpl.java:203)
at org.opencastproject.usertracking.endpoint.UserTrackingRestService.addFootprint(UserTrackingRestService.java:323)
…
The next error:
2018-04-12 12:01:27,389 | WARN | (WebApplicationExceptionMapper:72) - javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
at org.opencastproject.usertracking.endpoint.UserTrackingRestService.addFootprint(UserTrackingRestService.java:326)
…
In the nginx access logs I found this:
xx.yy.zz.ww - - [18/Apr/2018:11:49:41 +0200] "GET /usertracking/?_method=PUT&id=d66287e0-05d3-11e8-b07a-99dcae272a47&type=SEEK&in=3687&out=3687&playing=false&_=1524030811998 HTTP/1.1" 500 0 ...
I tested to make that call manually, and it worked. So. I does not know why I’m having these errors in the user tracking service.
Any Idea?
Thanks,
Miguel
Karen Dolan
Apr 18, 2018, 9:10:16 AM4/18/18
to us...@opencast.org
Hi Miguel,
Look into the log lines, before the Deadlock, error to see if you can tell if another service was also trying to persist or delete from similar tables/rows/objects as the addUserTrackingEvent().
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)at org.opencastproject.usertracking.impl.UserTrackingServiceImpl.addUserTrackingEvent(UserTrackingServiceImpl.java:203)at org.opencastproject.usertracking.endpoint.UserTrackingRestService.addFootprint(UserTrackingRestService.java:32
- Karen
Greg Logan
Apr 18, 2018, 11:39:04 AM4/18/18
to Opencast Users
Hi Miguel,
Ah, I misread the stack trace.
What kind of load are you looking at on your database? How many clients do you have playing videos roughly? The UserTracking service keeps track of which users have watched which parts of each video, so I suppose under enough load there might be some edge case that's being triggered in the JPA library. I don't think the service itself has seen any updates in newer versions of Opencast, but I'm sure the underlying libraries have. Can you try with 3.5?
G
Reply all
Reply to author
Forward
0 new messages