Querying in multi-thread environment issue

[Andrey Yesyev]

Hi there,

I'm issuing a number of queries to DB at the same time from different threads. Times to times I got this  exception

com.orientechnologies.orient.enterprise.channel.binary.OResponseProcessingException: Exception during response processing.

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:264)

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:231)

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:202)

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:113)

        at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1951)

        at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:982)

        at com.orientechnologies.orient.client.remote.OStorageRemoteThread.command(OStorageRemoteThread.java:431)

        at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:69)

        at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:82)

        at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:29)

        at com.tinkerpop.blueprints.impls.orient.OrientGraphCommand.execute(OrientGraphCommand.java:28)

        at com.ibm.forensics.controllers.EpController.queryRelations(EpController.java:66)

        at com.ibm.forensics.controllers.EpController.grid(EpController.java:40)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

        at java.lang.reflect.Method.invoke(Method.java:618)

        at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)

        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)

        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)

        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:746)

        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:687)

        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)

        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)

        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)

        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:915)

        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:811)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:796)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)

        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)

        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)

        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)

        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)

        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)

        at java.lang.Thread.run(Thread.java:804)

Caused by: com.orientechnologies.orient.core.exception.OCommandExecutionException: Error on execution of command: sql.select count(*) from (traverse both(RELATED) from #21:7 while $depth < 2 ) where @class='IPAddress' and @rid <> #21:7

        at com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:102)

        at com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:85)

        at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:69)

        at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:82)

        at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:29)

        at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1320)

        at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:330)

        at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:125)

        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45)

Caused by: java.lang.NullPointerException

Looks like thread race to me.

Any ideas?

-Andrey

[Andrey Lomakin]

HI Andrey,

I see last line is:

"Caused by: java.lang.NullPointerException"

What is stackrace of this cause ?

--

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[Andrey Yesyev]

Hi Andrey,

Unfortunately the stacktrace ends there.

I'm trying now to get more information on this.

-Andrey

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/SUEqUvWFdJA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrey Lomakin]

Thank you !

Looking forward for it.

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrey Yesyev]

Ok, here is something new

com.orientechnologies.orient.enterprise.channel.binary.OResponseProcessingException: Exception during response processing.

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:264)

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:231)

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:202)

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:113)

        at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1951)

        at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:982)

        at com.orientechnologies.orient.client.remote.OStorageRemoteThread.command(OStorageRemoteThread.java:431)

        at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:69)

        at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:82)

        at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:29)

        at com.tinkerpop.blueprints.impls.orient.OrientGraphCommand.execute(OrientGraphCommand.java:28)

        at com.ibm.forensics.controllers.EpController.queryRelations(EpController.java:83)

Caused by: com.orientechnologies.orient.core.exception.OTransactionException: Invalid state of the transaction. The transaction must be begun.

        at com.orientechnologies.orient.core.tx.OTransactionRealAbstract.checkTransaction(OTransactionRealAbstract.java:342)

        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.addRecord(OTransactionOptimistic.java:314)

        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:291)

        at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:203)

        at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:39)

        at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:296)

        at com.tinkerpop.blueprints.impls.orient.OrientVertexIterator.createWrapper(OrientVertexIterator.java:31)

        at com.tinkerpop.blueprints.impls.orient.OrientVertexIterator.createWrapper(OrientVertexIterator.java:13)

        at com.orientechnologies.orient.core.iterator.OLazyWrapperIterator.hasNext(OLazyWrapperIterator.java:75)

        at com.orientechnologies.common.collection.OMultiCollectionIterator.hasNext(OMultiCollectionIterator.java:74)

        at com.orientechnologies.orient.core.command.traverse.OTraverseMultiValueDepthFirstProcess.process(OTraverseMultiValueDepthFirstProcess.java:33)

        at com.orientechnologies.orient.core.command.traverse.OTraverseMultiValueDepthFirstProcess.process(OTraverseMultiValueDepthFirstProcess.java:24)

        at com.orientechnologies.orient.core.command.traverse.OTraverseFieldProcess.process(OTraverseFieldProcess.java:66)

        at com.orientechnologies.orient.core.command.traverse.OTraverseFieldProcess.process(OTraverseFieldProcess.java:25)

        at com.orientechnologies.orient.core.command.traverse.OTraverse.next(OTraverse.java:101)

        at com.orientechnologies.orient.core.command.traverse.OTraverse.hasNext(OTraverse.java:71)

        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLTraverse.hasNext(OCommandExecutorSQLTraverse.java:163)

        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:343)

        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:313)

        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:57)

        at com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:96)

        at com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:85)

        at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:69)

        at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:82)

        at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:29)

        at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1320)

        at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:330)

        at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:125)

        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45)

[Andrey Lomakin]

Aha, I was hunting this issue for a months.

Is it reproducible ? 

[Andrey Yesyev]

Yes.

[Andrey Yesyev]

I can provide anything you need to fix it.

[Andrey Yesyev]

Any updates on this?

[Andrey Lomakin]

Hi Andrey,

Not yet, could you create issue in bug tracker, so you will track it and will not ping me :-))) .

[Andrey Yesyev]

Hi Andrey,

I guess I can :)

[Andrey Yesyev]

Issue is opened https://github.com/orientechnologies/orientdb/issues/2141