Error Processing Update

[Andreas S. Rath (teddius)]

Dear OpenAnzo Group,

I guess there are still issues with the update mechanism on the server
side. After some time working and issues updateRepository calls I get
following exceptions on the openanzo-server. When one update call is
failing then no triples can be added to this graph any more until the
server is restarted.

The version I am using in the OpenAnzo Snapshot 2008-11-07 with a DB2
Database and a replicaGraph that is updated every 30 seconds with
about 400 triples. While updating new updates will be issued and
triples are added and deleted from the graph by multiple threads.

Would you please explain me in more detail what the error message
means and when it will be thrown on the server side? In which ways can
I avoid this exception? I have tried to produce a unit test for that,
but I couldn't up to now.

Thanks for you help,
Andreas

Here are the exceptions:

2008-11-11 09:44:05,414 78 [ERROR] [http://openanzo.org/
service/CombusUpdateServiceListener] [OpName=update]
[OpId=dee71648-74de-4a09-9194-edcbd029ece6] [OpUser=ldap://
uid=default,ou=users,dc=openanzo,dc=org] [RunAs=] [UserDesc=]
org.openanzo.datasource.update.ServerUpdatesProcessor - Error
Processing Update:
org.openanzo.jdbc.utils.RdbException: ErrorCode[256:262181]
[RDB_ERROR] Failed to execute an sql operation:
[insertUncommittedReferences] SQL:[INSERT INTO {1} (ID,TRANSACTIONID)
SELECT {0}.ID,{2} FROM {0} WHERE {0}.TYPE IN (1,3)] Params:[]
Templates:
[bulkResolutionTableName=SESSION.ID_TMP,lockedIdsTable=ANZO_USED_IDS,transactionId=3581595302490874083]
Root Cause: com.ibm.db2.jcc.b.SqlException DB2 SQL Error:
SQLCODE=-805, SQLSTATE=51002, SQLERRMC=NULLID.SYSLH104
0X5359534C564C3031, DRIVER=3.50.152
at
org.openanzo.jdbc.container.sql.NodeSQL.insertUncommittedReferences(NodeSQL.java:
5071)
at
org.openanzo.jdbc.layout.NodeURILayout.resolveStoredNodes(NodeURILayout.java:
287)
at
org.openanzo.jdbc.layout.NodeLayoutCacheProxy.resolveStoredNodes(NodeLayoutCacheProxy.java:
173)
at
org.openanzo.jdbc.layout.CompositeNodeLayout.resolveStoredNodesFast(CompositeNodeLayout.java:
513)
at
org.openanzo.jdbc.layout.CompositeNodeLayout.resolveStoredNodes(CompositeNodeLayout.java:
425)
at
org.openanzo.datasource.nodecentric.DispatchingDeltaQuadStore.preCommit(DispatchingDeltaQuadStore.java:
168)
at
org.openanzo.datasource.nodecentric.DispatchingDeltaQuadStore.update(DispatchingDeltaQuadStore.java:
355)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.handleTransactionEnd(ServerUpdatesProcessor.java:
412)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.handleTransaction(ServerUpdatesProcessor.java:
312)
at org.openanzo.datasource.update.ServerUpdatesProcessor
$1.handleTransaction(ServerUpdatesProcessor.java:252)
at
org.openanzo.services.serialization.XMLUpdatesReader.parseUpdateTransactions(XMLUpdatesReader.java:
149)
at
org.openanzo.services.serialization.XMLUpdatesReader.read(XMLUpdatesReader.java:
82)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.read(ServerUpdatesProcessor.java:
247)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.update(ServerUpdatesProcessor.java:
202)
at
org.openanzo.datasource.nodecentric.NodeCentricUpdateService.updateInternal(NodeCentricUpdateService.java:
88)
at
org.openanzo.datasource.services.BaseUpdateService.update(BaseUpdateService.java:
139)
at
org.openanzo.combus.listener.CombusUpdateServiceListener.handleMessage(CombusUpdateServiceListener.java:
134)
at
org.openanzo.combus.endpoint.BaseServiceListener.processMessage(BaseServiceListener.java:
319)
at org.openanzo.combus.endpoint.BaseServiceListener.access
$000(BaseServiceListener.java:57)
at org.openanzo.combus.endpoint.BaseServiceListener
$ProcessThread.run(BaseServiceListener.java:262)
Caused by: com.ibm.db2.jcc.b.SqlException: DB2 SQL Error:
SQLCODE=-805, SQLSTATE=51002, SQLERRMC=NULLID.SYSLH104
0X5359534C564C3031, DRIVER=3.50.152
at com.ibm.db2.jcc.b.wc.a(wc.java:55)
at com.ibm.db2.jcc.b.wc.a(wc.java:126)
at com.ibm.db2.jcc.b.tk.c(tk.java:1901)
at com.ibm.db2.jcc.b.tk.d(tk.java:1889)
at com.ibm.db2.jcc.b.tk.b(tk.java:1356)
at com.ibm.db2.jcc.t4.db.h(db.java:219)
at com.ibm.db2.jcc.t4.db.b(db.java:45)
at com.ibm.db2.jcc.t4.t.b(t.java:38)
at com.ibm.db2.jcc.t4.sb.g(sb.java:120)
at com.ibm.db2.jcc.b.tk.M(tk.java:1351)
at com.ibm.db2.jcc.b.tk.a(tk.java:2246)
at com.ibm.db2.jcc.b.tk.c(tk.java:601)
at com.ibm.db2.jcc.b.tk.executeUpdate(tk.java:585)
at
org.apache.commons.dbcp.DelegatingStatement.executeUpdate(DelegatingStatement.java:
228)
at
org.apache.commons.dbcp.DelegatingStatement.executeUpdate(DelegatingStatement.java:
228)
at
org.openanzo.jdbc.container.sql.NodeSQL.insertUncommittedReferences(NodeSQL.java:
5066)
... 19 more
2008-11-11 09:44:09,336 4000 [ERROR] [http://openanzo.org/
service/CombusUpdateServiceListener] [OpName=update] [OpId=3a5a50b6-
e6ac-4aa3-9394-4b09709d90cf] [OpUser=ldap://
uid=default,ou=users,dc=openanzo,dc=org] [RunAs=] [UserDesc=]
org.openanzo.datasource.update.ServerUpdatesProcessor - Error
Processing Update:
org.openanzo.jdbc.utils.RdbException: ErrorCode[256:262181]
[RDB_ERROR] Failed to execute an sql operation:
[insertUncommittedReferences] SQL:[INSERT INTO {1} (ID,TRANSACTIONID)
SELECT {0}.ID,{2} FROM {0} WHERE {0}.TYPE IN (1,3)] Params:[]
Templates:
[bulkResolutionTableName=SESSION.ID_TMP,lockedIdsTable=ANZO_USED_IDS,transactionId=-674949639247164793]
Root Cause: com.ibm.db2.jcc.b.SqlException DB2 SQL Error:
SQLCODE=-805, SQLSTATE=51002, SQLERRMC=NULLID.SYSLH104
0X5359534C564C3031, DRIVER=3.50.152
at
org.openanzo.jdbc.container.sql.NodeSQL.insertUncommittedReferences(NodeSQL.java:
5071)
at
org.openanzo.jdbc.layout.NodeLiteralLayout.resolveStoredNodes(NodeLiteralLayout.java:
410)
at
org.openanzo.jdbc.layout.NodeLayoutCacheProxy.resolveStoredNodes(NodeLayoutCacheProxy.java:
173)
at
org.openanzo.jdbc.layout.CompositeNodeLayout.resolveStoredNodesFast(CompositeNodeLayout.java:
513)
at
org.openanzo.jdbc.layout.CompositeNodeLayout.resolveStoredNodes(CompositeNodeLayout.java:
425)
at
org.openanzo.datasource.nodecentric.DispatchingDeltaQuadStore.preCommit(DispatchingDeltaQuadStore.java:
168)
at
org.openanzo.datasource.nodecentric.DispatchingDeltaQuadStore.update(DispatchingDeltaQuadStore.java:
376)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.handleTransactionEnd(ServerUpdatesProcessor.java:
412)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.handleTransaction(ServerUpdatesProcessor.java:
312)
at org.openanzo.datasource.update.ServerUpdatesProcessor
$1.handleTransaction(ServerUpdatesProcessor.java:252)
at
org.openanzo.services.serialization.XMLUpdatesReader.parseUpdateTransactions(XMLUpdatesReader.java:
149)
at
org.openanzo.services.serialization.XMLUpdatesReader.read(XMLUpdatesReader.java:
82)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.read(ServerUpdatesProcessor.java:
247)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.update(ServerUpdatesProcessor.java:
202)
at
org.openanzo.datasource.nodecentric.NodeCentricUpdateService.updateInternal(NodeCentricUpdateService.java:
88)
at
org.openanzo.datasource.services.BaseUpdateService.update(BaseUpdateService.java:
139)
at
org.openanzo.combus.listener.CombusUpdateServiceListener.handleMessage(CombusUpdateServiceListener.java:
134)
at
org.openanzo.combus.endpoint.BaseServiceListener.processMessage(BaseServiceListener.java:
319)
at org.openanzo.combus.endpoint.BaseServiceListener.access
$000(BaseServiceListener.java:57)
at org.openanzo.combus.endpoint.BaseServiceListener
$ProcessThread.run(BaseServiceListener.java:262)
Caused by: com.ibm.db2.jcc.b.SqlException: DB2 SQL Error:
SQLCODE=-805, SQLSTATE=51002, SQLERRMC=NULLID.SYSLH104
0X5359534C564C3031, DRIVER=3.50.152
at com.ibm.db2.jcc.b.wc.a(wc.java:55)
at com.ibm.db2.jcc.b.wc.a(wc.java:126)
at com.ibm.db2.jcc.b.tk.c(tk.java:1901)
at com.ibm.db2.jcc.b.tk.d(tk.java:1889)
at com.ibm.db2.jcc.b.tk.b(tk.java:1356)
at com.ibm.db2.jcc.t4.db.h(db.java:219)
at com.ibm.db2.jcc.t4.db.b(db.java:45)
at com.ibm.db2.jcc.t4.t.b(t.java:38)
at com.ibm.db2.jcc.t4.sb.g(sb.java:120)
at com.ibm.db2.jcc.b.tk.M(tk.java:1351)
at com.ibm.db2.jcc.b.tk.a(tk.java:2246)
at com.ibm.db2.jcc.b.tk.c(tk.java:601)
at com.ibm.db2.jcc.b.tk.executeUpdate(tk.java:585)
at
org.apache.commons.dbcp.DelegatingStatement.executeUpdate(DelegatingStatement.java:
228)
at
org.apache.commons.dbcp.DelegatingStatement.executeUpdate(DelegatingStatement.java:
228)
at
org.openanzo.jdbc.container.sql.NodeSQL.insertUncommittedReferences(NodeSQL.java:
5066)
... 19 more
2008-11-11 09:53:51,289 585953 [ERROR] [http://openanzo.org/
service/CombusUpdateServiceListener] [OpName=update]
[OpId=57b228fd-378a-4ef4-ab75-ee88402a5d6f] [OpUser=ldap://
uid=default,ou=users,dc=openanzo,dc=org] [RunAs=] [UserDesc=]
org.openanzo.datasource.update.ServerUpdatesProcessor - Error
Processing Update:
org.openanzo.jdbc.utils.RdbException: ErrorCode[256:262181]
[RDB_ERROR] Failed to execute an sql operation:
[insertUncommittedReferences] SQL:[INSERT INTO {1} (ID,TRANSACTIONID)
SELECT {0}.ID,{2} FROM {0} WHERE {0}.TYPE IN (1,3)] Params:[]
Templates:
[bulkResolutionTableName=SESSION.ID_TMP,lockedIdsTable=ANZO_USED_IDS,transactionId=-1225892375072967805]
Root Cause: com.ibm.db2.jcc.b.SqlException DB2 SQL Error:
SQLCODE=-805, SQLSTATE=51002, SQLERRMC=NULLID.SYSLH104
0X5359534C564C3031, DRIVER=3.50.152

[Didier]

I'm a colleague of Andreas, and this the error that appeared on the
client side:

Update Repository Thread] ERROR [UpdateRepositoryThread.java:run:94]
- org.openanzo.services.UpdateServerException: ErrorCode[1024:32785]
[CLIENT_ERROR] Errors updating server. ErrorCode[256:262181]

[RDB_ERROR] Failed to execute an sql operation:
[insertUncommittedReferences] SQL:[INSERT INTO {1} (ID,TRANSACTIONID)
SELECT {0}.ID,{2} FROM {0} WHERE {0}.TYPE IN (1,3)] Params:[
] Templates:

[bulkResolutionTableName=SESSION.ID_TMP,lockedIdsTable=ANZO_USED_IDS,transactionId=2304350364741115389]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
org.openanzo.services.UpdateServerException: ErrorCode[1024:32785]
[CLIENT_ERROR] Errors updating server. ErrorCode[256:262181]

[RDB_ERROR] Failed to execute an sql operation:
[insertUncommittedReferences] SQL:[INSERT INTO {1} (ID,TRANSACTIONID)
SELECT {0}.ID,{2} FROM {0} WHERE {0}.TYPE IN (1,3)] Params:[
] Templates:

[bulkResolutionTableName=SESSION.ID_TMP,lockedIdsTable=ANZO_USED_IDS,transactionId=2304350364741115389]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
ErrorCode[32:270] [SERVICE_CONTAINER] Previous transaction failed
involving graph [http://www.mytest.com/graphs/contextGraph]
at org.openanzo.client.Replicator.update(Replicator.java:303)
at org.openanzo.client.AnzoClient.updateRepository(AnzoClient.java:
585)
at mypackage.UpdateRepositoryThread.run(UpdateRepositoryThread.java:
84)
at java.lang.Thread.run(Unknown Source)

[Matthew Roy]

Its hard to know why the database throws these exceptions, but it seems
like for some reason the internals of the system are not letting go of
all the prepared statements, so DB2 is throwing an exception. I'll have
to do some more debugging in order to determine if this is truly the
case, and where these leaks are coming from. In the mean time, I'll
check in an update today that will remove one source of prepared
statement caching, which may or may not fix your problem.

Matt

[Andreas S. Rath (teddius)]

Thank you Matt. I am testing tomorrow. I suppose the update will be in
tomorrows snapshot.

Andreas

[Andreas S. Rath (teddius)]

Thanks Matt! It seems to work more stable now. Although we got
following error on the server:

< ---- snip server exception ---->
-> All Currently Registered OpenAnzo services started.
2008-11-12 12:16:13,248 0 [ERROR] [http://openanzo.org/
service/CombusUpdateServiceListener] [OpName=update] [OpId=5a6568e1-
bc6c-4500-b076-52c016918c0c] [OpUser=ldap://

uid=default,ou=users,dc=openanzo,dc=org] [RunAs=] [UserDesc=]
org.openanzo.datasource.update.ServerUpdatesProcessor - Error
Processing Update:

org.openanzo.exceptions.AnzoException: ErrorCode[256:259] [RDB_ERROR]
Failed executing batch operation:[{0}] Root Cause:
com.ibm.db2.jcc.b.em [jcc][t4][102][10040][3.50.152] Non-atomic batch
failure. The batch was submitted, but at least one exception occurred
on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched
elements. ERRORCODE=-4228, SQLSTATE=null
at
org.openanzo.datasource.nodecentric.NodeCentricServerQuadStore.executeNamedGraphsUpdates
(NodeCentricServerQuadStore.java:173)
at
org.openanzo.datasource.nodecentric.DispatchingDeltaQuadStore.update
(DispatchingDeltaQuadStore.java:383)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.handleTransactionEnd
(ServerUpdatesProcessor.java:412)
at
org.openanzo.datasource.update.ServerUpdatesProcessor.handleTransaction
(ServerUpdatesProcessor.java:312)

at org.openanzo.datasource.update.ServerUpdatesProcessor
$1.handleTransaction(ServerUpdatesProcessor.java:252)
at
org.openanzo.services.serialization.XMLUpdatesReader.parseUpdateTransactions
(XMLUpdatesReader.java:149)
at org.openanzo.services.serialization.XMLUpdatesReader.read
(XMLUpdatesReader.java:82)
at org.openanzo.datasource.update.ServerUpdatesProcessor.read
(ServerUpdatesProcessor.java:247)
at org.openanzo.datasource.update.ServerUpdatesProcessor.update
(ServerUpdatesProcessor.java:202)
at
org.openanzo.datasource.nodecentric.NodeCentricUpdateService.updateInternal
(NodeCentricUpdateService.java:88)
at org.openanzo.datasource.services.BaseUpdateService.update
(BaseUpdateService.java:139)
at
org.openanzo.combus.listener.CombusUpdateServiceListener.handleMessage
(CombusUpdateServiceListener.java:134)
at
org.openanzo.combus.endpoint.BaseServiceListener.processMessage

(BaseServiceListener.java:319)

at org.openanzo.combus.endpoint.BaseServiceListener.access$000
(BaseServiceListener.java:57)
at org.openanzo.combus.endpoint.BaseServiceListener
$ProcessThread.run(BaseServiceListener.java:262)

Caused by: com.ibm.db2.jcc.b.em: [jcc][t4][102][10040][3.50.152] Non-
atomic batch failure. The batch was submitted, but at least one
exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched
elements. ERRORCODE=-4228, SQLSTATE=null
at com.ibm.db2.jcc.b.wc.a(wc.java:365)
at com.ibm.db2.jcc.b.p.a(p.java:318)
at com.ibm.db2.jcc.b.uk.a(uk.java:3511)
at com.ibm.db2.jcc.b.uk.b(uk.java:3340)
at com.ibm.db2.jcc.b.uk.executeBatch(uk.java:1944)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch
(DelegatingStatement.java:297)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch
(DelegatingStatement.java:297)
at
org.openanzo.datasource.nodecentric.NodeCentricServerQuadStore.executeNamedGraphsUpdates
(NodeCentricServerQuadStore.java:168)
... 14 more


<---- snip server exception end --->


and the following is the exception on the client:

< ----- snip client side exception ---->
Update Repository Thread] ERROR [UpdateRepositoryThread.java:run:98]
- org.openanzo.exceptions.AnzoException: ErrorCode[16:16393]
[COMBUS_ERROR] No Response from Server for request 081e5821-a1a6-4c5e-
b22a-f357bd7ed7c8.
org.openanzo.exceptions.AnzoException: ErrorCode[16:16393]
[COMBUS_ERROR] No Response from Server for request 081e5821-a1a6-4c5e-
b22a-f357bd7ed7c8.
at org.openanzo.combus.CombusConnection.requestResponse
(CombusConnection.java:686)
at org.openanzo.combus.proxy.CombusUpdateServiceProxy.update
(CombusUpdateServiceProxy.java:172)
at org.openanzo.combus.proxy.CombusUpdateServiceProxy.update
(CombusUpdateServiceProxy.java:138)
at org.openanzo.client.Replicator.update(Replicator.java:144)
at org.openanzo.client.AnzoClient.updateRepository(AnzoClient.java:
585)
at kc.dyonipos.client.eventlogger.UpdateRepositoryThread.run
(UpdateRepositoryThread.java:86)
at java.lang.Thread.run(Unknown Source)

<---- snip client side exception end ---->

I guess this is a timeout exception. Just for noticing, we set the
timeout with
ServicesProperties.setTimeout(this.anzoClientProperties, 180000);

Can it be that a timeout exception influences the graph consistency?
(...if the graph has not been closed and remains open all the time
until the application is closed?)

Thanks,
Andreas