Transactions in a distributed environment lead to a corrupt database

[Daniel]

Because we've had daily outages of 15 minutes due to the backup of the

DB, we follwed the advice of Luca to set up a distributed environment

with two nodes.

Here is the config:

default-distributed-db-config.json

{

     "replication": true,

     "autoDeploy": true,

     "hotAlignment": true,

     "resyncEvery": 15,

     "clusters": {

         "internal": {

             "replication": false

         },

         "index": {

             "replication": false

         },

         "ODistributedConflict": {

             "replication": false

         },

         "*": {

             "replication": true,

             "readQuorum": 1,

             "writeQuorum": 1,

             "failureAvailableNodesLessQuorum": false,

             "readYourWrites": true,

             "partitioning": {

                 "strategy": "round-robin",

                 "default": 0,

                 "partitions": [

                     [ "<NEW_NODE>" ]

                 ]

             }

         }

     }

}

So now here is what happended and finally led to a corrupted DB:

- Stopped the application server

- Setup and configured replication with the settings above

- Two nodes: node01 and node02

- node02 had no existing database, so node01 exported and zipped the db

and sent it to node02

- node02 extracted the db successfully, log message: INFO [node02] installed database  [OHazelcastPlugin]

- We started the application server

- After some time the following exception (sometimes) appeares in the orient-server.log:

Cannot route TX operation against distributed node

Error on committing distributed transaction

-> com.orientechnologies.orient.server.distributed.ODistributedStorage.commit(ODistributedStorage.java:502)

-> com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:109)

-> com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:146)

-> com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:440)

-> com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:435)

-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.commit(ONetworkProtocolBinary.java:1253)

-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:325)

-> com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:126)

-> com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45)

- After a while the following exception appeared in our application with increased frequency:

Caused by:

com.orientechnologies.orient.core.exception.OTransactionException: Cannot insert item in mvrb-tree because the transactional item was not found.

     at com.orientechnologies.orient.core.type.tree.OMVRBTreeRID.internalPut(OMVRBTreeRID.java:156)

     at com.orientechnologies.orient.core.type.tree.OMVRBTreeRID.internalPut(OMVRBTreeRID.java:57)

     at com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.put(OMVRBTreePersistent.java:468)

     at com.orientechnologies.orient.core.type.tree.provider.OMVRBTreeRIDProvider.lazyUnmarshall(OMVRBTreeRIDProvider.java:227)

     at com.orientechnologies.orient.core.type.tree.OMVRBTreeRID.getTreeSize(OMVRBTreeRID.java:332)

     at com.orientechnologies.orient.core.type.tree.OMVRBTreeRID.size(OMVRBTreeRID.java:318)

     at com.orientechnologies.orient.core.type.tree.OMVRBTreeRIDSet.size(OMVRBTreeRIDSet.java:91)

     at com.orientechnologies.common.collection.OMultiValue.getSize(OMultiValue.java:82)

     at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:165)

     at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:92)

     at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toStream(ORecordSerializerSchemaAware2CSV.java:518)

     at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:127)

     at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:122)

     at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:391)

     at com.orientechnologies.orient.client.remote.OStorageRemote.commitEntry(OStorageRemote.java:1919)

     ... 75 more

- Then we took a look at the corresponding dataset to those exception via the orientdb console, i.e.:

select from #12:155580

Error:

com.orientechnologies.orient.core.exception.OTransactionException:

Cannot insert item in mvrb-tree because the transactional item was not found.

- Simple properties could be selected without problems, i.e.:

select email from #12:155580

----+-----+---------

#   |@RID |email

----+-----+---------

0   |#-2:1|<removed>

----+-----+---------

- Selections of linked Edges resulted sometimes in errors, i.e.:

select out_Friend from #12:155580

Error:

com.orientechnologies.orient.core.exception.OTransactionException:

Cannot insert item in mvrb-tree because the transactional item was not found.

- Whereas others worked

select in_Friend from #12:155580

----+-----+---------

#   |@RID |in_Friend

----+-----+---------

0   |#-2:1|[63]

----+-----+---------

1 item(s) found. Query executed in 0.004 sec(s).

Any idea or hint for us?

Thanks a million

Daniel 

[Mateusz Dymczyk]

Which version are you using? I remember similar exceptions before switching to 1.7, seems they had some bugs.

Also I don't know if Luca suggested exactly that setup but running such a DB with 2 nodes doesn't seem like a great idea - should you have a link (or well any sort of network error) between them or if one of them should go down for some time you will have inconsistent data (since both quorums are set to only 1). An odd number of servers with quorum (N/2)+1 is a better idea.

Mateusz

[Daniel Hanelt [STARGAST Systems GmbH]]

Hey,

we are using the 1.7 RC2 but we've had the same problems with 1.6

You are right with the numbers of nodes, three would make more sense. But I doubt that this would solve the problems we've experienced, the exceptions started right away and there was no network issue at that time.

Thanks

Daniel

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrey Lomakin]

Hi guys,

Could you create issue about it so we will not forget to look on it ?

--

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[Luca Garulli]

Hi guys,

As Mateusz pointed the quorum settings should be to 2 and failureAvailableNodesLessQuorum = true to maintain consistency. Then I suggest you to try 1.7-SNAPSHOT where we fixed a few issues against distributed configuration.

About your database is it a plocal? The fact your logs show OMVRBTreeRIDSet usage let me think this is an old database created with older version of OrientDB, maybe with local?

Lvc@

[Daniel Hanelt [STARGAST Systems GmbH]]

Hey Luca,

thanks for your reply.

-We chose writeQuorum=1 because we wanted to have some sort of asynchronous replication. We just work with node01, node02 is just needed for handling the backup

- the db is plocal. Back in time it was created with local but then with EXPORT exported and imported in a newly created plocal database with 1.6 (But we were wondering too regarding the logs with OMVRBTreeRIDSet)

thanks a million

Daniek

[Luca Garulli]

Hi Daniel,

to avoid using OMVRBTreeRIDSet you should migrate the graph:

https://github.com/orientechnologies/orientdb/wiki/Upgrade#migrate-graph-to-ridbag

Lvc@

[Daniel Hanelt [STARGAST Systems GmbH]]

Hey Luca,

thanks, we will try it. But it won't help with our distributed transactions, right?

Thanks!

Daniel

[Luca Garulli]

Yes, should fix the tx problem.

Lvc@

[Daniel]

Hey Luca,

After we did the 'upgrade graph' and it finished without any errors or warnings some documents have invalid relationship content. For example our acccounts have a 'Friend' relationship, and after the upgrade 1851/254352 records don't have any friends anymore. I picked one of those broken records for example:

When trying to resolve the friends I get an empty list:

orientdb {upgrade.broken}> select out('Friend') from #12:155580

----+-----+----

#   |@RID |out

----+-----+----

0   |#-2:1|[0]

----+-----+----

1 item(s) found. Query executed in 0.098 sec(s).

Where the same query BEFORE the upgrade returned 35 results:

orientdb {before.upgrade}> select out('Friend') from #12:155580

----+-----+----

#   |@RID |out

----+-----+----

0   |#-2:1|[35]

----+-----+----

1 item(s) found. Query executed in 0.098 sec(s).

So I checked to see what's wrong and did the following query:

orientdb {upgrade.broken}> select out_Friend from #12:155580

----+-----+-----------------------------------------------------------------------------------------------------------------

#   |@RID |out_Friend

----+-----+-----------------------------------------------------------------------------------------------------------------

0 |#-2:1|%AQAAACMAEwAAAAAAATybABMAAAAAAAE9OAATAAAAAAABP7kAEwAAAAAAAUAcABMAAAAAAAFB6QATAAAAAAABRFwAEwAAAAAAAUTPABMAAAAAAAFJ1wATAAAAAAABSqkAEwAAAAAAAU6CABMAAAAAAAFO9gATAAAAAAABUyYAEwAAAAAAAVadABMAAAAAAAFcZgATAAAAAAABXYgAEwAAAAAAAV9bABMAAAAAAAFf2AATAAAAAAABYOgAEwAAAAAAAWnaABMAAAAAAAFulgATAAAAAAABbqcAEwAAAAAAAXFxABMAAAAAAAF1mQATAAAAAAABiIYAEwAAAAAAAYmvABMAAAAAAAGLzAATAAAAAAABlBcAEwAAAAAAAZx0ABMAAAAAAAGktQATAAAAAAABqPkAEwAAAAAAAakTABMAAAAAAAG98wATAAAAAAAB3o4AEwAAAAAAAeyEABMAAAAAAAH9WQ==;

----+-----+-----------------------------------------------------------------------------------------------------------------

1 item(s) found. Query executed in 0.012 sec(s).

This seems to be the base64 encoded RidBag content. How can we fix the relationships BEFORE the upgrade to prevent such broken records AFTER the upgrade?

When trying to add a new 'Friend' relation to such broken records I get the following exception:

java.lang.IllegalStateException: Relationship content is invalid on field out_Friend. Found: %AQAAACMAEwAAAAAAATybABMAAAAAAAE9OAATAAAAAAABP7kAEwAAAAAAAUAcABMAAAAAAAFB6QATAAAAAAABRFwAEwAAAAAAAUTPABMAAAAAAAFJ1wATAAAAAAABSqkAEwAAAAAAAU6CABMAAAAAAAFO9gATAAAAAAABUyYAEwAAAAAAAVadABMAAAAAAAFcZgATAAAAAAABXYgAEwAAAAAAAV9bABMAAAAAAAFf2AATAAAAAAABYOgAEwAAAAAAAWnaABMAAAAAAAFulgATAAAAAAABbqcAEwAAAAAAAXFxABMAAAAAAAF1mQATAAAAAAABiIYAEwAAAAAAAYmvABMAAAAAAAGLzAATAAAAAAABlBcAEwAAAAAAAZx0ABMAAAAAAAGktQATAAAAAAABqPkAEwAAAAAAAakTABMAAAAAAAG98wATAAAAAAAB3o4AEwAAAAAAAeyEABMAAAAAAAH9WQ==;

    at com.tinkerpop.blueprints.impls.orient.OrientVertex.createLink(OrientVertex.java:603)

    at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:285)

    ...

Any hints?

Thanks a million!

Daniel

[Artem Orobets]

Hi Daniel,

Yes it is a RidBag content, but for some reason it is recognized as a string.

Could you provide a steps how to reproduce it?

Best regards,

Artem Orobets

Orient Technologies

the Company behind OrientDB