Serious Unique index violation - ORecordDuplicatedException: Cannot index record found duplicate
62 views
Skip to first unread message
Simon White
Jun 8, 2016, 8:22:59 AM6/8/16
to OrientDB
Affecting 2.1.19
We have quite a large database - about a million entries for a particular vertex. Recently I noticed by chance a duplicate, but surely this is impossible when there is a unique index?
I attempted to rebuild the index but unsurprisingly it fails with an ORecordDuplicatedException. Upon investigation it appears that there are many many duplicates - hundreds in fact.
I am guessing this is a race condition as we insert the entities via both Java API and HTTP under heavy load (many threads from several separate applications).
Possibly unrelated but - when creating one of these entries we run an onRecordBeforeCreate hook to do some validation and also add a 'date created' field. This takes milliseconds but under heavy load could it be enough to throw the index off?
This is a very serious issue.
Any hints to avoid the problem?
We have quite a large database - about a million entries for a particular vertex. Recently I noticed by chance a duplicate, but surely this is impossible when there is a unique index?
I attempted to rebuild the index but unsurprisingly it fails with an ORecordDuplicatedException. Upon investigation it appears that there are many many duplicates - hundreds in fact.
I am guessing this is a race condition as we insert the entities via both Java API and HTTP under heavy load (many threads from several separate applications).
Possibly unrelated but - when creating one of these entries we run an onRecordBeforeCreate hook to do some validation and also add a 'date created' field. This takes milliseconds but under heavy load could it be enough to throw the index off?
This is a very serious issue.
Any hints to avoid the problem?
odbuser
Jun 11, 2016, 2:24:12 AM6/11/16
to OrientDB
Sounds serious. Is it possibly due to the way that you wrote the hooks? Can you post the hook? Is there code in there that is not transactional? Are you using sql in the hook or a non-transactional graphdb?
Simon White
Jun 13, 2016, 5:42:16 AM6/13/16
to OrientDB
To create the entity we are running pure sql and are NOT using transactions, this is because within the hook we do...
final ODatabaseDocumentInternal db = ODatabaseRecordThreadLocal.INSTANCE.get();
OrientGraph graph = getGraph((ODatabaseDocumentTx) db);
And use this to create a new edge between the newly created smk (just a domain concept) and another entity.
If we run this in a transaction the entity gets a temporary orid and therefore cannot be connected to anything...!
I see this is not working though! I guess we might have to split this into two parts? a create transaction and then an update edges transaction?
final ODatabaseDocumentInternal db = ODatabaseRecordThreadLocal.INSTANCE.get();
OrientGraph graph = getGraph((ODatabaseDocumentTx) db);
And use this to create a new edge between the newly created smk (just a domain concept) and another entity.
If we run this in a transaction the entity gets a temporary orid and therefore cannot be connected to anything...!
I see this is not working though! I guess we might have to split this into two parts? a create transaction and then an update edges transaction?
Reply all
Reply to author
Forward
0 new messages