Exceptions on locks to indexes

[Odysseas]

Hi,

We have been doing some load testing and profiling to try and improve the performance of our application running on top of OrientDB. We are

currently using version 1.5.1. We noticed that once the concurrency level goes above a certain value we get the following exceptions and

then we start to see other errors with NullPointerExceptions from our application that do not occur when the concurrency level is lower. Do

you have any suggestions on what might be causing this and how to resolve it?

04:38:01,873 INFO  [STDOUT] [2013-09-09 04:38:01,868]ERROR310752[http-127.0.0.1-8080-32] - EntityDaoOrientdb.saveRecord(EntityDaoOrientdb.java:154) - Failed while trying to save an instance of entity: person due to com.orientechnologies.common.concur.OTimeoutException: Timeout on acquiring shared lock against resource of class : class com.orientechnologies.orient.core.index.OIndexNotUnique with timeout=5000

com.orientechnologies.common.concur.OTimeoutException: Timeout on acquiring shared lock against resource of class : class com.orientechnologies.orient.core.index.OIndexNotUnique with timeout=5000

at com.orientechnologies.common.concur.resource.OSharedResourceAdaptive.acquireSharedLock(OSharedResourceAdaptive.java:120)

at com.orientechnologies.common.concur.resource.OSharedResourceAdaptiveExternal.acquireSharedLock(OSharedResourceAdaptiveExternal.java:40)

at com.orientechnologies.orient.core.index.OIndexAbstract.getDefinition(OIndexAbstract.java:891)

at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.getDefinition(OIndexAbstractDelegate.java:212)

at com.orientechnologies.orient.core.index.OClassIndexManager.checkIndexedPropertiesOnCreation(OClassIndexManager.java:531)

at com.orientechnologies.orient.core.index.OClassIndexManager.checkIndexesAndAquireLock(OClassIndexManager.java:517)

at com.orientechnologies.orient.core.index.OClassIndexManager.onRecordBeforeCreate(OClassIndexManager.java:69)

at com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:232)

at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:1018)

at com.orientechnologies.orient.core.tx.OTransactionOptimistic.addRecord(OTransactionOptimistic.java:239)

at com.orientechnologies.orient.core.tx.OTransactionOptimistic.saveRecord(OTransactionOptimistic.java:231)

at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:264)

at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:37)

at com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:312)

at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1365)

at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1354)

at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1343)

[Luca Garulli]

Hi,

the attached stack is not relative to a NPE, is that the exception or another one?

Lvc@

--
 
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Odysseas]

This exception happens first then we get NPEs within our code but these NPEs only occur with high concurrency and only after the exception I attached previously. Let me know if you need any additional information.

By the way, we also tried the plocal option with 1.5.1 to see if that helps and run into a minor issue. If the name of the database contains a period '.' then we get the following exception. This does not happen with a local

database but only with a plocal one. We got around it by changing the database name but I thought you may want to fix it at some point.

Thanks,

Odysseas

Caused by: java.lang.NumberFormatException: For input string: "db"

at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)

at java.lang.Integer.parseInt(Integer.java:449)

at java.lang.Integer.parseInt(Integer.java:499)

at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.extractOrder(OWriteAheadLog.java:680)

[Luca Garulli]

Hi Odysseas,

please could you open a new issue for this?

About the issue we're working to provide a better management of cross locks between indexes and database, but this will be available in 2.0, sorry. In the meanwhile you can:

1) reduce the indexes: do you have different indexes against multiple fields of the same class? Could you rather create a composite index that embrace all of them?

2) enlarge timeouts with:

OGlobalConfiguration.STORAGE_RECORD_LOCK_TIMEOUT.setValue( 10000 )

OGlobalConfiguration.MVRBTREE_TIMEOUT.setValue( 10000 )

Lvc@

[Odysseas]

Hi Luca,

Thanks for the suggestions. I changed those parameters but the only effect they had was to cause the exception to be reported when the new timeout value was exceeded. I also tried using composite indexes but didn't eliminate the exceptions as well. It is behaving as though there is a race condition that causes the threads to deadlock but I haven't identified the cause yet.

I will open a new issue for the exception regarding the database name.

Odysseas

[Odysseas]

After looking at a thread dump of our application, I see threads that are permanently stuck waiting for a lock. Here are a couple of examples below. I suspect that this is a configuration issue but do you have any suggestions on what else I may try.

Thanks,

Odysseas

"http-127.0.0.1-8080-11" daemon prio=10 tid=0x00007f47a4013800 nid=0x7b5f in Object.wait() [0x00007f47903ea000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at com.orientechnologies.orient.client.remote.OStorageRemote.beginRequest(OStorageRemote.java:1805)

- locked <0x000000078363d2a8> (a java.util.ArrayList)

at com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:1106)

at com.orientechnologies.orient.client.remote.OStorageRemoteThread.commit(OStorageRemoteThread.java:266)

at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:69)

at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:114)

at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:458)

at EntityDaoOrientdb.saveRecord(EntityDaoOrientdb.java:152)

"http-127.0.0.1-8080-12" daemon prio=10 tid=0x00007f47a4015000 nid=0x7b60 in Object.wait() [0x00007f47902e9000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at com.orientechnologies.orient.client.remote.OStorageRemote.beginRequest(OStorageRemote.java:1805)

- locked <0x000000078363d2a8> (a java.util.ArrayList)

at com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1552)

at com.orientechnologies.orient.client.remote.OStorageRemote.open(OStorageRemote.java:176)

at com.orientechnologies.orient.client.remote.OStorageRemoteThread.open(OStorageRemoteThread.java:70)

at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.open(ODatabaseRaw.java:105)

at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:49)

at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.open(ODatabaseRecordAbstract.java:119)

at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:49)

at com.orientechnologies.orient.core.db.graph.OGraphDatabase.open(OGraphDatabase.java:92)

at com.orientechnologies.orient.core.db.graph.OGraphDatabasePooled.<init>(OGraphDatabasePooled.java:44)

at com.orientechnologies.orient.core.db.graph.OGraphDatabasePool.createResource(OGraphDatabasePool.java:39)

at com.orientechnologies.orient.core.db.graph.OGraphDatabasePool.createResource(OGraphDatabasePool.java:20)

at com.orientechnologies.orient.core.db.ODatabasePoolBase$1.createNewResource(ODatabasePoolBase.java:68)

at com.orientechnologies.orient.core.db.ODatabasePoolBase$1.createNewResource(ODatabasePoolBase.java:57)

at com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:68)

at com.orientechnologies.orient.core.db.ODatabasePoolAbstract.acquire(ODatabasePoolAbstract.java:74)

at com.orientechnologies.orient.core.db.ODatabasePoolAbstract.acquire(ODatabasePoolAbstract.java:59)

at com.orientechnologies.orient.core.db.ODatabasePoolBase.acquire(ODatabasePoolBase.java:99)

at EntityDaoOrientdb.connect(EntityDaoOrientdb.java:1386)

[Luca Garulli]

Hi Odysseas,

we've replaced MVRB-Tree indexes with SB-Tree as default indexes, so if you export and reimport the database indexes will be built with the new SB-Tree much more performant, lightweight and without such lock limitations.

Lvc@

[Odysseas]

Luca,

The new index implementation is available on the 1.6.0-SNAPSHOT release?

Thanks,

Odysseas

[Luca Garulli]

Yes (see the Andrey's announce few days ago).

Lvc@

[Odysseas]

I tried out the new indexes in 1.6.0-SNAPSHOT and I got an exception. Any ideas on what may be going wrong?
Thanks.

Caused by: java.lang.NullPointerException
    at com.orientechnologies.orient.core.index.sbtree.local.OSBTreeBucket.getKey(OSBTreeBucket.java:209)
    at com.orientechnologies.orient.core.index.sbtree.local.OSBTreeBucket.find(OSBTreeBucket.java:125)
    at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.findBucket(OSBTree.java:1032)
    at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.get(OSBTree.java:124)
    at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.get(OSBTreeIndexEngine.java:307)
    at com.orientechnologies.orient.core.index.OIndexMultiValues.get(OIndexMultiValues.java:49)
    at com.orientechnologies.orient.core.index.OIndexMultiValues.get(OIndexMultiValues.java:38)
    at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.get(OIndexAbstractDelegate.java:76)
    at com.orientechnologies.orient.core.index.OIndexTxAwareMultiValue.get(OIndexTxAwareMultiValue.java:52)
    at com.orientechnologies.orient.core.index.OIndexTxAwareMultiValue.get(OIndexTxAwareMultiValue.java:39)
    at com.orientechnologies.orient.core.sql.operator.OQueryOperatorEquals.executeIndexQuery(OQueryOperatorEquals.java:129)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:696)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:610)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:137)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:360)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:338)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:327)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:57)
    at com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:99)
    ... 35 more

[Andrey Lomakin]

Hi,

I am really sorry,  fixed this issue, will commit after 10 000 000 records CRUD tests will pass.

--

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[Andrey Lomakin]

Pushed in develop branch.

[Melissa Mifsud]

I have been having similar lock problems with 1.5.1.

I will move onto 1.6.0-SNAPSHOT and see how this goes. I am still using local (not plocal) with Object database and transactions.

Do you recommend any other settings?

---

Melissa Anne Mifsud

[Andrey Lomakin]

Hi Melissa,

To control disk cache size you should set "storage.diskCache.bufferSize" to size which fits your environment, it is measured in megabytes and is 4096 be default. 

[Melissa Mifsud]

Is this setting still relevant when using Orient embedded? 

[Andrey Lomakin]

Yes, 

It does not matter which one do you use.

[Melissa Mifsud]

Ok thanks. 

Is this cache space additional to the application's memory? Sorry for the questions. There are a lot of settings which I would really like to understand better rather than just guess which to use and the ideal values for them!

[Melissa Mifsud]

Also, is this setting related/similar to the 2nd level cache?

[Odysseas]

Thanks for the quick response Andrey. I tried the latest version and the NullPointerException is gone but I am getting the timeout exceptions again. I find it curious that a query request would be contesting for the same lock on the index as an operation that is making a change. Any suggestions on what I may be doing wrong?

10:51:13,179 INFO  [STDOUT] [2013-09-12 10:51:13,173]ERROR127122[http-127.0.0.1-8080-21] - EntityDaoOrientdb.findRecordsByAttributes(EntityDaoOrientdb.java:502) - Failed while trying to query the system using entity: person due to com.orientechnologies.common.concur.OTimeoutException: Timeout on acquiring shared lock against resource of class : class com.orientechnologies.orient.core.index.OIndexNotUnique with timeout=20000
com.orientechnologies.common.concur.OTimeoutException: Timeout on acquiring shared lock against resource of class : class com.orientechnologies.orient.core.index.OIndexNotUnique with timeout=20000

    at com.orientechnologies.common.concur.resource.OSharedResourceAdaptive.acquireSharedLock(OSharedResourceAdaptive.java:120)
    at com.orientechnologies.common.concur.resource.OSharedResourceAdaptiveExternal.acquireSharedLock(OSharedResourceAdaptiveExternal.java:40)

    at com.orientechnologies.orient.core.index.OIndexAbstract.getDefinition(OIndexAbstract.java:896)
    at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.getDefinition(OIndexAbstractDelegate.java:212)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:649)

    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:610)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:137)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:360)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:338)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:327)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:57)
    at com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:99)

    at com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:88)
    at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:69)
    at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:82)
    at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:29)
    at EntityDaoOrientdb.findRecordsByAttributes(EntityDaoOrientdb.java:496)

10:52:13,726 INFO  [STDOUT] [2013-09-12 10:52:13,715]ERROR187664[http-127.0.0.1-8080-2] - EntityDaoOrientdb.updateRecord(EntityDaoOrientdb.java:379) - Failed while trying to update an instance of entity: person due to com.orientechnologies.common.concur.OTimeoutException: Timeout on acquiring shared lock against resource of class : class com.orientechnologies.orient.core.index.OIndexNotUnique with timeout=20000
com.orientechnologies.common.concur.OTimeoutException: Timeout on acquiring shared lock against resource of class : class com.orientechnologies.orient.core.index.OIndexNotUnique with timeout=20000

    at com.orientechnologies.common.concur.resource.OSharedResourceAdaptive.acquireSharedLock(OSharedResourceAdaptive.java:120)
    at com.orientechnologies.common.concur.resource.OSharedResourceAdaptiveExternal.acquireSharedLock(OSharedResourceAdaptiveExternal.java:40)

    at com.orientechnologies.orient.core.index.OIndexAbstract.getDefinition(OIndexAbstract.java:896)
    at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.getDefinition(OIndexAbstractDelegate.java:212)
    at com.orientechnologies.orient.core.index.OClassIndexManager.updateIndexEntries(OClassIndexManager.java:164)
    at com.orientechnologies.orient.core.index.OClassIndexManager.onRecordAfterUpdate(OClassIndexManager.java:142)
    at com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:265)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:1022)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.addRecord(OTransactionOptimistic.java:385)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.saveRecord(OTransactionOptimistic.java:253)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:270)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:38)
    at com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:312)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1329)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1318)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1307)
    at EntityDaoOrientdb.updateRecord(EntityDaoOrientdb.java:375)

[Andrey Lomakin]

Hi,

We need index definition for both cases to know whether we should update index or not. Index definition contains fields which are hmm indexed ... about  SQL query, we need to know whether we can speed up query by using indexed fields  that is why index definition is used here too.

What is curious for me that is who hold exclusive lock because shared lock (used both in update and read) do allow concurrent usage.

It is later and I will be able to analyse where exclusive locks hold tomorrow.

Would you be able so kind to do following set breakpoint on  OSharedResourceAdaptive.java:120 line and find which thread holds exclusive lock and in which point this thread stopped.

If you do not have such possibility I will look where exclusive lock may be hold tomorrow.

[Andrey Lomakin]

That is different settings 

but disk cache is pretty similar to 2-nd level cache so we are thinking about removing 2-nd level cache.

If you use embedded db I suggest you to disable 2-nd level cache and rely on disk cache.

[Andrey Lomakin]

It seems I know why you have current problem.

Could you help me a bit, could you run your test but in not tx mode, without tx enabeld ?

If all tests passed I will fix concurrent modification problem during couple of days, I think.

[Odysseas]

Hi Andrey,
The exception I submitted in my previous email was when running in embedded mode. We re-run our tests using the remote mode and the exceptions went away and we were able to get much higher concurrency. Great work by the way! The numbers look awesome as compared to the 1.5.1 tests.

I will re-run the tests in local mode and investigate what you suggested and will get back to you. Thanks again for the help.

Odysseas

[Andrey Lomakin]

Cool !

We are working on even more performance optimizations like here https://github.com/orientechnologies/orientdb/issues/1665, here https://github.com/orientechnologies/orientdb/issues/1610 and here https://github.com/orientechnologies/orientdb/issues/1611. 

First will be in 1.6.

And the last two will be in 1.7.

[Odysseas]

Andrey,

After a while I see these exceptions in remote mode. I am not sure if it is helpful to you.

ntityDaoOrientdb.java:527) - Failed while trying to query the system using entity: person due to com.orientechnologies.orient.core.exception.OSerializationException: Error on unmarshalling field 'identifierSet' in record #11:109723 with value:
com.orientechnologies.orient.core.exception.OSerializationException: Error on unmarshalling field 'identifierSet' in record #11:109723 with value:
    at sun.reflect.GeneratedConstructorAccessor138.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:162)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.fromString(ORecordSerializerSchemaAware2CSV.java:512)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.fromStream(ORecordSerializerStringAbstract.java:79)
    at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.deserializeFields(ORecordSchemaAwareAbstract.java:150)
    at com.orientechnologies.orient.core.record.impl.ODocument.deserializeFields(ODocument.java:1377)
    at com.orientechnologies.orient.core.record.impl.ODocument.checkForFields(ODocument.java:1267)
    at com.orientechnologies.orient.core.record.impl.ODocument.rawField(ODocument.java:506)
    at com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:526)
    at OrientdbConverter.convertODocumentToRecord(OrientdbConverter.java:62)

    at java.lang.Thread.run(Thread.java:722)
Caused by: com.orientechnologies.common.concur.OTimeoutException: Timeout on acquiring exclusive lock against resource of class: class com.orientechnologies.common.concur.resource.OSharedResourceAdaptiveExternal with timeout=20000
    at com.orientechnologies.common.concur.resource.OSharedResourceAdaptive.acquireExclusiveLock(OSharedResourceAdaptive.java:87)
    at com.orientechnologies.common.concur.resource.OSharedResourceAdaptiveExternal.acquireExclusiveLock(OSharedResourceAdaptiveExternal.java:31)
    at com.orientechnologies.orient.core.type.tree.OMVRBTreeRID.addChangeListener(OMVRBTreeRID.java:708)
    at com.orientechnologies.orient.core.type.tree.OMVRBTreeRIDSet.addChangeListener(OMVRBTreeRIDSet.java:199)
    at com.orientechnologies.orient.core.record.impl.ODocument.addCollectionChangeListener(ODocument.java:1435)
    at com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:754)
    at com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:614)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.fromString(ORecordSerializerSchemaAware2CSV.java:506)

[Odysseas]

Andrey,

This is the thread that has the exclusive lock:
    - parking to wait for  <0x767c22d8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:929)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1245)
    at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:445)
    at com.orientechnologies.common.concur.lock.OAdaptiveLock.lock(OAdaptiveLock.java:64)
    at com.orientechnologies.orient.core.db.ODatabasePoolAbstract.release(ODatabasePoolAbstract.java:89)
    at com.orientechnologies.orient.core.db.ODatabasePoolBase.release(ODatabasePoolBase.java:142)
    at com.orientechnologies.orient.core.db.graph.OGraphDatabasePooled.close(OGraphDatabasePooled.java:120)
    at EntityDaoOrientdb.saveRecord(EntityDaoOrientdb.java:160)

On Thursday, September 12, 2013 12:38:01 PM UTC-4, Andrey Lomakin wrote:

[Luca Garulli]

Hi Odysseas,

in the meanwhile this is fixed, could you avoid the pool and create a new DB instance every time?

Lvc@

[Andrey Lomakin]

Hi,

It is not the reason of first exception.

I committed fix (possible fix because I can not reproduce your problem on my PC) of this issue. 

Could you try not remote storage at first and then if tests passed remote one ?

Could you also create issue about pool lock ?

[Andrey Lomakin]

Hi Odysseas,

Do you still have concurrent modification exceptions in your performance tests ?

[Andrey Lomakin]

Sorry  timeout exceptions.

[Odysseas]

Hi Andrey,

Yes, we are still getting the timeout exceptions but I believe that the timeout exceptions are a result of the locking issues we are experiencing. I reported those on another
thread on the forum and I refer to them as deadlocks. Another thing we found during our investigation is that the locking issues are exacerbated by the relative speed of the
disk on which the database is running on. When we run our load test against the database on a fast SSD, it takes quite a few threads before the locking issues start
to arise whereas on a slower disk the locking issues come up much earlier. This indicates to me that there are race conditions that are easier to identify when the service
time for a request is higher.

Let me know if I can provide any additional information.

Thanks,
Odysseas

On Saturday, September 21, 2013 12:39:44 AM UTC-4, Andrey Lomakin wrote:

Sorry  timeout exceptions.

[Andrey Lomakin]

Hi Odysseas,

I updated your dead lock issue, could you provide additional information ?

--
 
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Odysseas]

Hi Andrey,

I tried to reproduce the error and posted thread dumps from the OrientDB side. I used a more recent version of OrientDB 1.6 develop so the stack traces looked different although the behaviour is the same. Let me know if you need additional
info

Thanks,
Odysseas

[Andreas Egloff]

Has a resolution been found to the timeout/lock/index issue? Can you point me to the ticket mentioned that tracks it?

Thanks,

Andi

[Andrey Lomakin]

Hi,

Sure https://github.com/orientechnologies/orientdb/issues/1693