RE: bad hole
65 views
Skip to first unread message
Pieter Martin
Jul 6, 2011, 1:59:17 AM7/6/11
to orientdb
Hi,
I am trying to insert about 20000 vertixes in one transaction (one
thread also) on our app startup and getting the following exception on
commit.
SEVERE: An error occured while initializing MyFaces: Found bad order in
hole list: 33) 142918111 [88060] is higher than 0) 8384 [1180]
com.orientechnologies.orient.core.exception.OStorageException: Found bad
order in hole list: 33) 142918111 [88060] is higher than 0) 8384 [1180]
at
com.orientechnologies.orient.core.storage.impl.local.ODataLocalHole.getCloserHole(ODataLocalHole.java:141)
at
com.orientechnologies.orient.core.storage.impl.local.ODataLocal.getCloserHole(ODataLocal.java:465)
at
com.orientechnologies.orient.core.storage.impl.local.ODataLocal.handleHole(ODataLocal.java:340)
at
com.orientechnologies.orient.core.storage.impl.local.ODataLocal.setRecord(ODataLocal.java:230)
at
com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.updateRecord(OStorageLocal.java:1114)
at
com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.updateRecord(OStorageLocal.java:591)
at
com.orientechnologies.orient.core.db.raw.ODatabaseRaw.save(ODatabaseRaw.java:215)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:508)
at
com.orientechnologies.orient.core.tx.OTransactionOptimistic.addRecord(OTransactionOptimistic.java:112)
at
com.orientechnologies.orient.core.tx.OTransactionOptimistic.saveRecord(OTransactionOptimistic.java:100)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:203)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:1)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:281)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.save(OMVRBTreeEntryPersistent.java:205)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:91)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:1)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:427)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:65)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.commitChanges(OMVRBTreeDatabaseLazySave.java:53)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.put(OMVRBTreePersistent.java:371)
at
com.orientechnologies.orient.core.index.OIndexNotUnique.put(OIndexNotUnique.java:50)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.commit(OIndexMVRBTreeAbstract.java:764)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.executeCommit(ODatabaseRecordTx.java:225)
at
com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:42)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:111)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:1)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.commit(ODatabaseRecordWrapperAbstract.java:122)
at
com.tinkerpop.blueprints.pgm.impls.orientdb.OrientGraph.stopTransaction(OrientGraph.java:347)
This happens with or without schemas.
Thanks
Pieter
Pieter Martin
Jul 6, 2011, 2:01:01 AM7/6/11
to orientdb
If I reduce the number of vertexes to 5000 or so it works.
Pieter
Luca Garulli
Jul 6, 2011, 9:41:37 AM7/6/11
to orient-...@googlegroups.com
This is pretty crazy. What release of OrientDB are you using?
Lvc@
Pieter Martin
Jul 6, 2011, 10:24:52 AM7/6/11
to orient-...@googlegroups.com
Hi,
I am on the latest snapshot from svn built on my machine.
Unfortunately its kinda hard to duplicate in a simple test case as the vertices's are wrapped in our domain entities now. It does however happen consistently.
Pieter
I am on the latest snapshot from svn built on my machine.
Unfortunately its kinda hard to duplicate in a simple test case as the vertices's are wrapped in our domain entities now. It does however happen consistently.
Pieter
Luca Garulli
Jul 6, 2011, 11:04:45 AM7/6/11
to orient-...@googlegroups.com
Hi,
have you tried without a transaction? Just set the tx mode to manual and leave the same code.
Lvc@
Pieter Martin
Jul 6, 2011, 3:18:10 PM7/6/11
to orient-...@googlegroups.com
Hi,
I think you meant using automatic transaction?
Tried that but then its so slow I gave up.
Getting the same exception in a unit test now.
@Test
public void test() {
StopWatch stopWatch = new StopWatch();
stopWatch.start();
db.startTransaction();
God god = new God();
god.setName("didthiswork");
for (int i = 0; i < 5000; i++) {
Universe universe1 = new Universe(god);
universe1.setName("universe" + i);
BlackHole blackHole1 = new BlackHole(universe1);
blackHole1.setName("blackHole1");
BlackHole blackHole2 = new BlackHole(universe1);
blackHole2.setName("blackHole2");
BlackHole blackHole3 = new BlackHole(universe1);
blackHole3.setName("blackHole3");
if (i % 500 == 0) {
System.out.println(i);
db.stopTransaction(Conclusion.SUCCESS);
db.startTransaction();
}
}
db.stopTransaction(Conclusion.SUCCESS);
stopWatch.stop();
System.out.println(stopWatch.toString());
}
Some timing stats
0
0:00:00.085
500
0:00:07.234
1000
0:00:07.356
1500
0:00:12.092
2000
0:00:19.534
2500
0:00:24.610
3000
0:00:41.788
3500
0:01:21.415
4000
Then the exception.
The db is 2.8 GB
Tried it commiting on every 100.
It went thru once without exception, but mostly it throws the same exception.
Timing stats for every 100
0
0:00:00.105
100
0:00:01.699
200
0:00:01.070
300
0:00:01.030
400
0:00:00.886
500
0:00:00.916
600
0:00:00.898
700
0:00:01.109
800
0:00:01.212
900
0:00:01.458
1000
0:00:01.283
1100
0:00:01.762
1200
0:00:01.561
1300
0:00:01.847
1400
0:00:02.131
1500
0:00:02.094
1600
0:00:02.634
1700
0:00:06.356
1800
0:00:03.888
1900
0:00:06.559
2000
0:00:02.502
2100
0:00:03.342
2200
0:00:03.121
2300
0:00:03.221
2400
0:00:03.245
2500
0:00:05.988
2600
0:00:04.737
2700
0:00:03.905
2800
0:00:03.945
2900
0:00:08.054
3000
0:00:04.622
3100
0:00:06.002
3200
0:00:19.246
3300
0:00:10.183
3400
0:00:06.299
3500
0:00:23.294
3600
0:00:04.835
3700
0:00:06.874
3800
0:00:25.780
3900
0:00:08.788
4000
Then exception.
The db is 2.4GB
Is the db size normal, and rate at which the commit slows down, especially when committing every 500?
Thanks
Pieter
I think you meant using automatic transaction?
Tried that but then its so slow I gave up.
Getting the same exception in a unit test now.
@Test
public void test() {
StopWatch stopWatch = new StopWatch();
stopWatch.start();
db.startTransaction();
God god = new God();
god.setName("didthiswork");
for (int i = 0; i < 5000; i++) {
Universe universe1 = new Universe(god);
universe1.setName("universe" + i);
BlackHole blackHole1 = new BlackHole(universe1);
blackHole1.setName("blackHole1");
BlackHole blackHole2 = new BlackHole(universe1);
blackHole2.setName("blackHole2");
BlackHole blackHole3 = new BlackHole(universe1);
blackHole3.setName("blackHole3");
if (i % 500 == 0) {
System.out.println(i);
db.stopTransaction(Conclusion.SUCCESS);
db.startTransaction();
}
}
db.stopTransaction(Conclusion.SUCCESS);
stopWatch.stop();
System.out.println(stopWatch.toString());
}
Some timing stats
0
0:00:00.085
500
0:00:07.234
1000
0:00:07.356
1500
0:00:12.092
2000
0:00:19.534
2500
0:00:24.610
3000
0:00:41.788
3500
0:01:21.415
4000
Then the exception.
The db is 2.8 GB
Tried it commiting on every 100.
It went thru once without exception, but mostly it throws the same exception.
Timing stats for every 100
0
0:00:00.105
100
0:00:01.699
200
0:00:01.070
300
0:00:01.030
400
0:00:00.886
500
0:00:00.916
600
0:00:00.898
700
0:00:01.109
800
0:00:01.212
900
0:00:01.458
1000
0:00:01.283
1100
0:00:01.762
1200
0:00:01.561
1300
0:00:01.847
1400
0:00:02.131
1500
0:00:02.094
1600
0:00:02.634
1700
0:00:06.356
1800
0:00:03.888
1900
0:00:06.559
2000
0:00:02.502
2100
0:00:03.342
2200
0:00:03.121
2300
0:00:03.221
2400
0:00:03.245
2500
0:00:05.988
2600
0:00:04.737
2700
0:00:03.905
2800
0:00:03.945
2900
0:00:08.054
3000
0:00:04.622
3100
0:00:06.002
3200
0:00:19.246
3300
0:00:10.183
3400
0:00:06.299
3500
0:00:23.294
3600
0:00:04.835
3700
0:00:06.874
3800
0:00:25.780
3900
0:00:08.788
4000
Then exception.
The db is 2.4GB
Is the db size normal, and rate at which the commit slows down, especially when committing every 500?
Thanks
Pieter
Pieter Martin
Jul 6, 2011, 4:10:31 PM7/6/11
to orient-...@googlegroups.com
Hi,
ran the same unit test again. Got a different exception this time.
com.orientechnologies.orient.core.exception.OStorageException: Found corrupted record hole for rid 6:6849: data position is wrong: 3594599878120320028<->245154917
at com.orientechnologies.orient.core.storage.impl.local.ODataLocal.moveRecord(ODataLocal.java:494)
at com.orientechnologies.orient.core.storage.impl.local.ODataLocal.handleHole(ODataLocal.java:413)
ran the same unit test again. Got a different exception this time.
com.orientechnologies.orient.core.exception.OStorageException: Found corrupted record hole for rid 6:6849: data position is wrong: 3594599878120320028<->245154917
at com.orientechnologies.orient.core.storage.impl.local.ODataLocal.moveRecord(ODataLocal.java:494)
at com.orientechnologies.orient.core.storage.impl.local.ODataLocal.handleHole(ODataLocal.java:413)
at com.orientechnologies.orient.core.storage.impl.local.ODataLocal.setRecord(ODataLocal.java:230)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.updateRecord(OStorageLocal.java:1114)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.updateRecord(OStorageLocal.java:591)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.save(ODatabaseRaw.java:215)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:508)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.addRecord(OTransactionOptimistic.java:112)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.saveRecord(OTransactionOptimistic.java:100)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:203)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:1)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:210)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:159)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:1)
at com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:1)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:685)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:230)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:242)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:77)
at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:131)
at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:126)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:468)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:159)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:1)
at com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:1)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:685)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:230)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:242)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:77)
at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:131)
at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:126)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:468)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.addRecord(OTransactionOptimistic.java:112)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.saveRecord(OTransactionOptimistic.java:100)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:203)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:1)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:210)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:159)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:1)
at com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:1)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:685)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkSetToStream(ORecordSerializerCSVAbstract.java:411)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:306)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:242)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:77)
at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:131)
at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:126)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocalTxExecuter.commitEntry(OStorageLocalTxExecuter.java:230)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocalTxExecuter.commitAllPendingRecords(OStorageLocalTxExecuter.java:186)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.commit(OStorageLocal.java:670)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.executeCommit(ODatabaseRecordTx.java:217)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:159)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:1)
at com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:1)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:685)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkSetToStream(ORecordSerializerCSVAbstract.java:411)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:306)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:242)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:77)
at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:131)
at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.toStream(ORecordSchemaAwareAbstract.java:126)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocalTxExecuter.commitEntry(OStorageLocalTxExecuter.java:230)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocalTxExecuter.commitAllPendingRecords(OStorageLocalTxExecuter.java:186)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.commit(OStorageLocal.java:670)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.executeCommit(ODatabaseRecordTx.java:217)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:42)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:111)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:1)
at com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.commit(ODatabaseRecordWrapperAbstract.java:122)
at com.tinkerpop.blueprints.pgm.impls.orientdb.OrientGraph.stopTransaction(OrientGraph.java:347)
at
org.nakeduml.nakeduml.tinker.runtime.NakedOrientGraph.stopTransaction(NakedOrientGraph.java:48)
at org.tinker.TestLargeSet2.test(TestLargeSet2.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
at org.tinker.TestLargeSet2.test(TestLargeSet2.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Thanks
Pieter
On 06/07/2011 17:04, Luca Garulli wrote:
Luca Garulli
Jul 11, 2011, 6:59:13 AM7/11/11
to orient-...@googlegroups.com
Hi Pieter,
on Saturday I've fixed a bug on hole recycling algorithm (now in SVN trunk). Now the database should be much smaller and hope this bug has been fixed as well (I can't reproduce it)
Lvc@
Pieter Martin
Jul 11, 2011, 3:27:29 PM7/11/11
to orient-...@googlegroups.com
Hi,
Great, I have not had chance to reproduce it using orient api, but will test it on the latest.
Thanks
Pieter
Great, I have not had chance to reproduce it using orient api, but will test it on the latest.
Thanks
Pieter
Pieter Martin
Jul 16, 2011, 5:59:19 AM7/16/11
to orient-...@googlegroups.com
Hi,
I have managed to redo the test using plain vertexes and edges. The db size now remains small and the test passes but it still executes slowly and increasingly so as the number of vertexes and edges increases.
The test takes 0:03:22.622 to execute.
The same test on Neo4j takes 11 seconds.
Here is the test
@Test
public void testPlain() {
StopWatch totalStopWatch = new StopWatch();
totalStopWatch.start();
I have managed to redo the test using plain vertexes and edges. The db size now remains small and the test passes but it still executes slowly and increasingly so as the number of vertexes and edges increases.
The test takes 0:03:22.622 to execute.
The same test on Neo4j takes 11 seconds.
Here is the test
@Test
public void testPlain() {
StopWatch totalStopWatch = new StopWatch();
totalStopWatch.start();
StopWatch stopWatch = new StopWatch();
stopWatch.start();
db.startTransaction();
Vertex god = db.addVertex(null);
god.setProperty("name", "didthiswork");
Vertex auditGod = db.addVertex(null);
auditGod.setProperty("name", "didthiswork");
god.setProperty("name", "didthiswork");
Vertex auditGod = db.addVertex(null);
auditGod.setProperty("name", "didthiswork");
for (int i = 0; i < 5000; i++) {
if (i % 100 == 0) {
db.stopTransaction(Conclusion.SUCCESS);
db.startTransaction();
System.out.println(i);
stopWatch.stop();
System.out.println(stopWatch.toString());
stopWatch.reset();
stopWatch.start();
}
Vertex universe = db.addVertex(null);
universe.setProperty("name", "universe" + i);
Edge godUniverseEdge = GraphDb.getDb().addEdge(null, god, universe, "A__god___universe_");
godUniverseEdge.setProperty("outClass", "God");
godUniverseEdge.setProperty("inClass", "Universe");
Vertex auditUniverse = db.addVertex(null);
auditUniverse.setProperty("name", "universe" + i);
Edge auditUniverseEdge = GraphDb.getDb().addEdge(null, universe, auditUniverse, "original");
auditUniverseEdge.setProperty("outClass", "Universe");
auditUniverseEdge.setProperty("inClass", "Universe");
Vertex blackHole1 = db.addVertex(null);
blackHole1.setProperty("name", "blackHole1_" + i);
Edge universeBlackHoleEdge1 = GraphDb.getDb().addEdge(null, universe, blackHole1, "A__universe_blackhole");
universeBlackHoleEdge1.setProperty("outClass", "Universe");
universeBlackHoleEdge1.setProperty("inClass", "BlackHole");
Vertex auditBlackHole1 = db.addVertex(null);
auditBlackHole1.setProperty("name", "blackHole1_" + i);
Edge auditBlackHoleEdge1 = GraphDb.getDb().addEdge(null, blackHole1, auditBlackHole1, "original");
auditBlackHoleEdge1.setProperty("outClass", "BlackHole");
auditBlackHoleEdge1.setProperty("inClass", "BlackHole");
Vertex blackHole2 = db.addVertex(null);
blackHole2.setProperty("name", "blackHole2_" + i);
Edge universeBlackHoleEdge2 = GraphDb.getDb().addEdge(null, universe, blackHole2, "A__universe_blackhole");
universeBlackHoleEdge2.setProperty("outClass", "Universe");
universeBlackHoleEdge2.setProperty("inClass", "BlackHole");
Vertex auditBlackHole2 = db.addVertex(null);
auditBlackHole2.setProperty("name", "blackHole2_" + i);
Edge auditBlackHoleEdge2 = GraphDb.getDb().addEdge(null, blackHole2, auditBlackHole2, "original");
auditBlackHoleEdge2.setProperty("outClass", "BlackHole");
auditBlackHoleEdge2.setProperty("inClass", "BlackHole");
Vertex blackHole3 = db.addVertex(null);
blackHole3.setProperty("name", "blackHole3_" + i);
Edge universeBlackHoleEdge3 = GraphDb.getDb().addEdge(null, universe, blackHole3, "A__universe_blackhole");
universeBlackHoleEdge3.setProperty("outClass", "Universe");
universeBlackHoleEdge3.setProperty("inClass", "BlackHole");
Vertex auditBlackHole3 = db.addVertex(null);
auditBlackHole3.setProperty("name", "blackHole3_" + i);
Edge auditBlackHoleEdge3 = GraphDb.getDb().addEdge(null, blackHole3, auditBlackHole3, "original");
auditBlackHoleEdge3.setProperty("outClass", "BlackHole");
auditBlackHoleEdge3.setProperty("inClass", "BlackHole");
}
db.stopTransaction(Conclusion.SUCCESS);
stopWatch.stop();
System.out.println(stopWatch.toString());
totalStopWatch.stop();
System.out.println("Total time taken = " + totalStopWatch.toString());
}
The output...
0
0:00:00.004
100
0:00:01.405
200
0:00:01.258
300
0:00:01.005
400
0:00:00.969
500
0:00:00.859
600
0:00:00.998
700
0:00:01.047
800
0:00:01.112
900
0:00:01.323
1000
0:00:01.515
1100
0:00:01.597
1200
0:00:01.891
1300
0:00:01.901
1400
0:00:01.978
1500
0:00:02.169
1600
0:00:02.416
1700
0:00:02.641
1800
0:00:02.709
1900
0:00:02.771
2000
0:00:03.139
2100
0:00:03.184
2200
0:00:03.264
2300
0:00:03.343
2400
0:00:03.567
2500
0:00:03.752
2600
0:00:03.996
2700
0:00:04.073
2800
0:00:04.371
2900
0:00:04.358
3000
0:00:04.552
3100
0:00:04.703
3200
0:00:04.914
3300
0:00:05.021
3400
0:00:05.117
3500
0:00:05.365
3600
0:00:05.635
3700
0:00:05.987
3800
0:00:06.385
3900
0:00:06.457
4000
0:00:06.520
4100
0:00:06.539
4200
0:00:06.706
4300
0:00:06.999
4400
0:00:06.899
4500
0:00:06.962
4600
0:00:07.343
4700
0:00:07.567
4800
0:00:07.761
4900
0:00:08.387
0:00:08.173
Total time taken = 0:03:22.622
The same code on Neo4j takes 11 seconds to execute and there is no noticeable difference in the time taken for commits as the number of vertexes and edges increases.
My 5 cents...
At first I tried to execute all 5000 in one transaction. Some debugging showed that in ODatabaseRecordTx.executeCommit() the first line getStorage().commit(currentTx); is quite fast and then the next bit to do with indexes takes long to complete.
db.stopTransaction(Conclusion.SUCCESS);
db.startTransaction();
System.out.println(i);
stopWatch.stop();
System.out.println(stopWatch.toString());
stopWatch.reset();
stopWatch.start();
}
Vertex universe = db.addVertex(null);
universe.setProperty("name", "universe" + i);
Edge godUniverseEdge = GraphDb.getDb().addEdge(null, god, universe, "A__god___universe_");
godUniverseEdge.setProperty("outClass", "God");
godUniverseEdge.setProperty("inClass", "Universe");
Vertex auditUniverse = db.addVertex(null);
auditUniverse.setProperty("name", "universe" + i);
Edge auditUniverseEdge = GraphDb.getDb().addEdge(null, universe, auditUniverse, "original");
auditUniverseEdge.setProperty("outClass", "Universe");
auditUniverseEdge.setProperty("inClass", "Universe");
Vertex blackHole1 = db.addVertex(null);
blackHole1.setProperty("name", "blackHole1_" + i);
Edge universeBlackHoleEdge1 = GraphDb.getDb().addEdge(null, universe, blackHole1, "A__universe_blackhole");
universeBlackHoleEdge1.setProperty("outClass", "Universe");
universeBlackHoleEdge1.setProperty("inClass", "BlackHole");
Vertex auditBlackHole1 = db.addVertex(null);
auditBlackHole1.setProperty("name", "blackHole1_" + i);
Edge auditBlackHoleEdge1 = GraphDb.getDb().addEdge(null, blackHole1, auditBlackHole1, "original");
auditBlackHoleEdge1.setProperty("outClass", "BlackHole");
auditBlackHoleEdge1.setProperty("inClass", "BlackHole");
Vertex blackHole2 = db.addVertex(null);
blackHole2.setProperty("name", "blackHole2_" + i);
Edge universeBlackHoleEdge2 = GraphDb.getDb().addEdge(null, universe, blackHole2, "A__universe_blackhole");
universeBlackHoleEdge2.setProperty("outClass", "Universe");
universeBlackHoleEdge2.setProperty("inClass", "BlackHole");
Vertex auditBlackHole2 = db.addVertex(null);
auditBlackHole2.setProperty("name", "blackHole2_" + i);
Edge auditBlackHoleEdge2 = GraphDb.getDb().addEdge(null, blackHole2, auditBlackHole2, "original");
auditBlackHoleEdge2.setProperty("outClass", "BlackHole");
auditBlackHoleEdge2.setProperty("inClass", "BlackHole");
Vertex blackHole3 = db.addVertex(null);
blackHole3.setProperty("name", "blackHole3_" + i);
Edge universeBlackHoleEdge3 = GraphDb.getDb().addEdge(null, universe, blackHole3, "A__universe_blackhole");
universeBlackHoleEdge3.setProperty("outClass", "Universe");
universeBlackHoleEdge3.setProperty("inClass", "BlackHole");
Vertex auditBlackHole3 = db.addVertex(null);
auditBlackHole3.setProperty("name", "blackHole3_" + i);
Edge auditBlackHoleEdge3 = GraphDb.getDb().addEdge(null, blackHole3, auditBlackHole3, "original");
auditBlackHoleEdge3.setProperty("outClass", "BlackHole");
auditBlackHoleEdge3.setProperty("inClass", "BlackHole");
}
db.stopTransaction(Conclusion.SUCCESS);
stopWatch.stop();
System.out.println(stopWatch.toString());
totalStopWatch.stop();
System.out.println("Total time taken = " + totalStopWatch.toString());
}
The output...
0
0:00:00.004
100
0:00:01.405
200
0:00:01.258
300
0:00:01.005
400
0:00:00.969
500
0:00:00.859
600
0:00:00.998
700
0:00:01.047
800
0:00:01.112
900
0:00:01.323
1000
0:00:01.515
1100
0:00:01.597
1200
0:00:01.891
1300
0:00:01.901
1400
0:00:01.978
1500
0:00:02.169
1600
0:00:02.416
1700
0:00:02.641
1800
0:00:02.709
1900
0:00:02.771
2000
0:00:03.139
2100
0:00:03.184
2200
0:00:03.264
2300
0:00:03.343
2400
0:00:03.567
2500
0:00:03.752
2600
0:00:03.996
2700
0:00:04.073
2800
0:00:04.371
2900
0:00:04.358
3000
0:00:04.552
3100
0:00:04.703
3200
0:00:04.914
3300
0:00:05.021
3400
0:00:05.117
3500
0:00:05.365
3600
0:00:05.635
3700
0:00:05.987
3800
0:00:06.385
3900
0:00:06.457
4000
0:00:06.520
4100
0:00:06.539
4200
0:00:06.706
4300
0:00:06.999
4400
0:00:06.899
4500
0:00:06.962
4600
0:00:07.343
4700
0:00:07.567
4800
0:00:07.761
4900
0:00:08.387
0:00:08.173
Total time taken = 0:03:22.622
The same code on Neo4j takes 11 seconds to execute and there is no noticeable difference in the time taken for commits as the number of vertexes and edges increases.
My 5 cents...
At first I tried to execute all 5000 in one transaction. Some debugging showed that in ODatabaseRecordTx.executeCommit() the first line getStorage().commit(currentTx); is quite fast and then the next bit to do with indexes takes long to complete.
Thanks
Pieter
On 11/07/2011 12:59, Luca Garulli wrote:
Luca Garulli
Jul 16, 2011, 9:40:42 AM7/16/11
to orient-...@googlegroups.com
Hi Pieter,
I've just executed this test on my notebook with just 2 changes:
- declared the Massive Intent
- don't used transactions. I've created a flag in order to see the difference when use TX or not in OrientDB
The last one is much more important when you do a massive insertion, because (in the 99% of cases) you don't really that a block is atomic. AFAIK Neo4J always works in transactions, so probably it's the default for it.
This is the end of my output (my StopWatch class dump the ms).
4900
12426 ms
12665 ms
Total time taken = 12665 ms
Total vertices = 40002, total edges= 40000
Conclusion: 12,6 seconds not so bad.
Lvc@
Pieter Martin
Jul 16, 2011, 10:54:20 AM7/16/11
to orient-...@googlegroups.com
Great, thanks
Without transactions the test is fast on my machine also.
Just to clarify, transactioning has a very big impact on performance in orientdb!?
In general we wrap every request (web/jms/timers/whatever) to the system in a transaction and on special bulk inserts we would manage the transactions manually.
Coming from a JPA world I am used to executing everything always inside a transaction context.
Will we have to change our behavior and only use transactions where absolutely necessary or do you foresee transactioned persistence becoming faster later?
Thanks
Pieter
Without transactions the test is fast on my machine also.
Just to clarify, transactioning has a very big impact on performance in orientdb!?
In general we wrap every request (web/jms/timers/whatever) to the system in a transaction and on special bulk inserts we would manage the transactions manually.
Coming from a JPA world I am used to executing everything always inside a transaction context.
Will we have to change our behavior and only use transactions where absolutely necessary or do you foresee transactioned persistence becoming faster later?
Thanks
Pieter
Luca Garulli
Jul 16, 2011, 11:00:09 AM7/16/11
to orient-...@googlegroups.com
Hi,
in general transactions have an overhead because OrientDB has to manage a lot of things. When you work using the "remote" protocol (open a database against a remote OrientDB Server) performance could be better with TX than NO-TX. The reason is that the remote protocol bufferizes changes to shoot at commit time.
Release 1.0rc4 aims to fix the remaining bugs and improve transaction safety. While with indexes I know we've large edges of improvement, with transactions I don't know. Or at least with 1.0.
Lvc@
Pieter Martin
Jul 16, 2011, 11:11:18 AM7/16/11
to orient-...@googlegroups.com
Thanks, I'll give remote a try tho I have a strong preference for
the embedded mode.
Looking forward to all future releases.
Pieter
Looking forward to all future releases.
Pieter
Pieter Martin
Jul 29, 2011, 1:30:57 PM7/29/11
to orient-...@googlegroups.com
Hi,
Just a heads up on the orientdb's usability for us at present.
In our app and in almost all apps I have ever worked on all interaction happens in a transaction. Running without transactions is not an option for us. For now orientdb is to slow to be usable when running in transaction mode.
That said I am sure things will improve as it is a great project.
Thanks
Pieter
Just a heads up on the orientdb's usability for us at present.
In our app and in almost all apps I have ever worked on all interaction happens in a transaction. Running without transactions is not an option for us. For now orientdb is to slow to be usable when running in transaction mode.
That said I am sure things will improve as it is a great project.
Thanks
Pieter
Reply all
Reply to author
Forward
0 new messages