Re: OrientDB Scaling

106 views
Skip to first unread message

David Forslund

unread,
Mar 8, 2011, 4:48:34 PM3/8/11
to orient-database
I'm running OrientDB on Vista 64bit with 12GBytes
of memory on a 2.6GHz Intel i7920 and jdk1.6.0_22

I've gotten my code so that it behaves correctly
on small database sizes. I now tried to load
(again) a data set with 13492 vertices and 35137
links from and RDF file. The RDF load to memory
using the Jena RDF library takes perhaps 10
seconds or less.

As the data is loaded into the database the links
increase. The heap limit is set to 4Gbytes
which it reaches and saturates.
It gets about 6800 elements loaded in about 44
seconds and then the links build up reaching over
24000 before the GC Limit is exceeded.
The data in the generated database is about
500Mbytes when the GC Limit is reached.

I'm keeping a copy of the vertexes in memory
because the linkage is nearly random. I can't
predict which vertex will be linked to by the next
vertex accessed.
If I query the database instead of keeping a copy,
the memory use goes up more slowly and it takes
about 240 seconds to load 6800 elements. It still
fails
with a GC overhead limit reached.

2011-03-08 01:53:34:174 WARN [] Low memory (2.37Gb
of 2.67Gb), calling listeners to free
memory...Size after free: 26025
Size after free: 4
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:30919 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #22
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 2 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at
java.lang.StringBuilder.toString(StringBuilder.java:430)
at
com.orientechnologies.orient.core.id.ORecordId.generateString(ORecordId.java:171)
at
com.orientechnologies.orient.core.id.ORecordId.toString(ORecordId.java:90)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:667)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:296)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
... 19 more
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:31109 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:333)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #10
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at
com.orientechnologies.orient.core.serialization.OBinaryProtocol.string2bytes(OBinaryProtocol.java:56)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
... 20 more
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:31272 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:333)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #1
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:497)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at
com.orientechnologies.common.util.OArrays.copyOfRange(OArrays.java:36)
at
com.orientechnologies.orient.core.serialization.OMemoryInputStream.getAsByteArray(OMemoryInputStream.java:77)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.serializeNewKey(OMVRBTreeEntryPersistent.java:797)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.toStream(OMVRBTreeEntryPersistent.java:758)
at
com.orientechnologies.orient.core.record.impl.ORecordBytesLazy.toStream(ORecordBytesLazy.java:37)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:497)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:31401 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:333)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #1
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:497)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at
java.lang.StringBuilder.toString(StringBuilder.java:430)
at
com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.split(OStringSerializerHelper.java:346)
at
com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.split(OStringSerializerHelper.java:333)
at
com.orientechnologies.orient.core.id.ORecordId.fromString(ORecordId.java:206)
at
com.orientechnologies.orient.core.id.ORecordId.<init>(ORecordId.java:53)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldFromStream(ORecordSerializerCSVAbstract.java:97)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.fromString(ORecordSerializerSchemaAware2CSV.java:372)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.fromStream(ORecordSerializerStringAbstract.java:59)
at
com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.deserializeFields(ORecordSchemaAwareAbstract.java:135)
at
com.orientechnologies.orient.core.record.ORecordVirtualAbstract.checkForFields(ORecordVirtualAbstract.java:137)
at
com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.getSchemaClass(ORecordSchemaAwareAbstract.java:80)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.embeddedCollectionToStream(ORecordSerializerCSVAbstract.java:557)
at
com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerListRID.toStream(OStreamSerializerListRID.java:44)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.serializeNewValue(OMVRBTreeEntryPersistent.java:809)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.toStream(OMVRBTreeEntryPersistent.java:761)
at
com.orientechnologies.orient.core.record.impl.ORecordBytesLazy.toStream(ORecordBytesLazy.java:37)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:31506 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:333)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #1
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:497)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded

In the case of querying to get the vertex, we
still get the GC overhead limit exceeded, but the
stacktrace is a little different:
Exception in thread "main"
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
java.util.HashMap.newKeyIterator(HashMap.java:840)
at
java.util.HashMap$KeySet.iterator(HashMap.java:874)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.setDatabase(ORecordTrackedSet.java:142)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.setDatabase(ORecordLazySet.java:199)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.setDatabase(ORecordTrackedSet.java:144)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.setDatabase(ORecordLazySet.java:199)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.setDatabase(ORecordTrackedSet.java:144)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.setDatabase(ORecordLazySet.java:199)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.setDatabase(ORecordTrackedSet.java:144)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.setDatabase(ORecordLazySet.java:199)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.cache.ODatabaseRecordCache.findRecord(ODatabaseRecordCache.java:103)
at
com.orientechnologies.orient.core.storage.OStorageEmbedded.browseCluster(OStorageEmbedded.java:155)
at
com.orientechnologies.orient.core.storage.OStorageEmbedded.browse(OStorageEmbedded.java:109)
at
com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.scanEntireClusters(OCommandExecutorSQLSelect.java:535)
at
com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:185)
at
com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:78)
at
com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:57)
at
com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:62)
at
com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:72)
at
com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:38)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.query(ODatabaseRecordAbstract.java:246)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.query(ODatabaseRecordWrapperAbstract.java:98)
at
dataManager.orientdb.BuildGraphDatabase.getNode(BuildGraphDatabase.java:259)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:308)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:104)

-Dave

David Forslund

unread,
Mar 8, 2011, 5:34:19 PM3/8/11
to orient-...@googlegroups.com
I'm running with the latest SVN 1.0rc1-SNAPSHOT
(2426). Also when I turn off caching, the code
does speed up reaching 6800 elements in 37
seconds. However, now after it reaches 25000+
links it fails with exceeding the heap size of
4Gbytes. The database size on disk reaches about
750Mbytes.

Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:462)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.add(ORecordTrackedSet.java:52)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.add(ORecordLazySet.java:66)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:308)


at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
... 19 more

-Dave

Luca Garulli

unread,
Mar 8, 2011, 5:39:28 PM3/8/11
to orient-...@googlegroups.com

Hi,
Many users faced the same problem about memory and GC. I'm working with the profiler to see what happens and if there's some memory leaks with indexes...

Lvc@

Il giorno 08/mar/2011 23:34, "David Forslund" <fors...@mail.com> ha scritto:

I'm running with the latest SVN 1.0rc1-SNAPSHOT (2426).    Also when I turn off caching, the code does speed up reaching 6800 elements in 37 seconds.   However, now after it reaches 25000+ links it fails with exceeding the heap size of 4Gbytes.   The database size on disk reaches about 750Mbytes.

Caused by: java.lang.OutOfMemoryError: Java heap space
   at java.util.HashMap.resize(HashMap.java:462)
   at java.util.HashMap.addEntry(HashMap.java:755)
   at java.util.HashMap.put(HashMap.java:385)
   at com.orientechnologies.orient.core.db.record.ORecordTrackedSet.add(ORecordTrackedSet.java:52)
   at com.orientechnologies.orient.core.db.record.ORecordLazySet.add(ORecordLazySet.java:66)
   at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:308)


   at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSch...

-Dave



On 3/8/2011 2:48 PM, David Forslund wrote:
>

> I'm running OrientDB on Vista 64bit with 12GBytes o...

David Forslund

unread,
Mar 8, 2011, 5:49:26 PM3/8/11
to orient-...@googlegroups.com
When I up the maximum heap size it seems to progress further along before it fails.     I know my data set is rather nasty (large number of links per vertex).  It also gives trouble to our Hibernate implementation but does work.    I expect OrientDB to be able to deal with this well, once some of the kinks are worked out.   I don't know if I might be able to load all the vertices and the make a second pass linking them up, but I was hoping not to have to do that.

Thanks,

Dave
Reply all
Reply to author
Forward
0 new messages