I've gotten my code so that it behaves correctly
on small database sizes. I now tried to load
(again) a data set with 13492 vertices and 35137
links from and RDF file. The RDF load to memory
using the Jena RDF library takes perhaps 10
seconds or less.
As the data is loaded into the database the links
increase. The heap limit is set to 4Gbytes
which it reaches and saturates.
It gets about 6800 elements loaded in about 44
seconds and then the links build up reaching over
24000 before the GC Limit is exceeded.
The data in the generated database is about
500Mbytes when the GC Limit is reached.
I'm keeping a copy of the vertexes in memory
because the linkage is nearly random. I can't
predict which vertex will be linked to by the next
vertex accessed.
If I query the database instead of keeping a copy,
the memory use goes up more slowly and it takes
about 240 seconds to load 6800 elements. It still
fails
with a GC overhead limit reached.
2011-03-08 01:53:34:174 WARN [] Low memory (2.37Gb
of 2.67Gb), calling listeners to free
memory...Size after free: 26025
Size after free: 4
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:30919 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #22
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 2 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at
java.lang.StringBuilder.toString(StringBuilder.java:430)
at
com.orientechnologies.orient.core.id.ORecordId.generateString(ORecordId.java:171)
at
com.orientechnologies.orient.core.id.ORecordId.toString(ORecordId.java:90)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:667)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:296)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
... 19 more
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:31109 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:333)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #10
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at
com.orientechnologies.orient.core.serialization.OBinaryProtocol.string2bytes(OBinaryProtocol.java:56)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
... 20 more
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:31272 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:333)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #1
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:497)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at
com.orientechnologies.common.util.OArrays.copyOfRange(OArrays.java:36)
at
com.orientechnologies.orient.core.serialization.OMemoryInputStream.getAsByteArray(OMemoryInputStream.java:77)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.serializeNewKey(OMVRBTreeEntryPersistent.java:797)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.toStream(OMVRBTreeEntryPersistent.java:758)
at
com.orientechnologies.orient.core.record.impl.ORecordBytesLazy.toStream(ORecordBytesLazy.java:37)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:497)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:31401 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:333)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #1
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:497)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at
java.lang.StringBuilder.toString(StringBuilder.java:430)
at
com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.split(OStringSerializerHelper.java:346)
at
com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.split(OStringSerializerHelper.java:333)
at
com.orientechnologies.orient.core.id.ORecordId.fromString(ORecordId.java:206)
at
com.orientechnologies.orient.core.id.ORecordId.<init>(ORecordId.java:53)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldFromStream(ORecordSerializerCSVAbstract.java:97)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.fromString(ORecordSerializerSchemaAware2CSV.java:372)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.fromStream(ORecordSerializerStringAbstract.java:59)
at
com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.deserializeFields(ORecordSchemaAwareAbstract.java:135)
at
com.orientechnologies.orient.core.record.ORecordVirtualAbstract.checkForFields(ORecordVirtualAbstract.java:137)
at
com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.getSchemaClass(ORecordSchemaAwareAbstract.java:80)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.embeddedCollectionToStream(ORecordSerializerCSVAbstract.java:557)
at
com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerListRID.toStream(OStreamSerializerListRID.java:44)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.serializeNewValue(OMVRBTreeEntryPersistent.java:809)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.toStream(OMVRBTreeEntryPersistent.java:761)
at
com.orientechnologies.orient.core.record.impl.ORecordBytesLazy.toStream(ORecordBytesLazy.java:37)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record #6:31506 of class 'OGraphEdge'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:317)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:333)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:97)
Caused by:
com.orientechnologies.orient.core.exception.ODatabaseException:
Error on saving record in cluster #1
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:270)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryPersistent.flush2Record(OMVRBTreeEntryPersistent.java:862)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:88)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeEntryDatabase.save(OMVRBTreeEntryDatabase.java:37)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreePersistent.commitChanges(OMVRBTreePersistent.java:422)
at
com.orientechnologies.orient.core.type.tree.OMVRBTreeDatabaseLazySave.lazySave(OMVRBTreeDatabaseLazySave.java:64)
at
com.orientechnologies.orient.core.index.OIndexMVRBTreeAbstract.lazySave(OIndexMVRBTreeAbstract.java:307)
at
com.orientechnologies.orient.core.index.OPropertyIndexManager.onRecordAfterUpdate(OPropertyIndexManager.java:95)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:78)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:601)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:497)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
In the case of querying to get the vertex, we
still get the GC overhead limit exceeded, but the
stacktrace is a little different:
Exception in thread "main"
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
java.util.HashMap.newKeyIterator(HashMap.java:840)
at
java.util.HashMap$KeySet.iterator(HashMap.java:874)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.setDatabase(ORecordTrackedSet.java:142)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.setDatabase(ORecordLazySet.java:199)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.setDatabase(ORecordTrackedSet.java:144)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.setDatabase(ORecordLazySet.java:199)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.setDatabase(ORecordTrackedSet.java:144)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.setDatabase(ORecordLazySet.java:199)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.setDatabase(ORecordTrackedSet.java:144)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.setDatabase(ORecordLazySet.java:199)
at
com.orientechnologies.orient.core.record.impl.ODocument.setDatabase(ODocument.java:845)
at
com.orientechnologies.orient.core.cache.ODatabaseRecordCache.findRecord(ODatabaseRecordCache.java:103)
at
com.orientechnologies.orient.core.storage.OStorageEmbedded.browseCluster(OStorageEmbedded.java:155)
at
com.orientechnologies.orient.core.storage.OStorageEmbedded.browse(OStorageEmbedded.java:109)
at
com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.scanEntireClusters(OCommandExecutorSQLSelect.java:535)
at
com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:185)
at
com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:78)
at
com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:57)
at
com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:62)
at
com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:72)
at
com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:38)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.query(ODatabaseRecordAbstract.java:246)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.query(ODatabaseRecordWrapperAbstract.java:98)
at
dataManager.orientdb.BuildGraphDatabase.getNode(BuildGraphDatabase.java:259)
at
dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:308)
at
dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:104)
-Dave
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:462)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at
com.orientechnologies.orient.core.db.record.ORecordTrackedSet.add(ORecordTrackedSet.java:52)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.add(ORecordLazySet.java:66)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:308)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:232)
at
com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
at
com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:107)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
... 19 more
-Dave
Hi,
Many users faced the same problem about memory and GC. I'm working with the profiler to see what happens and if there's some memory leaks with indexes...
Lvc@
Il giorno 08/mar/2011 23:34, "David Forslund" <fors...@mail.com> ha scritto:
I'm running with the latest SVN 1.0rc1-SNAPSHOT (2426). Also when I turn off caching, the code does speed up reaching 6800 elements in 37 seconds. However, now after it reaches 25000+ links it fails with exceeding the heap size of 4Gbytes. The database size on disk reaches about 750Mbytes.
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:462)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at com.orientechnologies.orient.core.db.record.ORecordTrackedSet.add(ORecordTrackedSet.java:52)
at com.orientechnologies.orient.core.db.record.ORecordLazySet.add(ORecordLazySet.java:66)
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:308)
-Dave
at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSch...
On 3/8/2011 2:48 PM, David Forslund wrote:
>
> I'm running OrientDB on Vista 64bit with 12GBytes o...