Server runs out of memory on bulk insert

[mike....@runmyprocess.com]

I am using orientdb version 1.5.1-Snapshot and perform a bulk insert operation (1.2 mio elements which are also linked with each other via one edge).

I added some debug statements to my application, also printing out the used time. After roughly 200k elements the server runs out of memory, what seems to indicate a memory leak. Has anyone observed a similar behavior before? I think running the server with more heap space is not really a solution, as it would just delay the failure... so what can I do?

Here is the output:

Created 0 vertices in 10 ms
Created 1000 vertices in 1479 ms
Created 2000 vertices in 1181 ms
Created 3000 vertices in 1163 ms
Created 4000 vertices in 1160 ms
Created 5000 vertices in 1165 ms
Created 6000 vertices in 1145 ms
Created 7000 vertices in 1183 ms
Created 8000 vertices in 1142 ms
Created 9000 vertices in 1147 ms
Created 10000 vertices in 1220 ms
Created 11000 vertices in 1141 ms
Created 12000 vertices in 1273 ms
Created 13000 vertices in 1274 ms
Created 14000 vertices in 1366 ms
Created 15000 vertices in 1170 ms
Created 16000 vertices in 1136 ms
Created 17000 vertices in 1145 ms
Created 18000 vertices in 1158 ms
Created 19000 vertices in 1144 ms
Created 20000 vertices in 1133 ms
Created 21000 vertices in 1150 ms
...
Created 179000 vertices in 1170 ms
Created 180000 vertices in 1517 ms
Created 181000 vertices in 1560 ms
Created 182000 vertices in 1568 ms
Created 183000 vertices in 1553 ms
Created 184000 vertices in 1182 ms
Created 185000 vertices in 1532 ms
Created 186000 vertices in 1558 ms
Created 187000 vertices in 1600 ms
Created 188000 vertices in 1583 ms
Created 189000 vertices in 1583 ms
Created 190000 vertices in 1574 ms
Created 191000 vertices in 1590 ms
Created 192000 vertices in 1602 ms
Created 193000 vertices in 1966 ms
Created 194000 vertices in 1612 ms
Created 195000 vertices in 1571 ms
Created 196000 vertices in 2014 ms
Created 197000 vertices in 1611 ms
Created 198000 vertices in 1636 ms
Created 199000 vertices in 1992 ms
Created 200000 vertices in 2029 ms
Created 201000 vertices in 2070 ms
Created 202000 vertices in 2039 ms
Created 203000 vertices in 2043 ms
Created 204000 vertices in 2087 ms
Created 205000 vertices in 2467 ms
Created 206000 vertices in 2105 ms
Created 207000 vertices in 2899 ms
Created 208000 vertices in 2571 ms
Created 209000 vertices in 2961 ms
Created 210000 vertices in 3517 ms
Created 211000 vertices in 3830 ms
Created 212000 vertices in 4193 ms
Created 213000 vertices in 5851 ms
Created 214000 vertices in 8321 ms
Created 215000 vertices in 28768 ms
Exception in thread "OrientDB MemoryWatchDog" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
    at java.lang.AbstractStringBuilder.append(Unknown Source)
    at java.lang.StringBuilder.append(Unknown Source)
    at java.util.logging.LogManager$LoggerContext.processParentHandlers(Unknown Source)
    at java.util.logging.LogManager$LoggerContext.addLocalLogger(Unknown Source)
    at java.util.logging.LogManager.addLogger(Unknown Source)
    at java.util.logging.LogManager.demandLogger(Unknown Source)
    at java.util.logging.Logger.demandLogger(Unknown Source)
    at java.util.logging.Logger.getLogger(Unknown Source)
    at com.orientechnologies.common.log.OLogManager.log(OLogManager.java:34)
    at com.orientechnologies.common.log.OLogManager.debug(OLogManager.java:59)
    at com.orientechnologies.orient.core.memory.OMemoryWatchDog.run(OMemoryWatchDog.java:127)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at com.orientechnologies.orient.core.serialization.OBinaryProtocol.bytes2string(OBinaryProtocol.java:209)
    at com.orientechnologies.orient.core.serialization.OBinaryProtocol.bytes2string(OBinaryProtocol.java:159)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.fromStream(ORecordSerializerStringAbstract.java:79)
    at com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.deserializeFields(ORecordSchemaAwareAbstract.java:150)
    at com.orientechnologies.orient.core.record.impl.ODocument.deserializeFields(ODocument.java:1386)
    at com.orientechnologies.orient.core.record.impl.ODocument.checkForFields(ODocument.java:1276)
    at com.orientechnologies.orient.core.record.impl.ODocument.rawField(ODocument.java:515)
    at com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:535)
    at com.tinkerpop.blueprints.impls.orient.OrientVertex.createLink(OrientVertex.java:573)
    at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:282)
    at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:202)
    at com.fujitsu.qaas.persistence.orientdb.TestScenario1.main(TestScenario1.java:23)

[mike....@runmyprocess.com]

One additional remark: I call commit() after the creation of 1000 elements.

[Luca Garulli]

Hi,

have you looked at: https://github.com/orientechnologies/orientdb/wiki/Performance-Tuning-Blueprints ?

Lvc@

On 9 August 2013 14:09, <mike....@runmyprocess.com> wrote:

One additional remark: I call commit() after the creation of 1000 elements.

--
 
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Yin Wang]

I have the same problem. I'm just using OrientGraph.addVertex to create about 600K vertices on a laptop with 8Gb of ram. It eats over 1.5G of memory before get to 300K. When running on a server, it used 24G of ram after inserting 5M vertices. It doesn't matter how often I call commit().

I'm sure there no place in my code using much memory. I looked at the page you pointed to but found nothing related to this issue. It looks like the importing code is leaking memory.

[Luca Garulli]

Hi Yin

what release are you using?

Seems that the cache has no limit on server side, can you post here the server.sh or server.bat script?

Lvc@

[Yin Wang]

I'm using the 1.5.1 graph version. I tried both the packaged version and the one built from github source.

The problem probably has nothing to do with the server settings. I'm just running a Java program using the API, just to generate the initial database from some existing data.

A minimal reproducer of this problem looks like this:

import com.tinkerpop.blueprints.Vertex;

import com.tinkerpop.blueprints.impls.orient.OrientGraph;

public class MemoryTest {

    public static void main(String[] args) {

        OrientGraph graph = new OrientGraph("local:test1");

        for (int i = 0; i < Integer.MAX_VALUE; i++) {

            Vertex v = graph.addVertex("class:Test");

            // set some properties just to make it grow faster

            // removing this, will has the same effect, although grows slower

            for (int x = 0; x < 100; x++) {

                v.setProperty("name" + x, "unknown" + x);

            }

            graph.commit();

        }

    }

}

Thank you.

-- Yin

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/9bYZObDxMJg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

[Yin Wang]

An additional observation is that the memory usage is almost always greater than the size of the database. So I guess it's not freeing up the buffer used for the transactions after the commits. Notice that a commit happen after each vertex creation.

-- Yin

[Luca Garulli]

Hi,

try using the massive insert intent: https://github.com/orientechnologies/orientdb/wiki/Performance-Tuning#massive-insertion

Or just disable Level1 and 2 caches.

Lvc@

[Yin Wang]

I added these lines, but it still behave the same.

        OGlobalConfiguration.CACHE_LEVEL1_ENABLED.setValue(false);

        OGlobalConfiguration.CACHE_LEVEL2_ENABLED.setValue(false);

        OrientGraph graph = new OrientGraph("local:test1");

        graph.getRawGraph().declareIntent( new OIntentMassiveInsert() );

        ...

-- Yin