on going menory issue

[John Fry]

Hi All,

can anyone suggest what to do to fix the following memory issue.

I simply need to iterate across all realtionships to reset one of their property values.

This is getting very frustrating now :( so any help is very much appreciated...


Regards John.



Graph nodes = 10,000,000
Realtionships = 100,000,000
graph.db directory size = 6.5GByte

Run with: -Xms4096m -Xmx8192m

public class SetWeights
{
   
    int i=0;
    int m=0;
    long startTime;
    long endTime;
   
    private static final String DB_PATH = "/usr/local/Cellar/neo4j/2.0.2/libexec/data/graph.db";

    public static void main( String[] args )
    {
        SetWeights setWeights = new SetWeights();
        setWeights.run();
    }

    void run()
    {
        GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( DB_PATH );

        //ExecutionEngine engine = new ExecutionEngine( db );
        System.out.println("DB Open");


        startTime = System.currentTimeMillis();
        try (Transaction tx = db.beginTx())
        {
            for ( Relationship rel : GlobalGraphOperations.at(db).getAllRelationships() ) {
                rel.setProperty("weight",0.5);
                i++;
                if (i>1000000) {
                    endTime = System.currentTimeMillis();
                    i=0;
                    m++;
                    System.out.println((m*1000000) + " weights set in " + (endTime-startTime) + "ms");
                    startTime = System.currentTimeMillis();
                    tx.success();
                }
            }
            tx.success();
        }
        System.out.println((m*1000000)+i);
       
        db.shutdown();
        System.out.println("DB Shutdown");
    }

}




OUTPUT -->


1000000 weights set in 12330ms
2000000 weights set in 19081ms
3000000 weights set in 29783ms
Exception in thread "JMX server connection timeout 16" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:64)
    at java.lang.StringBuilder.<init>(StringBuilder.java:85)
    at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.logtime(ServerCommunicatorAdmin.java:210)
    at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.access$400(ServerCommunicatorAdmin.java:32)
    at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:175)
    at java.lang.Thread.run(Thread.java:744)
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    at org.neo4j.kernel.impl.nioneo.store.PersistenceRow.<init>(PersistenceRow.java:47)
    at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:170)
    at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
    at org.neo4j.kernel.impl.nioneo.store.PropertyStore.getLightRecord(PropertyStore.java:284)
    at org.neo4j.kernel.impl.nioneo.store.PropertyStore.getPropertyRecordChain(PropertyStore.java:715)
    at org.neo4j.kernel.impl.api.store.DiskLayer.loadAllPropertiesOf(DiskLayer.java:457)
    at org.neo4j.kernel.impl.api.store.DiskLayer.relationshipGetAllProperties(DiskLayer.java:423)
    at org.neo4j.kernel.impl.api.store.CacheLayer$3.load(CacheLayer.java:100)
    at org.neo4j.kernel.impl.api.store.CacheLayer$3.load(CacheLayer.java:96)
    at org.neo4j.kernel.impl.core.Primitive.ensurePropertiesLoaded(Primitive.java:85)
    at org.neo4j.kernel.impl.core.Primitive.getProperties(Primitive.java:57)
    at org.neo4j.kernel.impl.api.store.PersistenceCache.relationshipGetProperties(PersistenceCache.java:167)
    at org.neo4j.kernel.impl.api.store.CacheLayer.relationshipGetAllProperties(CacheLayer.java:266)
    at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.relationshipGetAllProperties(StateHandlingStatementOperations.java:723)
    at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.relationshipGetProperty(StateHandlingStatementOperations.java:693)
    at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.relationshipSetProperty(StateHandlingStatementOperations.java:547)
    at org.neo4j.kernel.impl.api.ConstraintEnforcingEntityOperations.relationshipSetProperty(ConstraintEnforcingEntityOperations.java:160)
    at org.neo4j.kernel.impl.api.LockingStatementOperations.relationshipSetProperty(LockingStatementOperations.java:277)
    at org.neo4j.kernel.impl.api.OperationsFacade.relationshipSetProperty(OperationsFacade.java:474)
    at org.neo4j.kernel.impl.core.RelationshipProxy.setProperty(RelationshipProxy.java:247)
    at SetWeights.run(SetWeights.java:55)
    at SetWeights.main(SetWeights.java:40)

[Peter Hunsberger]

Add a commit every one and a while...

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Hunger]

Batch your transactions and commit (tx.success() and tx.close()) every 1M rels and create a new tx.

Sent from mobile device

--

[John Fry]

Still have the same issue - it ran a bit longer.
Code changes below.

i=0; m=0;
        startTime = System.currentTimeMillis();
        Transaction tx = db.beginTx();

        for ( Relationship rel : GlobalGraphOperations.at(db).getAllRelationships() ) {      

            i++;
            //rel.setProperty("weight",0.5);

            if (i>1000000) {
                endTime = System.currentTimeMillis();
                i=0;
                m++;
                System.out.println((m*1000000) + " weights set in " + (endTime-startTime) + "ms");
                startTime = System.currentTimeMillis();
                tx.success();

                tx.close();
                tx = db.beginTx();
            }
        }
        System.out.println((m*1000000+i) + " weights set in " + (endTime-startTime) + "ms");


1000000 weights set in 1225ms
2000000 weights set in 827ms
3000000 weights set in 1322ms
4000000 weights set in 792ms
5000000 weights set in 721ms
6000000 weights set in 3546ms
7000000 weights set in 1095ms
8000000 weights set in 870ms
9000000 weights set in 994ms
10000000 weights set in 1057ms
11000000 weights set in 734ms
12000000 weights set in 1144ms
13000000 weights set in 6488ms
14000000 weights set in 702ms
15000000 weights set in 959ms
16000000 weights set in 977ms
17000000 weights set in 8227ms
18000000 weights set in 3292ms
19000000 weights set in 3325ms
20000000 weights set in 3435ms
21000000 weights set in 3486ms
22000000 weights set in 155231ms
23000000 weights set in 1057ms
24000000 weights set in 8230ms
25000000 weights set in 11331ms
26000000 weights set in 27733ms
27000000 weights set in 8641ms
28000000 weights set in 19007ms
29000000 weights set in 61770ms

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

    at java.lang.Long.valueOf(Long.java:577)

[John Fry]

     
Hi All, fixed now.

turn caches off & make sure you load the neo4j.properties !!!!

  GraphDatabaseService db = new GraphDatabaseFactory()
            .newEmbeddedDatabaseBuilder( DB_PATH )
            .loadPropertiesFromFile( CONF_PATH )
            .newGraphDatabase();

for 1M nodes & 100M relationships I used:

neostore.nodestore.db.mapped_memory=200M
neostore.relationshipstore.db.mapped_memory=3500M
neostore.propertystore.db.mapped_memory=2500M
neostore.propertystore.db.strings.mapped_memory=300M
neostore.propertystore.db.arrays.mapped_memory=100M
cache_type=none

[Michael Hunger]

Right, sorry, I wanted to reply that (cache_type=none) but got distracted with the family afternoon. Good point.

Just make sure you have enough RAM in total, I saw you had 20GB heap used, on Linux and OSX the MMIO resides outside of the heap on windows on the heap.

Michael

--

[John Fry]

Hi Michael,

it performs consistently now.

Thx, JF