1.7-rc2 Memory Issue -- Unexpected loss of data

[Shishya]

Hi,

After hours of work, I could run my database to normal. But when I left it to run with some processing functions for 3-4 hours, it closed with @Unable to allocate space" exception.

Can you please let me know if there is memory leakage issue.

My batch script is as follows

connect remote:localhost/colls root password
jss var g = orient.getGraph();


var r = g.command( "SQL","select count(*)  as count from mynode" );
var len = r[0].getRecord().field("count"); //Total in count
var j = 1000;
        len = 10;
var l = 0;
while(l < len){
        var r = g.command("SQL","Select from patents skip " +l+" LIMIT 1");
        // Do the job then go to Next
l++;
}

Content of record #11:43457 was broken.

-> com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:242)

-> com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:768)

-> com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:64)

-> com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:216)

-> com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:39)

-> com.orientechnologies.orient.core.iterator.OIdentifiableIterator.readCurrentRecord(OIdentifiableIterator.java:285)

-> com.orientechnologies.orient.core.iterator.ORecordIteratorCluster.hasNext(ORecordIteratorCluster.java:140)

-> com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:541)

-> com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:491)

-> com.orientechnologies.orient.core.index.OIndexAbstractDelegate.rebuild(OIndexAbstractDelegate.java:170)

-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLRebuildIndex.execute(OCommandExecutorSQLRebuildIndex.java:77)

-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:57)

-> com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:96)

-> com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:85)

-> com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:59)

-> com.orientechnologies.orient.console.OConsoleDatabaseApp.sqlCommand(OConsoleDatabaseApp.java:1886)

-> com.orientechnologies.orient.console.OConsoleDatabaseApp.rebuildIndex(OConsoleDatabaseApp.java:903)

-> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

-> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

-> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

-> java.lang.reflect.Method.invoke(Method.java:606)

-> com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:302)

-> com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:171)

-> com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:94)

-> com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:54)

[Andrey Lomakin]

Hi,

Could you send me your database too ?

It is very important and I will fix it during 3 days.

--
 
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[Shishya]

My Db size is in GBs. Should I share the schema?

Thanks

[Andrey Lomakin]

No,

I need to reproduce issue, if it is up to 10GB I can take it in dropbox, is it doable ?

[Andrey Lomakin]

BTW 

Unable to allocate space exception is it from file system ?

Which space system unable to allocate ?

[prabhat]

Hi,

Initial part of erro log is like this.

#

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 40894464 bytes for committing reserved memory.

# Possible reasons:

#   The system is out of physical RAM or swap space

#   In 32 bit mode, the process size limit was hit

# Possible solutions:

#   Reduce memory load on the system

#   Increase physical memory or swap space

#   Check if swap backing store is full

#   Use 64 bit Java on a 64 bit OS

#   Decrease Java heap size (-Xmx/-Xms)

#   Decrease number of Java threads

#   Decrease Java thread stack sizes (-Xss)

#   Set larger code cache with -XX:ReservedCodeCacheSize=

# This output file may be truncated or incomplete.

#

#  Out of Memory Error (os_linux.cpp:2726), pid=19994, tid=140209769162496

#

# JRE version: Java(TM) SE Runtime Environment (7.0_51-b13) (build 1.7.0_51-b13)

# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode linux-amd64 compressed oops)

# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#

---------------  T H R E A D  ---------------

Current thread (0x00007f8528073800):  VMThread [stack: 0x00007f85216c2000,0x00007f85217c3000] [id=20004]

Stack: [0x00007f85216c2000,0x00007f85217c3000],  sp=0x00007f85217c1220,  free space=1020k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V  [libjvm.so+0x992f4a]  VMError::report_and_die()+0x2ea

V  [libjvm.so+0x4931ab]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b

V  [libjvm.so+0x81338e]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe

V  [libjvm.so+0x81383f]  os::Linux::commit_memory_impl(char*, unsigned long, unsigned long, bool)+0x4f

V  [libjvm.so+0x813a2c]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0xc

V  [libjvm.so+0x80daea]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a

V  [libjvm.so+0x87fcd3]  PSVirtualSpace::expand_by(unsigned long)+0x53

V  [libjvm.so+0x8810a8]  PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8

V  [libjvm.so+0x880212]  PSYoungGen::resize(unsigned long, unsigned long)+0x22

V  [libjvm.so+0x87d6ab]  PSScavenge::invoke_no_policy()+0xf3b

V  [libjvm.so+0x87de11]  PSScavenge::invoke()+0x41

V  [libjvm.so+0x834f10]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70

V  [libjvm.so+0x9947e7]  VM_ParallelGCFailedAllocation::doit()+0x97

V  [libjvm.so+0x99bc15]  VM_Operation::evaluate()+0x55

V  [libjvm.so+0x99a04a]  VMThread::evaluate_operation(VM_Operation*)+0xba

                                                                                                                                                       1,1           Top

You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/bTCjSRNuKCw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

[Andrey Lomakin]

HI,

I would be really appreciate if you send me your db so I will reproduce error.

Is it possible ?

[Andrey Lomakin]

I mean I will manage to find space for database but is it possible according to your rules ?

Also could you provide your test along with database ?

[Shishya]

I could not share the database as per our rules. 

I can provide any kind of test logs though.

[Andrey Lomakin]

Can you create series of memory snapshots to track memory status during test ?

[Andrey Lomakin]

Will be really appreciate for this.

[prabhat]

Hi Andrey,

I am myself very keen on resolving it. But in limitations with corporate.

PFA error log for this.

Many Regards

[Andrey Lomakin]

Ok.

Could you describe what are you doing in this use case, do you read data or you read and change data ?

[Shishya]

I am doing read, then insert or update as per the processing logic.

read i cluster and update 1 or more of other 5.

[Andrey Lomakin]

Hi,

OK lets start with following could you use noTx db  version and disable wal by using setting storage.useWAL=flase and repeat test ?

[Andrey Lomakin]

Hi,

Do  you have any updates about this issue, can we continue conversation in this issue https://github.com/orientechnologies/orientdb/issues/2116

[Shishya]

Hi Andrey,

As suggested by Luca, I am using a POJO code for importing data from flat exported Mongodb dump.

That error was due to previous hack we discussed in that discussion.

It doesnt appear anymore

Thanks

[Andrey Lomakin]

Thank you.

I need to receive confirmation from one more user (they experienced this problem too) and after that I will close the bug.

For more options, visit https://groups.google.com/d/optout.