[OProfiler]java.lang.OutOfMemoryError: GC overhead limit exceeded

[Shivanandan Gupta]

Hi All,

I am trying to select 100,000 records from a class (class is having 1,000,000 records and each record is having 74 properties)  but I am getting out of memory GC overhead limit exceeded error. My normal query will be to select 1,000,000 records from the class and the class will have 70+ properties.

memory configuration used is given below in server.bat file

rem ORIENTDB MAXIMUM HEAP. USE SYNTAX -Xmx<memory>, WHERE <memory> HAS THE TOTAL MEMORY AND SIZE UNIT. EXAMPLE: -Xmx512m

set MAXHEAP=-Xmx6g

rem ORIENTDB MAXIMUM DISKCACHE IN MB, EXAMPLE 8192 FOR 8GB

set MAXDISKCACHE=-Dstorage.diskCache.bufferSize=40960

query executed is "select from abcd limit 100000"

2015-02-20 08:10:06:775 INFO  {db=eigenin1} [TIP] Query 'select from s_asset limit 100000' returned a result set with more than 10000 records. Reduce it to improve performance and reduce RAM used [OProfiler]java.lang.OutOfMemoryError: GC overhead limit exceeded

Dumping heap to java_pid3952.hprof ...

Dump file is incomplete: No space left on device

2015-02-20 08:14:36:759 INFO  {db=eigenin1} [TIP] Query 'select from s_asset limit 20000' returned a result set with more than 10000 records. Reduce it to improve performance and reduce RAM used [OProfiler]

2015-02-20 08:16:48:239 INFO  {db=eigenin1} [TIP] Query 'select from s_asset limit 30000' returned a result set with more than 10000 records. Reduce it to improve performance and reduce RAM used [OProfiler]

Thanks,

Shivanandan Gupta

[Shivanandan Gupta]

configuration of Xmx in server.bat is given below.

:doneSetArgs

if NOT exist "%CONFIG_FILE%" set CONFIG_FILE=%ORIENTDB_HOME%/config/orientdb-server-config.xml

set LOG_FILE=%ORIENTDB_HOME%/config/orientdb-server-log.properties

set WWW_PATH=%ORIENTDB_HOME%/www

set ORIENTDB_SETTINGS=-Dprofiler.enabled=true

set JAVA_OPTS_SCRIPT=-Xmx6g -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9

rem ORIENTDB MAXIMUM HEAP. USE SYNTAX -Xmx<memory>, WHERE <memory> HAS THE TOTAL MEMORY AND SIZE UNIT. EXAMPLE: -Xmx512m

set MAXHEAP=-Xmx6g

rem ORIENTDB MAXIMUM DISKCACHE IN MB, EXAMPLE 8192 FOR 8GB

set MAXDISKCACHE=-Dstorage.diskCache.bufferSize=40960

call %JAVA% -server %JAVA_OPTS% %MAXHEAP% %JAVA_OPTS_SCRIPT% %ORIENTDB_SETTINGS% %MAXDISKCACHE% -Djava.util.logging.config.file="%LOG_FILE%" -Dorientdb.config.file="%CONFIG_FILE%" -Dorientdb.www.path="%WWW_PATH%" -Dorientdb.build.number="UNKNOWN@r${buildNumber}; 2015-01-28 18:33:00+0000" -cp "%ORIENTDB_HOME%\lib\*;" %CMD_LINE_ARGS% com.orientechnologies.orient.server.OServerMain

[Keith Freeman]

You might be able to do it by allocating more heap and less disk cache, but really you should try to find a way to paginate your query (you'll probably never have enough memory to select 1M records).  There are several ways:

• use limit and @rid, e.g. "select * from widget limit 1000" then "select * from widget where @rid > RID limit 1000"  where RID is the max @rid from the previous query
• organize your data by field value, e.g. "select * from widget where color = 'blue'" then "select * from friom widget where color = 'red'"....
• organize your data into orient's clusters, which are a lot like other database systems partitions -- cluster by time & date or whatever works for your data set
• ...

[Shivanandan Gupta]

Hi Keith,

Thanks for your suggestion, actually the need here is to retrieve approx. 1 million records and it will be used by an application, so paginating will not help us.

the disk cache is high because we want to make plocal to behave as in-memory database with persistent data.

Thanks for your help.

-Shivanandan

[Andrey Lomakin]

HI,

Did you consider asynchronous SQL queries ?

When all results are sent on the fly to the client and do not collected on server side.

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards,
Andrey Lomakin.

[Shivanandan Gupta]

Thanks Andrey for your suggestion I will try this out.

-- Shivanandan Gupta

[Shivanandan Gupta]

Hi Andrey,

I tried Asynchronous select query using java but while connecting to the db it throwing exception given below, I am attaching the code which I used.

C:\Program Files\Java\jdk1.8.0_31\bin>java Async_query

Feb 24, 2015 12:20:19 PM com.orientechnologies.common.log.OLogManager log

INFO: OrientDB auto-config DISKCACHE=224MB (heap=13,890MB os=62,499MB disk=448MB)

Exception in thread "main" com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'localhost:2480/GratefulDeadConcerts' with mode=rw

        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:214)

        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:243)

        at Async_query.main(Async_query.java:18)

Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot open the storage 'GratefulDeadConcerts' because it does not exist in path: localhost:2480/GratefulDeadConcerts

        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:159)

        ... 2 more

Thanks,

--Shivanandan Gupta

[Shivanandan Gupta]

Hi Andrey,

I got success in writing the working code for async select and it worked as expected. Thanks for your suggestion. I am attaching java program to this thread so that any newbie can refer it for there work.

About the error which was saying that cannot access the database , what I have observer that if this error occur then shutdown the server and restart it , it will work fine.

Because of my program , opened the db at starting of the program and when any error comes in between of the program then the db is locked as the db.close(); command didnt ran as the program terminated before that.

Thank you.

--Shivanandan Gupta

[Andrey Lomakin]

Hi Emanuele,

Could you take a look on that ?