[OProfiler]java.lang.OutOfMemoryError: GC overhead limit exceeded

389 views
Skip to first unread message

Shivanandan Gupta

unread,
Feb 20, 2015, 3:30:33 AM2/20/15
to orient-...@googlegroups.com
Hi All,

I am trying to select 100,000 records from a class (class is having 1,000,000 records and each record is having 74 properties)  but I am getting out of memory GC overhead limit exceeded error. My normal query will be to select 1,000,000 records from the class and the class will have 70+ properties.

memory configuration used is given below in server.bat file

rem ORIENTDB MAXIMUM HEAP. USE SYNTAX -Xmx<memory>, WHERE <memory> HAS THE TOTAL MEMORY AND SIZE UNIT. EXAMPLE: -Xmx512m
set MAXHEAP=-Xmx6g
rem ORIENTDB MAXIMUM DISKCACHE IN MB, EXAMPLE 8192 FOR 8GB
set MAXDISKCACHE=-Dstorage.diskCache.bufferSize=40960



query executed is "select from abcd limit 100000"

2015-02-20 08:10:06:775 INFO  {db=eigenin1} [TIP] Query 'select from s_asset limit 100000' returned a result set with more than 10000 records. Reduce it to improve performance and reduce RAM used [OProfiler]java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid3952.hprof ...
Dump file is incomplete: No space left on device

2015-02-20 08:14:36:759 INFO  {db=eigenin1} [TIP] Query 'select from s_asset limit 20000' returned a result set with more than 10000 records. Reduce it to improve performance and reduce RAM used [OProfiler]
2015-02-20 08:16:48:239 INFO  {db=eigenin1} [TIP] Query 'select from s_asset limit 30000' returned a result set with more than 10000 records. Reduce it to improve performance and reduce RAM used [OProfiler]

Thanks,
Shivanandan Gupta

Shivanandan Gupta

unread,
Feb 20, 2015, 6:16:21 AM2/20/15
to orient-...@googlegroups.com
configuration of Xmx in server.bat is given below.

:doneSetArgs

if NOT exist "%CONFIG_FILE%" set CONFIG_FILE=%ORIENTDB_HOME%/config/orientdb-server-config.xml

set LOG_FILE=%ORIENTDB_HOME%/config/orientdb-server-log.properties
set WWW_PATH=%ORIENTDB_HOME%/www
set ORIENTDB_SETTINGS=-Dprofiler.enabled=true
set JAVA_OPTS_SCRIPT=-Xmx6g -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9

rem ORIENTDB MAXIMUM HEAP. USE SYNTAX -Xmx<memory>, WHERE <memory> HAS THE TOTAL MEMORY AND SIZE UNIT. EXAMPLE: -Xmx512m
set MAXHEAP=-Xmx6g
rem ORIENTDB MAXIMUM DISKCACHE IN MB, EXAMPLE 8192 FOR 8GB
set MAXDISKCACHE=-Dstorage.diskCache.bufferSize=40960

call %JAVA% -server %JAVA_OPTS% %MAXHEAP% %JAVA_OPTS_SCRIPT% %ORIENTDB_SETTINGS% %MAXDISKCACHE% -Djava.util.logging.config.file="%LOG_FILE%" -Dorientdb.config.file="%CONFIG_FILE%" -Dorientdb.www.path="%WWW_PATH%" -Dorientdb.build.number="UNKNOWN@r${buildNumber}; 2015-01-28 18:33:00+0000" -cp "%ORIENTDB_HOME%\lib\*;" %CMD_LINE_ARGS% com.orientechnologies.orient.server.OServerMain



Keith Freeman

unread,
Feb 20, 2015, 4:20:37 PM2/20/15
to orient-...@googlegroups.com
You might be able to do it by allocating more heap and less disk cache, but really you should try to find a way to paginate your query (you'll probably never have enough memory to select 1M records).  There are several ways:
  • use limit and @rid, e.g. "select * from widget limit 1000" then "select * from widget where @rid > RID limit 1000"  where RID is the max @rid from the previous query
  • organize your data by field value, e.g. "select * from widget where color = 'blue'" then "select * from friom widget where color = 'red'"....
  • organize your data into orient's clusters, which are a lot like other database systems partitions -- cluster by time & date or whatever works for your data set
  • ...

Shivanandan Gupta

unread,
Feb 23, 2015, 12:31:08 AM2/23/15
to orient-...@googlegroups.com
Hi Keith,

Thanks for your suggestion, actually the need here is to retrieve approx. 1 million records and it will be used by an application, so paginating will not help us.

the disk cache is high because we want to make plocal to behave as in-memory database with persistent data.

Thanks for your help.

-Shivanandan

Andrey Lomakin

unread,
Feb 23, 2015, 2:03:39 AM2/23/15
to orient-database
HI,
Did you consider asynchronous SQL queries ?
When all results are sent on the fly to the client and do not collected on server side.

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Best regards,
Andrey Lomakin.

Shivanandan Gupta

unread,
Feb 23, 2015, 2:14:10 AM2/23/15
to orient-...@googlegroups.com
Thanks Andrey for your suggestion I will try this out.

-- Shivanandan Gupta

Shivanandan Gupta

unread,
Feb 24, 2015, 7:26:26 AM2/24/15
to orient-...@googlegroups.com
Hi Andrey,

I tried Asynchronous select query using java but while connecting to the db it throwing exception given below, I am attaching the code which I used.

C:\Program Files\Java\jdk1.8.0_31\bin>java Async_query
Feb 24, 2015 12:20:19 PM com.orientechnologies.common.log.OLogManager log
INFO: OrientDB auto-config DISKCACHE=224MB (heap=13,890MB os=62,499MB disk=448MB)
Exception in thread "main" com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'localhost:2480/GratefulDeadConcerts' with mode=rw
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:214)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:243)
        at Async_query.main(Async_query.java:18)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot open the storage 'GratefulDeadConcerts' because it does not exist in path: localhost:2480/GratefulDeadConcerts
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:159)
        ... 2 more


Thanks,
--Shivanandan Gupta
Async_query.java

Shivanandan Gupta

unread,
Feb 26, 2015, 12:11:20 AM2/26/15
to orient-...@googlegroups.com
Hi Andrey,

I got success in writing the working code for async select and it worked as expected. Thanks for your suggestion. I am attaching java program to this thread so that any newbie can refer it for there work.

About the error which was saying that cannot access the database , what I have observer that if this error occur then shutdown the server and restart it , it will work fine.
Because of my program , opened the db at starting of the program and when any error comes in between of the program then the db is locked as the db.close(); command didnt ran as the program terminated before that.


Thank you.

--Shivanandan Gupta
Async_query.java

Andrey Lomakin

unread,
Feb 26, 2015, 2:52:51 AM2/26/15
to orient-database, Emanuele Tagliaferri
Hi Emanuele,
Could you take a look on that ?
Reply all
Reply to author
Forward
0 new messages