Memory footprint of Orient-Lucene index

Jing Chen

unread,

Dec 29, 2014, 11:49:36 PM12/29/14

to orient-...@googlegroups.com

Hi there,

We are using OrientDb 1.7.10 with Orient-Lucene index. We only use lucene text index. We use Orientdb as an embedded server in our web application. The application ran out of memory a few times recently. Our log shows that it was because Lucene can't allocate more memory. Is there a way to estimate the memory footprint of Orientdb-lucene index? When our database grows, does the memory requirement of Lucene index grow as well?

Thanks advance for your answers.

Jing Chen

Enrico Risa

unread,

Dec 30, 2014, 2:52:50 AM12/30/14

to orient-...@googlegroups.com

Hi Jin Chen

can you tell me the configuration of your application?

Do you have some log or profiling session?

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jing Chen

unread,

Dec 30, 2014, 1:05:09 PM12/30/14

to orient-...@googlegroups.com, enric...@gmail.com

Thanks for the quick response Enrico. Our application is a tomcat web application running on a Amazon Redhat 7 instance which has 3.5G memory. We have 3G allocated to tomcat JVM. Here are the error message from the tomcat log:

2014-12-21 05:32:19:694 INFO Deleting partial network 31fdf743-88f5-11e4-a88d-0aa4c1de39d1 in order to rollback in response to error [NdexPersistenceService]Exception in thread "Lucene Merge Thread #11" org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded

at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)

at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.util.concurrent.FutureTask.report(Unknown Source)

at java.util.concurrent.FutureTask.get(Unknown Source)

at org.ndexbio.task.NdexTaskExecutor.call(NdexTaskExecutor.java:69)

at org.ndexbio.task.NdexTaskExecutor.call(NdexTaskExecutor.java:32)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

java.util.concurrent.ExecutionException: java.lang.Exception: Error occured when executing task. java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.util.concurrent.FutureTask.report(Unknown Source)

at java.util.concurrent.FutureTask.get(Unknown Source)

at org.ndexbio.task.NdexQueuedTaskProcessor.processQueuedTasks(NdexQueuedTaskProcessor.java:101)

at org.ndexbio.task.NdexQueuedTaskProcessor.processAll(NdexQueuedTaskProcessor.java:176)

at org.ndexbio.rest.services.AdminService.processTasks(AdminService.java:103)

at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)

at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)

at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)

at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)

at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)

at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)

at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)

at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)

at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)

at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

... 1 more

java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit

at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2842)

at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)

at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)

at com.orientechnologies.lucene.manager.OLuceneIndexManagerAbstract.commit(OLuceneIndexManagerAbstract.java:118)

at com.orientechnologies.lucene.OLuceneIndexEngine.closeDb(OLuceneIndexEngine.java:139)

at com.orientechnologies.orient.core.index.OIndexAbstract.onClose(OIndexAbstract.java:802)

at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callOnCloseListeners(ODatabaseRecordAbstract.java:1536)

at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTxPooled.close(ODatabaseDocumentTxPooled.java:112)

at org.ndexbio.common.access.NdexDatabase.close(NdexDatabase.java:82)

at org.ndexbio.rest.services.AdminService.processTasks(AdminService.java:109)