Dataverse glassfish server GC overhead

[Michel Bamouni]

Hi,

I use dataverse 4.8.5 and every day, dataverse is not accessible for the raison  "Caused by: javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean: java.lang.OutOfMemoryError: GC overhead limit exceeded".
Below is the stackTrace:

Caused by: javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean
        at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionClientTx(EJBContainerTransactionManager.java:662)
        at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:507)
        at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4566)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2074)
        ... 99 more
Caused by: javax.ejb.EJBTransactionRolledbackException
        at com.sun.ejb.containers.BaseContainer.mapLocal3xException(BaseContainer.java:2342)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2123)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2044)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
        at com.sun.proxy.$Proxy339.findDirectlyContainingGroups(Unknown Source)
        at edu.harvard.iq.dataverse.authorization.groups.impl.explicit.__EJB31_Generated__ExplicitGroupServiceBean__Intf____Bean__.findDirectlyContainingGroups(Unknown Source)
        at edu.harvard.iq.dataverse.authorization.groups.GroupServiceBean.collectAncestors(GroupServiceBean.java:207)
        at sun.reflect.GeneratedMethodAccessor365.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
        at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
        at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
        ... 97 more
Caused by: javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionClientTx(EJBContainerTransactionManager.java:665)
        at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:507)
        at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4566)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2074)
        ... 131 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

To not block users, I need to restart glassfish.
Does someone mette this error and how can I solve it?

Thanks,

Michel

[Don Sizemore]

Hello,

How much memory does your server/VM have, and how much are you specifying for the JVM heap in domain.xml?

Donald

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/261fe49a-a318-4eb9-b115-5b0f132ba300%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michel Bamouni]

Hi Donald,

In my domain.xim, my xmx=1421m

      3882020 K total memory
      2618444 K used memory
      2462532 K active memory
       738380 K inactive memory
       170972 K free memory
       132664 K buffer memory

Michel

About physical memory, below the resulr of command : vmstat -s | grep memory"

To post to this group, send email to dataverse...@googlegroups.com.

[Don Sizemore]

Can you give the VM/server more memory? Especially if it's running Solr as well.

I personally wouldn't give a Glassfish/Solr less than 24GB of RAM with 16GB allocated to the JVM.

Donald

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/261fe49a-a318-4eb9-b115-5b0f132ba300%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/49f6353b-8404-44e1-8a45-9e367b62d242%40googlegroups.com.

[Michel Bamouni]

Solr does not work on this VM.

In addition, I am in production with two VM which have the same memory configurations. Dataverse work fine on one VM and trow GC overhead on another.
It's for this reason I don't understand the cause of the error.


I launch JVisualVM which result is join to this post

Michel

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/261fe49a-a318-4eb9-b115-5b0f132ba300%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

[Pete Meyer]

Michel,

Assuming (so please correct me if I'm wrong) that you're running the two dataverse/glassfish VMs for load balancing, one possibility is that the difference in memory usage is due to a user who's doing something memory intensive landing on one of the application servers (my first guess would be file upload / checksum or UNF calculation / ingenst; but that could be off-base).  It could also be related to harvesting timers running on one application server and not the other (re-indexing might be another possibility).

Best,
Pete

[Michel Bamouni]

Hi Peter,

I effectively load balancing between the two dataverses.
I will chek check what you suggest me to see if I can see the cause of my problem.

Thanks Peter

Michel

[Michel Bamouni]

Hi,

I analyse my memory occupation with ganglia and I see that cache take a lot space. I joint my ganglia view.
So I want to know, if dataverse 4.5.1 put somme data in cache? If true, what data and how can I manage these caches?

best regards,

Michel

[Pete Meyer]

Hi Michel,

I'd guess that this is the OS doing filesystem caching, possibly due to dataverse (or another processing) doing something I/O intensive. 

Best,

Pete  

[Michel Bamouni]

Hi,

I profiling my application using yourkit.
I observe that, a lot of time is spend in SearchServiceBean like in the join screenshot.
So I want to know how does work interaction between dataverse 4.5.1 and Solr?
In cluster with two dataverses and one solr, how does solr interact with the two dataverse?
To finish in cluster mode, are there a special configuration for solr?

My apologizes for mistakes beacause, I am french.

Best regards,

Michel

[Pete Meyer]

Hi Michel,

I don't believe that the interactions between solr and dataverse have changed between 4.5.1 and more recent releases.  The Dataverse application in glassfish (regardless of how many copies) communicate with solr over a REST API.  This happens at several points (which I'm probably not the ideal source of knowledge for): dataset creation, browsing datasets/dataverses, viewing pages, and searching.  My experience has been that if solr is down or having problems, dataverse usually has problems - which is why it's a required dataverse dependency.  As far as I'm aware, there isn't a need for any solr configuration to use multiple glassfish (dataverse) servers, other than the schema.xml changes needed for a single dataverse application server.  It was a little surprising to me the first time I saw a solr-related function when profiling something unrelated to search as well.

Hopefully that helps explain what's going on instead of confusing things further - but feel free to ask if it doesn't.

No need for apologizes - your English is far better than my French.

Best,

Pete

[Philip Durbin]

YourKit sounds interesting. And helpful. Pete is right about how Dataverse interacts with Solr. Basically, the PostgreSQL database is the "source of truth" and some of that data is indexed into Solr. I'm not sure how much detail you want but when datasets are created, for example, Solr "documents" are created for them. The home page of dataverse is a "star" or "*" (asterisk) query of the data in Solr, which provides the "facets" you see on the left.

Like Pete said, it should work fine to have multiple Glassfish servers configured to use the same Solr server. Harvard Dataverse does this in production.

I hope this helps. I wrote most of this code having to do with Solr and I'd be happy to answer more questions if you have them. It's interesting that you're seeing performance problems on the search side because more commonly we've seen performance problems with indexing. You can read up a little more about Solr at http://guides.dataverse.org/en/4.8.5/admin/solr-search-index.html

Thanks,

Phil

Pete

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/261fe49a-a318-4eb9-b115-5b0f132ba300%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/49f6353b-8404-44e1-8a45-9e367b62d242%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse-community@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8b8f7e26-eb7a-4499-b0d1-2675ddfe3f09%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin