Thread Pool is maxing out.

[Sudharshini]

Hi,

I have been stuck with this issue for a couple of days. During our recent test, we found that the thread pools are getting maxed out.

max threads is set to 400. 

I have tried increasing the  default.repository.maxActive parameter, but the results are the same.

From the thread dumps ...

"http-bio-8080-exec-302" Id=492 in BLOCKED on lock=EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock@c3bf350

     owned by http-bio-8080-exec-336 Id=470

    at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(Unknown Source)

    at org.apache.jackrabbit.core.state.FineGrainedISMLocking.acquireReadLock(FineGrainedISMLocking.java:105)

    at org.apache.jackrabbit.core.state.SharedItemStateManager.acquireReadLock(SharedItemStateManager.java:1884)

    at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:267)

    at org.apache.jackrabbit.core.state.LocalItemStateManager.getPropertyState(LocalItemStateManager.java:135)

    at org.hippoecm.repository.jackrabbit.HippoLocalItemStateManager.getPropertyState(HippoLocalItemStateManager.java:536)

    at org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState(LocalItemStateManager.java:177)

      - locked org.hippoecm.repository.jackrabbit.HippoLocalItemStateManager@63cef1a1

    at org.apache.jackrabbit.core.state.XAItemStateManager.getItemState(XAItemStateManager.java:260)

    at org.hippoecm.repository.jackrabbit.HippoLocalItemStateManager.getItemState(HippoLocalItemStateManager.java:346)

    at org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:161)

    at org.apache.jackrabbit.core.ItemManager.getItemData(ItemManager.java:382)

    at org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:328)

    at org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:622)

      - locked org.hippoecm.repository.jackrabbit.ItemManager@f07b6c6

    at org.apache.jackrabbit.core.LazyItemIterator.prefetchNext(LazyItemIterator.java:122)

    at org.apache.jackrabbit.core.LazyItemIterator.next(LazyItemIterator.java:257)

    at org.hippoecm.repository.decorating.RangeIteratorDecorator.next(RangeIteratorDecorator.java:112)

    at org.hippoecm.repository.decorating.PropertyIteratorDecorator.nextProperty(PropertyIteratorDecorator.java:41)

    at org.hippoecm.hst.provider.jcr.JCRValueProviderImpl.populate(JCRValueProviderImpl.java:594)

    at org.hippoecm.hst.provider.jcr.JCRValueProviderImpl.getPropertyMap(JCRValueProviderImpl.java:425)

    at org.hippoecm.hst.provider.jcr.JCRValueProviderImpl.getProperties(JCRValueProviderImpl.java:416)

    at org.hippoecm.hst.content.beans.standard.HippoItem.getProperties(HippoItem.java:191)

    

 

"http-bio-8080-exec-302" Id=492 in TIMED_WAITING on lock=org.apache.commons.pool.impl.GenericObjectPool$Latch@58cae4ae

        at java.lang.Object.wait(Native Method)

        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1112)

        at org.hippoecm.hst.core.jcr.pool.BasicPoolingRepository.login(BasicPoolingRepository.java:326)

        at org.hippoecm.hst.core.jcr.pool.BasicPoolingRepository.login(BasicPoolingRepository.java:354)

        at org.hippoecm.hst.core.jcr.pool.LazyMultipleRepositoryImpl.login(LazyMultipleRepositoryImpl.java:182)

        at org.hippoecm.hst.core.jcr.pool.MultipleRepositoryImpl.login(MultipleRepositoryImpl.java:172)

        at org.hippoecm.hst.site.request.HstRequestContextImpl.getSession(HstRequestContextImpl.java:184)

        at org.hippoecm.hst.site.request.HstRequestContextImpl.getSession(HstRequestContextImpl.java:173)

        at org.hippoecm.hst.site.request.HstRequestContextImpl.getObjectBeanManager(HstRequestContextImpl.java:740)

        at org.hippoecm.hst.site.request.HstRequestContextImpl.getSiteContentBaseBean(HstRequestContextImpl.java:702)

        at org.onehippo.cms7.essentials.components.CommonComponent.getScopeBean(CommonComponent.java:213)

 

[Jeroen Reijn]

Hi Sudharshini,

if you max out of threads a couple of things could be happening:

1. You have a session leak somewhere. Somewhere in the code you get a session and you do not return that session. For most sessions from the pool you do not have to manually return it unless the API states otherwise. Persistable sessions require this if I remember correctly.

2. The thread count of your application container has a much higher maxThread count compared to the HST session pool. In essence, at least one session from the HST pool is used per request, so if you have 25 concurrent requests you will at least take 25 sessions from the pool. It seems you've already tried to increase the pool size. That should be sufficient in normal scenario's , but it might be that you have another problem, which is hard to tell from the information given above.

As a general side node giving mode threads at the container level will not always make your application go faster, it might make it even slower. You could also try to lower the maxThreads on the container, but give it a bigger queue size.

Cheers,

Jeroen

 

--
Hippo Community Group: The place for all discussions and announcements about Hippo CMS (and HST, repository etc. etc.)
 
To post to this group, send email to hippo-c...@googlegroups.com
RSS: https://groups.google.com/group/hippo-community/feed/rss_v2_0_msgs.xml?num=50
---
You received this message because you are subscribed to the Google Groups "Hippo Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hippo-communi...@googlegroups.com.
Visit this group at http://groups.google.com/group/hippo-community.
For more options, visit https://groups.google.com/d/optout.

--

Jeroen Reijn
Hippo

Amsterdam office - Oosteinde 11, 1017 WT Amsterdam

Boston office - 745 Atlantic Ave, Eight Floor, Boston MA 02111, United states of America.

US +1 877 414 4776 (toll free)
Europe +31(0)20 522 4466
www.onehippo.com

http://www.jeroenreijn.com | @jreijn | http://about.me/jeroenreijn

[Jeroen Reijn]

I was also wondering if this is happening during normal load or during a stress test?

ps. If you or your client has a support contract you might want to get in touch with us (Hippo), because this feels like it could require some further analysis to figure out what the cause of this is.

[Sudharshini]

Hi Jeroen,

This is happening during a load test. 

Regards,

Sudha

[Ard Schrijvers]

On Tue, Nov 3, 2015 at 9:59 AM, Sudharshini <n.sudh...@gmail.com> wrote:
>
> Hi Jeroen,
> This is happening during a load test.

How long does a single page take to render (say the tenth time it is
requested). If it takes too long, and you have many concurrent
requests, the HST will start lagging behind. Then you should
investigate whether your pages are too slow and where it comes from
(lot of searches, external server side calls, etc). Compare it with
the plain essentials created project (I can request then thousands of
pages per second with 50 concurrent threads with Apache Benchmark).
With HST page diagnostics (see at onehippo.org) you can find out which
parts take how long to execute

You can take a look at HST page cache. And you can search one
onehippo.org for performance improvements (for example bundle cache).
If for some reason you cannot make your pages faster (because for
example you rely on server side external rest call that take long),
then you can scale out horizontally

HTH,

Regards Ard

>
>
> Regards,
> Sudha
>
> On Tuesday, November 3, 2015 at 8:24:20 AM UTC, Jeroen Reijn wrote:
>>

>>> http://www.jeroenreijn.com | @jreijn | http://about.me/jeroenreijn
>>
>>
>>
>>
>> --
>> Jeroen Reijn
>> Hippo
>>
>> Amsterdam office - Oosteinde 11, 1017 WT Amsterdam
>> Boston office - 745 Atlantic Ave, Eight Floor, Boston MA 02111, United states of America.
>>
>> US +1 877 414 4776 (toll free)
>> Europe +31(0)20 522 4466
>> www.onehippo.com
>>
>>
>>
>> http://www.jeroenreijn.com | @jreijn | http://about.me/jeroenreijn
>
> --

> Hippo Community Group: The place for all discussions and announcements about Hippo CMS (and HST, repository etc. etc.)
>
> To post to this group, send email to hippo-c...@googlegroups.com
> RSS: https://groups.google.com/group/hippo-community/feed/rss_v2_0_msgs.xml?num=50
> ---
> You received this message because you are subscribed to the Google Groups "Hippo Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hippo-communi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/hippo-community.
> For more options, visit https://groups.google.com/d/optout.




--

Hippo Netherlands, Oosteinde 11, 1017 WT Amsterdam, Netherlands
Hippo USA, Inc.- 745 Atlantic Ave, Eight Floor, Boston MA 02111,