Blocked threads in Hermes under load

[Si]

Hi All,

I am using Hermes as the ebms message gateway to exchange ebms messages.

Under load testing, I observed that 

1. it takes a lot of time for the SOAP interface to return the message_id. 

2. it takes a lot of time to send the message subsequently to the third party.

Can you guys pls help me to solve this slowness issue?

I am attaching the thread dumps here for reference.

When checking the thread dump, I found most of the threads blocked waiting to lock at 

org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:157)

"http-nio-7080-exec-200" daemon prio=10 tid=0x00007f9fd80ad800 nid=0x1e9 waiting for monitor entry [0x00007f9fb30ae000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:157)

- waiting to lock <0x0000000500b8e3c8> (a org.apache.commons.dbcp.datasources.SharedPoolDataSource)

at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:669)

at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:653)

at hk.hku.cecid.piazza.commons.dao.ds.DataSourceTransaction.begin(DataSourceTransaction.java:92)

at hk.hku.cecid.piazza.commons.dao.ds.DataSourceProcess.start(DataSourceProcess.java:84)

- locked <0x000000053b8d7378> (a hk.hku.cecid.ebms.spa.dao.MessageServerDataSourceDAO$2)

at hk.hku.cecid.ebms.spa.dao.MessageServerDataSourceDAO.storeOutboxMessage(MessageServerDataSourceDAO.java:93)

at hk.hku.cecid.ebms.spa.handler.OutboundMessageProcessor.storeOutgoingMessage(OutboundMessageProcessor.java:287)

at hk.hku.cecid.ebms.spa.handler.OutboundMessageProcessor.generateAndStoreEbxmlMessage(OutboundMessageProcessor.java:352)

at hk.hku.cecid.ebms.spa.handler.OutboundMessageProcessor.processOutgoingMessage(OutboundMessageProcessor.java:111)

at hk.hku.cecid.ebms.spa.handler.MessageServiceHandler.processOutboundMessage(MessageServiceHandler.java:283)

"http-nio-7080-exec-199" daemon prio=10 tid=0x00007f9fe40ad800 nid=0x1e8 waiting for monitor entry [0x00007f9fb31af000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:157)

- waiting to lock <0x0000000500b8e3c8> (a org.apache.commons.dbcp.datasources.SharedPoolDataSource)

at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:669)

at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:653)

at hk.hku.cecid.piazza.commons.dao.ds.DataSourceTransaction.begin(DataSourceTransaction.java:92)

at hk.hku.cecid.piazza.commons.dao.ds.DataSourceProcess.start(DataSourceProcess.java:84)

- locked <0x0000000506172ed0> (a hk.hku.cecid.piazza.commons.dao.ds.DataSourceQuery)

at hk.hku.cecid.piazza.commons.dao.ds.DataSourceDAO.executeQuery(DataSourceDAO.java:437)

at hk.hku.cecid.piazza.commons.dao.ds.DataSourceDAO.executeQuery(DataSourceDAO.java:416)

at hk.hku.cecid.piazza.commons.dao.ds.DataSourceDAO.find(DataSourceDAO.java:348)

at hk.hku.cecid.ebms.spa.dao.PartnershipDataSourceDAO.findPartnershipsByCPA(PartnershipDataSourceDAO.java:31)

at hk.hku.cecid.ebms.spa.dao.PartnershipDataSourceDAO.findPartnershipByCPA(PartnershipDataSourceDAO.java:37)

at hk.hku.cecid.ebms.spa.service.EbmsMessageSenderService.serviceRequested(EbmsMessageSenderService.java:91)

My outbox collector xml is as given below:

<parameters>

        <parameter name="execution-interval" value="50" />

    </parameters>

        <component id="task-list" name="Outbox Collector">

        <class>hk.hku.cecid.ebms.spa.task.OutboxCollector</class>

                <parameter name="max-thread-count" value="200" />

        </component>

My datasource connection is as given below:

<component id="daofactory" name="System DAO Factory">

        <class>hk.hku.cecid.piazza.commons.dao.ds.SimpleDSDAOFactory</class>

                <parameter name="driver" value="oracle.jdbc.OracleDriver"/>

                <parameter name="url" value="jdbc:oracle:thin:@xxxx:1523/ebmsdb"/>

                <parameter name="username" value="xxxx"/>

                <parameter name="password" value="xxxx"/>

        <parameter name="pooling" value="true"/>

        <parameter name="maxActive" value="200"/>

        <parameter name="maxIdle" value="60"/>

        <parameter name="maxWait" value="60"/>

        <parameter name="testOnBorrow" value="true"/>

        <parameter name="testOnReturn" value="false"/>

        <parameter name="testWhileIdle" value="false"/>

                <parameter name="validationQuery" value="select 1 from dual"/>

                <parameter name="config">hk/hku/cecid/ebms/spa/conf/ebms.oracle.dao.xml</parameter>

        </component>

Thanks a lot.

Si