Blocked threads in Hermes 2

42 views
Skip to first unread message

Si

unread,
Aug 7, 2017, 4:43:57 PM8/7/17
to Hermes 2+ Community Forum

Hi All,

I am using Hermes as a ebms message gateway. When under load, I observed slowness at

1. Sending messages to Hermes using the SOAP sender. The SOAP  Sender takes around 40 seconds to return the message_id.
2. After the message_id is returned, it takes around 2-3 minutes to send the message to the third party B2B server.

When I observed the thread dumps, I observed a lot of blocked threads.

Also  I have given the collection configurations and the datasource configurations. Pls help why this slowness is there.


The blocked threads are observed here:

"http-nio-7080-exec-199" daemon prio=10 tid=0x00007f9fe40ad800 nid=0x1e8 waiting for monitor entry [0x00007f9fb31af000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:157)
- waiting to lock <0x0000000500b8e3c8> (a org.apache.commons.dbcp.datasources.SharedPoolDataSource)
at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:669)
at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:653)
at hk.hku.cecid.piazza.commons.dao.ds.DataSourceTransaction.begin(DataSourceTransaction.java:92)
at hk.hku.cecid.piazza.commons.dao.ds.DataSourceProcess.start(DataSourceProcess.java:84)
- locked <0x0000000506172ed0> (a hk.hku.cecid.piazza.commons.dao.ds.DataSourceQuery)
at hk.hku.cecid.piazza.commons.dao.ds.DataSourceDAO.executeQuery(DataSourceDAO.java:437)
at hk.hku.cecid.piazza.commons.dao.ds.DataSourceDAO.executeQuery(DataSourceDAO.java:416)
at hk.hku.cecid.piazza.commons.dao.ds.DataSourceDAO.find(DataSourceDAO.java:348)
at hk.hku.cecid.ebms.spa.dao.PartnershipDataSourceDAO.findPartnershipsByCPA(PartnershipDataSourceDAO.java:31)
at hk.hku.cecid.ebms.spa.dao.PartnershipDataSourceDAO.findPartnershipByCPA(PartnershipDataSourceDAO.java:37)
at hk.hku.cecid.ebms.spa.service.EbmsMessageSenderService.serviceRequested(EbmsMessageSenderService.java:91)


"http-nio-7080-exec-200" daemon prio=10 tid=0x00007f9fd80ad800 nid=0x1e9 waiting for monitor entry [0x00007f9fb30ae000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:157)
- waiting to lock <0x0000000500b8e3c8> (a org.apache.commons.dbcp.datasources.SharedPoolDataSource)
at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:669)
at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:653)
at hk.hku.cecid.piazza.commons.dao.ds.DataSourceTransaction.begin(DataSourceTransaction.java:92)
at hk.hku.cecid.piazza.commons.dao.ds.DataSourceProcess.start(DataSourceProcess.java:84)
- locked <0x000000053b8d7378> (a hk.hku.cecid.ebms.spa.dao.MessageServerDataSourceDAO$2)
at hk.hku.cecid.ebms.spa.dao.MessageServerDataSourceDAO.storeOutboxMessage(MessageServerDataSourceDAO.java:93)
at hk.hku.cecid.ebms.spa.handler.OutboundMessageProcessor.storeOutgoingMessage(OutboundMessageProcessor.java:287)
at hk.hku.cecid.ebms.spa.handler.OutboundMessageProcessor.generateAndStoreEbxmlMessage(OutboundMessageProcessor.java:352)
at hk.hku.cecid.ebms.spa.handler.OutboundMessageProcessor.processOutgoingMessage(OutboundMessageProcessor.java:111)
at hk.hku.cecid.ebms.spa.handler.MessageServiceHandler.processOutboundMessage(MessageServiceHandler.java:283)
at hk.hku.cecid.ebms.spa.service.EbmsMessageSenderService.serviceRequested(EbmsMessageSenderService.java:189)
at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at hk.hku.cecid.piazza.commons.util.Instance.invoke(Instance.java:158)
at hk.hku.cecid.piazza.commons.soap.WebServicesAdaptor.processRequest(WebServicesAdaptor.java:143)
at hk.hku.cecid.piazza.commons.soap.SOAPHttpAdaptor.processRequest(SOAPHttpAdaptor.java:132)
at hk.hku.cecid.piazza.commons.servlet.http.HttpDispatcher.processRequest(HttpDispatcher.java:126)
at hk.hku.cecid.piazza.commons.servlet.http.HttpDispatcher.doPost(HttpDispatcher.java:277)


The outbox collector xml is as given below:

<parameters>
        <parameter name="execution-interval" value="50" />
    </parameters>

        <component id="task-list" name="Outbox Collector">
        <class>hk.hku.cecid.ebms.spa.task.OutboxCollector</class>
                <parameter name="max-thread-count" value="200" />
        </component>


 <component id="daofactory" name="System DAO Factory">
        <class>hk.hku.cecid.piazza.commons.dao.ds.SimpleDSDAOFactory</class>
                <parameter name="driver" value="oracle.jdbc.OracleDriver"/>
                <parameter name="url" value="jdbc:oracle:thin:@172.20.177.239:1523:DB03"/>
                <parameter name="username" value="xxxx"/>
                <parameter name="password" value="xxxx"/>
        <parameter name="pooling" value="true"/>
        <parameter name="maxActive" value="100"/>
        <parameter name="maxIdle" value="60"/>
        <parameter name="maxWait" value="-1"/>
        <parameter name="testOnBorrow" value="true"/>
        <parameter name="testOnReturn" value="false"/>
        <parameter name="testWhileIdle" value="false"/>
                <parameter name="validationQuery" value="select 1 from dual"/>
                <parameter name="config">hk/hku/cecid/ebms/spa/conf/ebms.oracle.dao.xml</parameter>
        </component>

2.bin
1.bin

kenne...@throput.com

unread,
Aug 9, 2017, 9:47:37 PM8/9/17
to Hermes 2+ Community Forum
Dear Si,

Observing from below stack trace, we noted that the Hermes is waiting for the DB connection. 

In the outbox-collector.module.xml, configuring "max-thread-count" to be 10 (default) would able to solve the problem. Actually, value of 10 is pretty enough to handle some loading without problem. 

In your case, 200 threads will be running concurrently in Hermes under loading test. Please check whether the Database server can serves 200 connections simultaneously or not. Otherwise, these threads in Hermes will be blocked and waiting for the DB connection.

Hope it will help to solve the problem. If you got any further questions, please feel free to ask.

Best Regards,

Kenneth
Reply all
Reply to author
Forward
0 new messages