MDB and pool sizing

[Ryan Cuprak]

 

 I've run into an odd problem with MDBs. It seems that only 5 MDBs ever get a message at the same time and I haven't been able to determine the cause. The MDB is reading messages off of a queue from ActiveMQ. I've deployed the ActiveMQ RAR with a prefetch of 100. The activation config properties are 100 for both maxSessions and maxMessagesPerSessions. No matter how many messages I dump into the queue, only 5 beans are ever busy at any given point in time. I set the standing pool size to 15 - 15 instances are created but 10 never get any messages. Max pool size is 25, which for some reason it never tries expanding beyond the 15 (despite plenty of messages). I am on Payara 4.1.2.173.  

 Anyone run into this or have an idea what might throttle the MDBs? 

Thanks,

-Ryan

 

[Ondro Mihályi]

I was working with ActiveMQ and Payara 4.1.2.173 and I remember that by default max 10 messages were being processed at any time by default. This was affected by the value of maxSession in activation config, which is 10. Increasing the thread pool size wouldn't help because the ActiveMQ RAR respects the value of maxSessions and binds a thread to a session. Increasing maxSessions would increase the number of threads used by the RAR and thus the number of messages processed in parallel.

I'm very surprised that in your case only 5 messages are processed in parallel, I would expect 10 with the default configuration. Can you find what could limit the number of messages to 5? Is it possible that more MDBs listen to the same queue?

Ondro

[Ryan Cuprak]

 

I created a skeleton app with a single MDB and setup a fresh 4.1.2.173 with the ActiveMQ RAR - only 5 messages are ever processed concurrently. I am using ActiveMQ 5.15.4. 

I ran into that max 10 messages before - that was my first assumption that I screwed-up the configuration. Whether I change the maxSessions or not makes no difference. I haven't been able to reach 10 yet! I have tinkered with every parameter I could find - prefetch etc.

To test the concurrency, I added a sleep into the MDB. In my clean environment, when the bean finishes processing a message, the exception below appears. 

[2018-08-14T15:51:30.109-0400] [Payara 4.1] [SEVERE] [] [javax.enterprise.system.container.ejb.mdb.org.glassfish.ejb.mdb] [tid: _ThreadID=257 _ThreadName=__ejb-thread-pool16] [timeMillis: 1534276290109] [levelValue: 1000] [[

  java.lang.reflect.InvocationTargetException

java.lang.reflect.InvocationTargetException

at com.sun.ejb.containers.BaseContainer.createEjbInstanceAndContext(BaseContainer.java:1659)

at org.glassfish.ejb.mdb.MessageBeanContainer.createMessageDrivenEJB(MessageBeanContainer.java:767)

at org.glassfish.ejb.mdb.MessageBeanContainer.access$100(MessageBeanContainer.java:126)

at org.glassfish.ejb.mdb.MessageBeanContainer$MessageBeanContextFactory.create(MessageBeanContainer.java:529)

at com.sun.ejb.containers.util.pool.NonBlockingPool.preload(NonBlockingPool.java:314)

at com.sun.ejb.containers.util.pool.NonBlockingPool.doResize(NonBlockingPool.java:558)

at com.sun.ejb.containers.util.pool.NonBlockingPool$IdleBeanWork.run(NonBlockingPool.java:657)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.NullPointerException

at com.sun.ejb.containers.BaseContainer.injectEjbInstance(BaseContainer.java:1718)

at com.sun.ejb.containers.BaseContainer.createEjbInstanceAndContext(BaseContainer.java:1652)

... 11 more

My test MDB: (note, I didn't set the maxSessions, tried but didn't make a difference)

@MessageDriven(
        activationConfig = {
                @ActivationConfigProperty(propertyName="destinationType",propertyValue="javax.jms.Queue"),
                @ActivationConfigProperty(propertyName="destination",propertyValue="trainingQueue")
        }
)
public class OrderProcessorMDB {

    private static AtomicInteger counter = new AtomicInteger(0);

    private int myNumber;

    @PostConstruct
    public void init() {
       myNumber = counter.incrementAndGet();
    }

    public void onMessage(Message message) {
        TextMessage tm = (TextMessage)message;
        try {
            System.out.println("Count: " + myNumber + " msg: " + tm.getText());
        } catch (JMSException e) {
            e.printStackTrace();
        }
        try {
            Thread.sleep(300000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println("Resuming...");
    }
}

[Ryan Cuprak]

Update, replicated the 5 limit on Payara 5.

Looking at the log messages:

[2018-08-14T16:20:18.609-0400] [Payara 5.0] [INFO] [] [] [tid: _ThreadID=170 _ThreadName=orb-thread-pool-1 (pool #1): worker-2] [timeMillis: 1534278018609] [levelValue: 800] [[

  Count: 2 msg: 1]]

How is the orb-thread-pool-1 thread pool configured? 

I found this documentation:

https://docs.oracle.com/cd/E18930_01/html/821-2431/abegt.html#abegy

However, I am not sure what I need for the "serverInstance" - list-instances returns nothing. 

-Ryan

[Ryan Cuprak]

I went back to payara-4.1.1.161 and the MDB worked fine there (not limited to 5 threads). The log messages:

[2018-08-15T09:53:18.971-0400] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=101 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1534341198971] [levelValue: 800] [[

  Count: 1 msg: 1]]

It is using a completely different thread pool in 161 versus after 161. 

I don't suppose anyone knows why it was changed?

-Ryan