Asynchronous and timer beans linked?

[Darren H]

I've hit some sort of limit with asynchronous beans in Payara Server 5 that I wonder if anyone knows anything about. What I've got is a timer bean that fires once a minute and calls a singleton that checks what services are required to be running. For each service an asynchronous EJB method on a specific EJB for that service is called, and the future object from that call is stored in a map against the service ID so that they can be checked to ensure that they are running on subsequent timer calls. The services are required to stay running once started and only stop if they are turned off so those async method calls will not return for a long time. 

All is good until 19 asynchronous method calls have been made, and at that point the timer service seemingly stops working. There are no errors logged, but my ejbTimeout method never gets called again until I disable the services. At which point the timer service seems to kick into life again. I've added another EJB with an ejbTimeout that gets called every 5 seconds and just logs out to say it's run and that also stops logging for the time that 19+ async methods are running. As far as I can see I shouldn't be hitting any thread pool or EJB limitation as they are set way above that limit, and I've also ensured that the setting to "Limit Concurrent EJB Instances" is unchecked to allow as many EJBs to be created as are required. I just don't seem to be able to see what's happening with the timer service. Has any one any ideas of anything I could try?

Thanks in advance.

Darren

[Ondro Mihályi]

Hi Darren,

It looks like your asynchornous calls are blocked, waiting for services to finish or respond. This means they block their thread.With the @Asynchronous annotation, the call always blocks if there are no available threads. So when you call the asynchronous method 19 times and all threads are taken by these calls, the next call to the asynchronous method would be blocking your timer. The same timer is never triggered in parallel, it needs to wait until all previous calls are finished. That's why all comes back to normal when you disable the services, which I assume causes the asynchronous methods to return.

Calls to @Asynchronous methods are executed with the EJB thread pool, which has by default 16 core threads. Even if you configure max pool size to a higher value and queue capacity to more than 0, only core number of threads will be used for async calls. The core number of threads can be configured with the thread-core-pool-size property of the EJB container, see https://stackoverflow.com/questions/17175229/where-can-i-configure-the-thread-pool-behind-the-asynchronous-calls-in-java-ee/40507428#40507428

 

Even better is to use the executor service instead of @Asynchronous annotation. It's more code but more powerful, because it uses an executor thread pool, which is a standard thread pool and behaves like most other thread pools in Payara Server. By default, it would accept all tasks and run them on an available thread or a new thread of none is available. This can be configured in Resources/Concurrent resources/Managed Executor Services.

This is an example EJB which would use an executor service instead of the EJB thread pool:

@Stateless
public class AsyncEJB {

    @Resource
    ManagedExecutorService executor;
   
//    @Asynchronous - instead of this, executor.submit is used inside the method
    public void asyncCall() {
        executor.submit(() -> {

            // this is executed asynchronously without blocking

        });
    }
}

st 3. 4. 2019 o 18:52 'Darren H' via Payara Forum <payara...@googlegroups.com> napísal(a):

--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/049abfaa-a408-4317-9d21-cb8fb4e38ef9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Darren H]

Hi Ondro, 

Thanks that's been very useful information. I've checked and can't see any mention of that setting in the Payara Server 5 admin console or using the asadmin tool. The only properties for the EJB container are:

configs.config.server-config.ejb-container.cache-idle-timeout-in-seconds=600

configs.config.server-config.ejb-container.cache-resize-quantity=32

configs.config.server-config.ejb-container.commit-option=B

configs.config.server-config.ejb-container.limit-instances-enabled=false

configs.config.server-config.ejb-container.max-cache-size=512

configs.config.server-config.ejb-container.max-pool-size=32

configs.config.server-config.ejb-container.max-wait-time-in-millis=0

configs.config.server-config.ejb-container.pool-idle-timeout-in-seconds=600

configs.config.server-config.ejb-container.pool-resize-quantity=8

configs.config.server-config.ejb-container.removal-timeout-in-seconds=5400

configs.config.server-config.ejb-container.session-store=${com.sun.aas.instanceRoot}/session-store

configs.config.server-config.ejb-container.steady-pool-size=0

configs.config.server-config.ejb-container.victim-selection-policy=nru

Do you know where I might find this setting?

It sounds like a good option to use the managed executor service directly though. Is it possible to do an EJB lookup from the submitted task so that the code will execute in the EJB container? 

Thanks in advance,

Darren

To unsubscribe from this group and stop receiving emails from it, send an email to payara...@googlegroups.com.

[Ondro Mihályi]

Hi,

There are no dedicated fields in the admin console for these options but you can specify them as key-value properties of the EJB container.

With asadmin, you should be able to set them with a key such as configs.config.server-config.ejb-container.thread-core-pool-size

Dňa št 4. 4. 2019, 11:43 'Darren H' via Payara Forum <payara...@googlegroups.com> napísal(a):

To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/371a5125-ef10-438c-8a23-fb643e986ba0%40googlegroups.com.

[Darren H]

Hi Ondro, 

Thanks ever so much for your suggestion, it's worked a treat. Now all my async processes are running and the timer process keeps firing. 

Interesting that this isn't more visible in the admin console or that there are dedicated settings for something that seems to me to be quite important, especially as it completely blocks the timer service if too many background threads are being used. 

Many thanks again, 

Darren

[Ondro Mihályi]

Yes, it's a little bit unfortunate. But we would like to change that in the future and simplify configuration of thread pools. However, we need to prioritize customer requests and none of them has reported this issue, so it's much lower on our priority list than we would like.

If you have ideas how to improve Payara in tgis regard, feel free to submit an enhancement issue on github, ideally with a detailed proposed solution. Payara is opensource but you don't gave to contribute source if you'd like to help ;-)

Another way to help us is to connect us with your local JUG or a relevant meetup, if it's within our reach we might be able to pay a visit and give a talk.

Ondro

Dňa pi 5. 4. 2019, 11:16 'Darren H' via Payara Forum <payara...@googlegroups.com> napísal(a):

To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/817b2481-6b53-4568-8267-8494de3c265f%40googlegroups.com.