IBM BAMOE - Timers going crazy - JBPM-10242 side effects
52 views
Skip to first unread message
Aslam Kumhar
Feb 12, 2025, 2:17:36 PMFeb 12
to jBPM Usage
Hi,
We are using IBM BAMOE (formerly Redhat PAM) for more than 2 years. We have encounter a lot of issues but eventually through community posts or through online available docs we seem to resolve the issue.
Recently our production environment stopped responding, we had to restart but it keep going down because either CPU usage was very high and it crashes or eventually we get OOM. We noticed that the EJB queue was stuck and we were seeing a lot of timer errors/warning. We engaged IBM support but they were unable to help and the only way to resume jBPM was to cleanup both JBOSS_EJB_TIMER and TimerMappingInfo table. We have faced the issue twice and had to do cleanup twice.
We are using IBM BAMOE (formerly Redhat PAM) for more than 2 years. We have encounter a lot of issues but eventually through community posts or through online available docs we seem to resolve the issue.
Recently our production environment stopped responding, we had to restart but it keep going down because either CPU usage was very high and it crashes or eventually we get OOM. We noticed that the EJB queue was stuck and we were seeing a lot of timer errors/warning. We engaged IBM support but they were unable to help and the only way to resume jBPM was to cleanup both JBOSS_EJB_TIMER and TimerMappingInfo table. We have faced the issue twice and had to do cleanup twice.
After 10 days of constant debugging, environment upgrade to 8.0.6 and even DB downgrade (SQL 2022 to SQL 2019), we found that the issue could be because of this bug https://issues.redhat.com/browse/JBPM-10242.
Fortunately atleast IBM gave us a patch, and after applying it in a controlled environment, we saw some positive gains. Our environment was at-least stable, no more high CPU <50% and no more OOM crashes.
But our problem is not fully resolved, we are now seeing some side effects:
1. EJB queue is constantly processing tasks, completing task and creating new ones.
2. JBOSS_EJB_TIMER table kept increasing with new timers, although TimerMappingInfo table is not increasing in proportion.
3. We are seeing a lot of deadlocks because of many concurrent INSERT and UPDATES in JBOSS_EJB_TIMER table.
4. We are not sure if we need to set both properties as true:
org.jbpm.ejb.timer.disable.linear.search
org.jbpm.ejb.timer.disable.linear.remove
5. How to find and remove stale EJB timers so the constant processing can be avoided.
Fortunately atleast IBM gave us a patch, and after applying it in a controlled environment, we saw some positive gains. Our environment was at-least stable, no more high CPU <50% and no more OOM crashes.
But our problem is not fully resolved, we are now seeing some side effects:
1. EJB queue is constantly processing tasks, completing task and creating new ones.
2. JBOSS_EJB_TIMER table kept increasing with new timers, although TimerMappingInfo table is not increasing in proportion.
3. We are seeing a lot of deadlocks because of many concurrent INSERT and UPDATES in JBOSS_EJB_TIMER table.
4. We are not sure if we need to set both properties as true:
org.jbpm.ejb.timer.disable.linear.search
org.jbpm.ejb.timer.disable.linear.remove
5. How to find and remove stale EJB timers so the constant processing can be avoided.
Appreciate any help on this.
Alex Porcelli
Feb 12, 2025, 2:22:59 PMFeb 12
to Aslam Kumhar, jBPM Usage
Aslam,
This is a mailing list for community version, for commercial product related needs, please reach out directly to the vendor.
-
Alex
--
You received this message because you are subscribed to the Google Groups "jBPM Usage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jbpm-usage+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/jbpm-usage/b98b4d32-a399-43aa-9a54-5d522f6d6761n%40googlegroups.com.
Aslam Kumhar
Feb 12, 2025, 2:27:27 PMFeb 12
to jBPM Usage
Hi Alex - Already did that and believe me. I don't think they have any idea on the product itself, been engaged with them for 10 days with no results. Any since the product is based on community version, I thought of giving this a try.
Reply all
Reply to author
Forward
0 new messages