Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Some ActiveMQ messages getting stuck in Wildfly - and how to monitor?

74 views
Skip to first unread message

Christoffer A. Nilsen

unread,
Jan 3, 2025, 10:30:55 AMJan 3
to WildFly
I'm experiencing an issue that maybe 1/1.000.000 messages sent from our remote ActiveMQ gets stuck in Wildfly(31), and is never processed/ran. Usually it is multiple at once, but happens rarely.

The two ways I discover this is that when I go into the ActiveMQ(Artemis) Management Console and browse the message queue, I can see it says for example 1-10 of 251, but the table is empty. Also if I check the message-driven-bean stats inside of my deployment artifact management model I see them.

The only way I know of solving this is shutting down Wildfly (un/redeploying artifact does not work). Then the connection between Wildfly and ActiveMQ will release and ActiveMQ will put the messages back on queue as ActiveMQ has not gotten any acknowledgement for these messages. Then they will again reappear in my browse queue table. When I start Wildfly again, the messages are handled correctly. I also believe is sufficient time has passed, ActiveMQ will put the messages in the DLQ not back on the original queue.

Also want to point out that I see no Exceptions in the log relating to the stuck messages, or other signs of Wildfly attempting to handle/run the stuck messages. It's not like they have been attempted once and crashed or anything(as far as I can see).

Questions
  • I have tried to google, and look into the Wildfly documentation, but having hard time finding anything relevant, could anyone maybe point me to the relevant section?
  • Any idea why this is happening?
  • Is there anyway I can view/monitor the messages Wildfly is currently storing?
  • Any way to force/tell Wildfly to handle/drop/execute the old messages on the internal Wildfly queue?

As far as I understand, Wildfly consumes all messages on the ActiveMQ queue, and puts them on an internal Wildfly queue, meaning even if my MessageListener is setup to run 10 parallel threads (maxSession=10), and my strict-max-pool have a max-pool-size="10", and I insert 10,000 messages into ActiveMQ, Wildfly will consume all.  I have tried to test this by inserting thousands of messages into ActiveMQ, with the pool=10 and maxSession=10, were one message takes maybe a second to run, and I see that the browse the message table in ActiveMQ Management Console goes empty long long before my MessageListener.onMessage() have executed all messages. But the browse table will still display 1-10 of 11,212 even if the table is empty. Which I assume is messages that is yet to get an acknowledgement for Wildfly.

Any help in where I could even start researching this problem would be very helpful, or if I have said something wrong, please point it out so I can be better informed on the topic.

Emmanuel Hugonnet

unread,
Jan 3, 2025, 11:04:23 AMJan 3
to wil...@googlegroups.com
Are you using classic ActiveMQ or Artemis ActiveMQ ?
Do you reproduce with WildFly 35 Beta1 as I fixed some issues in the recovery process of WildFly for messaging.
Also I'm not sure about what WildFly queue you are talking about, as you don't need to have an embedded broker.

Emmanuel



Le 03/01/2025 à 16:30, 'Christoffer A. Nilsen' via WildFly a écrit :
> I'm experiencing an issue that maybe 1/1.000.000 messages sent from our remote ActiveMQ gets /stuck /in Wildfly(31), and is never
> processed/ran. Usually it is multiple at once, but happens rarely.
>
> The two ways I discover this is that when I go into the ActiveMQ(Artemis) Management Console and browse the message queue, I can see it
> says /for example /1-10 of 251, but the table is empty. Also if I check the message-driven-bean stats inside of my deployment artifact
> management model I see them.
>
> The only way I know of solving this is shutting down Wildfly (un/redeploying artifact does not work). Then the connection between Wildfly
> and ActiveMQ will release and ActiveMQ will put the messages back on queue as ActiveMQ has not gotten any acknowledgement for these
> messages. Then they will again reappear in my browse queue table. When I start Wildfly again, the messages are handled correctly. I also
> believe is sufficient time has passed, ActiveMQ will put the messages in the DLQ not back on the original queue.
>
> Also want to point out that I see no Exceptions in the log relating to the stuck messages, or other signs of Wildfly attempting to
> handle/run the stuck messages. It's not like they have been attempted once and crashed or anything(as far as I can see).
>
> *Questions*
>
> * I have tried to google, and look into the Wildfly documentation, but having hard time finding anything relevant, could anyone maybe
> point me to the relevant section?
> * Any idea why this is happening?
> * Is there anyway I can view/monitor the messages Wildfly is currently storing?
> * Any way to force/tell Wildfly to handle/drop/execute the old messages on the internal Wildfly queue?
>
>
> As far as I understand, Wildfly /consumes/ all messages on the ActiveMQ queue, and puts them on an internal Wildfly queue, meaning even if
> my MessageListener is setup to run 10 parallel threads (maxSession=10), and my strict-max-pool have a max-pool-size="10", and I insert
> 10,000 messages into ActiveMQ, Wildfly will consume all.  I have tried to test this by inserting thousands of messages into ActiveMQ, with
> the pool=10 and maxSession=10, were one message takes maybe a second to run, and I see that the browse the message table in ActiveMQ
> Management Console goes empty long long before my MessageListener.onMessage() have executed all messages. But the browse table will still
> display 1-10 of 11,212 even if the table is empty. Which I assume is messages that is yet to get an acknowledgement for Wildfly.
>
> Any help in where I could even start researching this problem would be very helpful, or if I have said something wrong, please point it
> out so I can be better informed on the topic.
> --
> You received this message because you are subscribed to the Google Groups "WildFly" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/wildfly/b1a41f7a-b951-45d1-a238-728143e2ef1dn%40googlegroups.com
> <https://groups.google.com/d/msgid/wildfly/b1a41f7a-b951-45d1-a238-728143e2ef1dn%40googlegroups.com?utm_medium=email&utm_source=footer>.

Christoffer A. Nilsen

unread,
Jan 3, 2025, 3:01:56 PMJan 3
to WildFly
So I just left work for the weekend so I won't be able to check before Monday, however I know our ActiveMQ installation shipped with the Artemis Management Console, so if only ActiveMQ Artemis, and not Classic has that Management Console, we use Artemis.
We also updated to the newest ActiveMQ version within the past year.

When say Wildfly internal queue I just mean somewhere the messages are stored waiting to be processed by the MessageListener. Like in my tests, somewhere internally in Wildfly the 10,000 messages must be stored somewhere waiting to be processed, I just called that a Queue.

Also our system is to big to easily upgrade from 31 to 35 to do tests reliably.
I also haven't setup a reproducible test case for this. Our remote ActiveMQ handles somewhere from 500,000 - 2,000,000 messages a day, and we experience this maybe every 3-4 weeks.
We get alerted when a message isn't processed within a few hours, and then I find them stuck and need to do a restart of our Wildfly.

I also believe our setup is fairly standard, with a little bit of configuration in the domain:naming subsystem and using @MessageDriven and @ResourceAdapter annotations on our MessageListener.

Christoffer A. Nilsen

unread,
Jan 6, 2025, 3:50:00 AMJan 6
to WildFly
We use ActiveMQ Artemis 2.30.0.

fredag 3. januar 2025 kl. 17:04:23 UTC+1 skrev Emmanuel Hugonnet:

Emmanuel Hugonnet

unread,
Jan 6, 2025, 4:21:47 AMJan 6
to wil...@googlegroups.com
Hello,
Do yo have any traces on the broker side ?
You could also enable the TRACE level on WildFly for the messaging part:
- org.apache.activemq.artemis
- org.wildfly.extension.messaging-activemq
Are the messages processed in some XA transaction ?
Regards,

Emmanuel
> <https://groups.google.com/d/msgid/wildfly/b1a41f7a-b951-45d1-a238-728143e2ef1dn%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --
> You received this message because you are subscribed to the Google Groups "WildFly" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/wildfly/e3fb2099-5a5d-4bd8-9bda-4e97805fd1bbn%40googlegroups.com
> <https://groups.google.com/d/msgid/wildfly/e3fb2099-5a5d-4bd8-9bda-4e97805fd1bbn%40googlegroups.com?utm_medium=email&utm_source=footer>.

Christoffer A. Nilsen

unread,
Jan 6, 2025, 5:55:15 AMJan 6
to WildFly
How much logging will enabling TRACE on those packages cause? Considering we handle anything from 500k-2m messages a day, and this error might only happen once every 3 weeks.
We don't have backup of the artemis.log files back to the day the error happen, I will change the backup duration and remember to check the Artemis logs next time it happens. But I believe I check last time a saw pretty much nothing except for some Management Console login lines.

Do you have any idea what we could try to do next time this happen? something we could investigate/check before we restart the Wildfly to release the connection? Is there somewhere we can check in Wildfly the current status of the messages or something?

Emmanuel Hugonnet

unread,
Jan 6, 2025, 9:46:34 AMJan 6
to wil...@googlegroups.com
Well I agree it might be quite voluminous so maybe we can't have those set.
Do you have any transaction recovery happening on the server restart ?
Because without any error message or logs and no reproducer it is going to be tricky to fix :(
Emmanuel.
> <https://groups.google.com/d/msgid/wildfly/e3fb2099-5a5d-4bd8-9bda-4e97805fd1bbn%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --
> You received this message because you are subscribed to the Google Groups "WildFly" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/wildfly/7015050c-a6db-4f46-a48f-2a68548e8aa9n%40googlegroups.com
> <https://groups.google.com/d/msgid/wildfly/7015050c-a6db-4f46-a48f-2a68548e8aa9n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Christoffer A. Nilsen

unread,
Jan 7, 2025, 7:44:43 AMJan 7
to WildFly
If you mean related to ActiveMQ, then no.

We have also tried to stop delivery for the affected artifact when we discover stuck messages.

ModelNode address = Operations.createAddress(new String[]{"deployment", queueInfo.deployment, "subdeployment", queueInfo.subDeployment, "subsystem", "ejb3", "message-driven-bean", queueInfo.messageDrivenBean});
ModelNode operation = Operations.createOperation(activity ? "start-delivery" : "stop-delivery", address);
ModelNode res = client.execute(operation);

While we are still able to control it all, and most messags continue to operate normally, stopping delivery does not affect the stuck message.
The only solution we have found so far is shutting down Wildfly as mentioned previously.

What about enabling TRACE when the problem has first occurred? Could it be that we then see Wildfly telling is it has having issues?
Maybe a workaround could be to enable some sort of timeout on ActiveMQ's side? If ActiveMQ have not received an acknowledgement within 1h the message will be reattempted?

Reply all
Reply to author
Forward
0 new messages