JMS message sending fails after failover of remote brokers
Manuel M.
For testing I use
- the helloworld-mdb quickstart example (https://github.com/wildfly/quickstart/tree/main/helloworld-mdb) deployed on the Wildfly application server with MDBs receiving messages and a servlet for sending messages using an injected JmsContext and injected Queue/Topic (I renamed the queue/topic and removed the @JMSDestinationDefinitions in order to define the destinations in the messaging-activemq subysystem configuration)
- and two simple JMS client programs: one for sending messages to a queue, and one for receiving messages from a queue (they just go the default route: obtain an InitialContext, look up a destination and a connection factory, and create a connection and a session)
My setup almost works. The client programs and MDBs reconnect automatically on messaging broker failover/failback. However, the servlet from the quickstart example blocks when sending messages after the live/backup pair fails over, and no messages are sent. It fails after around one minute with the following stack trace:
12:15:53,496 ERROR [io.undertow.request] (default task-1) UT005023: Exception handling request to /helloworld-mdb/HelloWorldMDBServletClient: javax.jms.JMSRuntimeException: AMQ219014: Timed out after waiting 30,000 ms for response when sending packet 71
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:88)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.jms.client.ActiveMQJMSProducer.send(ActiveMQJMSProducer.java:100)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.jms.client.ActiveMQJMSProducer.send(ActiveMQJMSProducer.java:124)
at deployment.helloworld-mdb.war//org.jboss.as.quickstarts.servlet.HelloWorldMDBServletClient.doGet(HelloWorldMDBServletClient.java:95)
at javax.se...@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:503)
at javax.se...@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at io.opentracing.contrib.opentracing-jaxrs2//io.opentracing.contrib.jaxrs2.server.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:52)
at io.undert...@2.2.14.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.security.ely...@1.10.1.Final//org.wildfly.elytron.web.undertow.server.ElytronRunAsHandler.lambda$handleRequest$1(ElytronRunAsHandler.java:68)
at org.wildfly.secu...@1.18.3.Final//org.wildfly.security.auth.server.FlexibleIdentityAssociation.runAsFunctionEx(FlexibleIdentityAssociation.java:103)
at org.wildfly.secu...@1.18.3.Final//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
at org.wildfly.secu...@1.18.3.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
at org.wildfly.security.ely...@1.10.1.Final//org.wildfly.elytron.web.undertow.server.ElytronRunAsHandler.handleRequest(ElytronRunAsHandler.java:67)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.under...@2.2.14.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.under...@2.2.14.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.under...@2.2.14.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at org.wildfly.security.elytron...@1.10.1.Final//org.wildfly.elytron.web.undertow.server.servlet.CleanUpHandler.handleRequest(CleanUpHandler.java:38)
at io.under...@2.2.14.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.ext...@26.0.1.Final//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.under...@2.2.14.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.ext...@26.0.1.Final//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
at io.under...@2.2.14.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
at io.undert...@2.2.14.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undert...@2.2.14.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.ext...@26.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
at org.wildfly.ext...@26.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
at org.wildfly.ext...@26.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
at org.wildfly.ext...@26.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
at io.undert...@2.2.14.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
at io.under...@2.2.14.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
at io.under...@2.2.14.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
at org.jbos...@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jbos...@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jbos...@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jbos...@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at org.jbo...@3.8.5.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.jms.JMSException: AMQ219014: Timed out after waiting 30,000 ms for response when sending packet 71
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:539)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:443)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:552)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:296)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:268)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:143)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:125)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:220)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:207)
at org.apache.acti...@2.19.0//org.apache.activemq.artemis.ra.ActiveMQRAMessageProducer.send(ActiveMQRAMessageProducer.java:137)
at org.apache.ac...@2.19.0//org.apache.activemq.artemis.jms.client.ActiveMQJMSProducer.send(ActiveMQJMSProducer.java:97)
... 53 more
Caused by: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219014: Timed out after waiting 30,000 ms for response when sending packet 71]
... 65 more
I followed 7.8.12. Connect a pooled-connection-factory to a Remote Artemis Server, but I'm using http-connectors instead of remote-connectors because of using Wildfly as remote brokers (I unsuccessfully tried remote-connectors, too, using netty-connectors in the broker Wildfly instances).
All configurations are based on standalone-full-ha.xml with just the following modifications.
Messaging live Wildfly server "messaging-node-1":
- changed cluster password
- added replication-master
- changed cluster connection name
- added two destinations
...
<subsystem xmlns="urn:jboss:domain:messaging-activemq:13.0">
<server name="default">
<security elytron-domain="ApplicationDomain"/>
<cluster password="${jboss.messaging.cluster.password:**changed-password**}"/>
<statistics enabled="${wildfly.messaging-activemq.statistics-enabled:${wildfly.statistics-enabled:false}}"/>
<replication-master cluster-name="messaging-cluster" group-name="activemq-group"/>
<security-setting name="#">
<role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
</security-setting>
<address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
<http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
<http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
<param name="batch-delay" value="50"/>
</http-connector>
<in-vm-connector name="in-vm" server-id="0">
<param name="buffer-pooling" value="false"/>
</in-vm-connector>
<http-acceptor name="http-acceptor" http-listener="default"/>
<http-acceptor name="http-acceptor-throughput" http-listener="default">
<param name="batch-delay" value="50"/>
<param name="direct-deliver" value="false"/>
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="0">
<param name="buffer-pooling" value="false"/>
</in-vm-acceptor>
<jgroups-broadcast-group name="bg-group1" jgroups-cluster="activemq-cluster" connectors="http-connector"/>
<jgroups-discovery-group name="dg-group1" jgroups-cluster="activemq-cluster"/>
<cluster-connection name="messaging-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
<jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
<jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
<jms-queue name="TestQueue" entries="java:jboss/exported/queue/TestQueue" durable="true"/>
<jms-topic name="TestTopic" entries="java:jboss/exported/topic/TestTopic"/>
<connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
<connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
<pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
</server>
</subsystem>
...
Messaging backup Wildfly server "messaging-node-2":
Same as above except replication-master replaced by replication-slave
...
<subsystem xmlns="urn:jboss:domain:messaging-activemq:13.0">
<server name="default">
...
<replication-slave cluster-name="messaging-cluster" group-name="activemq-group"/>
...
</server>
</subsystem>
...
Application Wildfly server "application-node-1":
Replaced integrated ActiveMQ server definition with a remote broker and added external destinations. Notice the use of the default name and JNDI entries for the connection factory (for the EE and EJB subsystems):
...
<subsystem xmlns="urn:jboss:domain:messaging-activemq:13.0">
<http-connector name="node-1-http-connector" socket-binding="messaging-node-1" endpoint="http-acceptor"/>
<http-connector name="node-2-http-connector" socket-binding="messaging-node-2" endpoint="http-acceptor"/>
<pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="node-1-http-connector node-2-http-connector" user="JmsTest" password="JmsTest"/>
<external-jms-queue name="TestQueue" entries="java:/queue/TestQueue"/>
<external-jms-topic name="TestTopic" entries="java:/topic/TestTopic"/>
</subsystem>
...
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
...
<outbound-socket-binding name="messaging-node-1">
<remote-destination host="..." port="8180"/>
</outbound-socket-binding>
<outbound-socket-binding name="messaging-node-2">
<remote-destination host="..." port="8180"/>
</outbound-socket-binding>
</socket-binding-group>
...
The servlet:
@WebServlet("/HelloWorldMDBServletClient")
public class HelloWorldMDBServletClient extends HttpServlet {
private static final long serialVersionUID = -8314035702649252239L;
private static final int MSG_COUNT = 5;
@Inject
private JMSContext context;
@Resource(lookup = "java:/queue/TestQueue")
private Queue queue;
@Resource(lookup = "java:/topic/TestTopic")
private Topic topic;
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
resp.setContentType("text/html");
PrintWriter out = resp.getWriter();
out.write("<h1>Quickstart: Example demonstrates the use of <strong>JMS 2.0</strong> and <strong>EJB 3.2 Message-Driven Bean</strong> in JBoss EAP.</h1>");
try {
boolean useTopic = req.getParameterMap().keySet().contains("topic");
final Destination destination = useTopic ? topic : queue;
out.write("<p>Sending messages to <em>" + destination + "</em></p>");
out.write("<h2>The following messages will be sent to the destination:</h2>");
for (int i = 0; i < MSG_COUNT; i++) {
String text = "This is message " + (i + 1);
**context.createProducer().send(destination, text); // fails**
out.write("Message (" + i + "): " + text + "</br>");
}
out.write("<p><i>Go to your JBoss EAP server console or server log to see the result of messages processing.</i></p>");
} finally {
if (out != null) {
out.close();
}
}
}
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
doGet(req, resp);
}
}
Procedure: start the live/backup pair and the application server, deploy helloworld-mdb.war to the application server and load the servlet page in a web browser. For failover, shut down the live messaging server. Reload the servlet page in the web browser. Result: the web browser keeps waiting for around one minute and then receives what the servlet has printed so far ("...The following messages will be sent to the destination:" <EOF>).
Sending and receiving messages using the JMS client programs works after failover. MDBs keep receiving messages after failover. Just application server message sending fails.
I have unsuccessfully
- tried changing the timeout configuration values of the cluster connection
- used remote-connectors (to netty-connectors)
- moved message sending to a stateless session bean (in order to test transactions)
- replaced the JmsContext with a connection factory to no avail; createSession() blocks, then fails with "...Caused by: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219014: Timed out after waiting 30,000 ms for response when sending packet 63]"
@Resource(mappedName = "java:/JmsXA")
private ConnectionFactory connectionFactory;
@Resource(mappedName = "java:/queue/TestQueue")
private Queue queue;
...
Connection connection = connectionFactory.createConnection();
Session session = connection.createSession();
I would appreciate some help.
Manuel M.
<external-jms-queue name="TestQueue" entries="java:/queue/TestQueue"/>
<external-jms-topic name="TestTopic" entries="java:/topic/TestTopic"/>
</subsystem>