GDS ActiveMQ in Tomcat Cluster Working. Doubt Is it Right Way ?

[R. S. Patil]

Hello,

We are evaluating GDS to replace BlazeDS for a application in Production for couple of years.

We are now testing server to Client Data push in haproxy - hazelcast - tomcat - ActiveMQ. After few trials following is our config which enables flex client to consume messages pushed from each tomcat instance.

Spring Config xml contains

    <graniteds:activemq-topic-destination id="amqTopic" name="amqTopic"
    no-local="true" session-selector="true"
     connection-factory="ConnectionFactory"
    destination-jndi-name="topic/amqTopic"
    transacted-sessions="true"
    acknowledge-mode="AUTO_ACKNOWLEDGE"
    broker-url="peer://localhost/${tcInstance}"
    create-broker="true"
    wait-for-start="true"
    durable="true"
 file-store-root="/Tej-EIS-Common/logs"/>

Actual Message sending service snippet

public void amqPushTimeAndInstance() {

String s = helper.systemSettings.getServerInstance() + "   <=====>   " + GenFun.FileNameTimeString(new Date());

AsyncMessage message = new AsyncMessage(); // <1>

message.setDestination("amqTopic");

message.setHeader(AsyncMessage.SUBTOPIC_HEADER, "NASDAQ");

message.setBody(s);

gravity.publishMessage(message);

}

with this now both tomcat instances which run above proc in a scheduled task are able to send messages and flex application is receiving it.

Since we have deviated a little bit from GDS documentation My doubts are

1. Changing from vm: to peer: is it right ? I got this idea from AMQ documentation.

2. Rather than using AMQ factory we used gravity approach. Is it correct ? 

    We are using sticky sessions for clustering so with AMQ factory way when flex application sticks to particular TC instance it was able to receive messages from that instance not from other instance so we switched to Gravity method which served the purpose.

3. Can you explain how durable works ? we have given proper destination but we do not see any file creation in that directory or feel any difference when it is made on/off.

Thanks and warm regards.

Raja Patil.

  

[wdrai]

All options are ignored when using broker-url.

Either you define a full activemq broken-url, or you define other parameters for a default vm transport.

In your case, that would be something like :

<graniteds:activemq-topic-destination id="amqTopic" name="amqTopic"
    no-local="true" session-selector="true"
     connection-factory="ConnectionFactory"
    destination-jndi-name="topic/amqTopic"
    transacted-sessions="true"
    acknowledge-mode="AUTO_ACKNOWLEDGE"

    broker-url="peer://localhost/${tcInstance}?create=true&amp;waitForStart=true&amp;broker.persistent=true&amp;broker.dataDirectory=/Tej-EIS-Common/logs"/>

[R. S. Patil]

Thanks  wdrai,

All options are ignored when using broker-url.

Either you define a full activemq broken-url, or you define other parameters for a default vm transport.

In your case, that would be something like :

<graniteds:activemq-topic-destination id="amqTopic" name="amqTopic"
    no-local="true" session-selector="true"
     connection-factory="ConnectionFactory"
    destination-jndi-name="topic/amqTopic"
    transacted-sessions="true"
    acknowledge-mode="AUTO_ACKNOWLEDGE"
    broker-url="peer://localhost/${tcInstance}?create=true&amp;waitForStart=true&amp;broker.persistent=true&amp;broker.dataDirectory=/Tej-EIS-Common/logs"/>

I did as above but I could not find any difference between earlier config and this one. Still I dont see any files being created in /Tej-EIS-Common/logs folder.

As well we have found few exceptions recorded in  tomcat log (Catalina) similarly when three instances are started, tests conducted and finally shutdown, there are three running java processes found in system monitor (KUbuntu 12.04). However we can still rerun above  procedure but at end it only add additional three processes. T

The catalina log is as follows. Can you please help us in finding out what issue is.

INFO    ~ A valid shutdown command was received via the shutdown port. Stopping the Server instance. ~ org.apache.catalina.core.StandardServer

INFO    ~ Pausing ProtocolHandler ["http-apr-8180"] ~ org.apache.coyote.http11.Http11AprProtocol

INFO    ~ Stopping service Catalina ~ org.apache.catalina.core.StandardService

INFO    ~ Destroying Spring FrameworkServlet 'GDS-dispatcher' ~ org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/GDS-Test]

INFO    ~ Closing Spring root WebApplicationContext ~ org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/GDS-Test]

INFO    ~ Shutting down log4j ~ org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/GDS-Test]

ERROR   ~ The web application [/GDS-Test] registered the JDBC driver [org.firebirdsql.jdbc.FBDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [FileWatchdog] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ Broker[ATC-1] Scheduler] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ Transport Server: tcp://0.0.0.0:0] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [MulticastDiscoveryAgent-advertise:tcp://DELL-15RSE:36357:Thread-9] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [MulticastDiscoveryAgent-listener:DiscoveryNetworkConnector:NC:BrokerService[ATC-1]:Thread-10] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ BrokerService[ATC-1] Task-2] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ BrokerService[ATC-1] Task-3] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ Transport: tcp:///127.0.0.1:59346@36357] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ Transport: tcp:///127.0.0.1:59345@36357] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ InactivityMonitor ReadCheckTimer] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ InactivityMonitor WriteCheckTimer] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [Notifier-MulticastDiscoveryAgent-listener:DiscoveryNetworkConnector:NC:BrokerService[ATC-1]] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ Transport: tcp://DELL-15RSE/127.0.1.1:47644@60005] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ Transport: tcp://DELL-15RSE/127.0.1.1:33804@56481] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] appears to have started a thread named [ActiveMQ InactivityMonitor Worker] but has failed to stop it. This is very likely to create a memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Locale context]) and a value of type [org.springframework.context.i18n.SimpleLocaleContext] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Request attributes]) and a value of type [org.springframework.web.context.request.ServletRequestAttributes] (value [org.apache.catalina.connector.Request@27da9eda]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [com.hazelcast.util.UuidUtil$1] (value [com.hazelcast.util.UuidUtil$1@1f30ea7]) and a value of type [java.util.Random] (value [java.util.Random@16fe464e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{loginID=, remoteAdrs=127.0.0.1, remoteHost=127.0.0.1, sessionID=HZ68A88D46625942F9A966C5922DDB1853}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Locale context]) and a value of type [org.springframework.context.i18n.SimpleLocaleContext] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Request attributes]) and a value of type [org.springframework.web.context.request.ServletRequestAttributes] (value [org.apache.catalina.connector.Request@67e7b36d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{loginID=, remoteAdrs=127.0.0.1, remoteHost=127.0.0.1, sessionID=HZ68A88D46625942F9A966C5922DDB1853}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Locale context]) and a value of type [org.springframework.context.i18n.SimpleLocaleContext] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Request attributes]) and a value of type [org.springframework.web.context.request.ServletRequestAttributes] (value [org.apache.catalina.connector.Request@2a8f7943]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{loginID=, remoteAdrs=127.0.0.1, remoteHost=127.0.0.1, sessionID=HZ68A88D46625942F9A966C5922DDB1853}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Locale context]) and a value of type [org.springframework.context.i18n.SimpleLocaleContext] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Request attributes]) and a value of type [org.springframework.web.context.request.ServletRequestAttributes] (value [org.apache.catalina.connector.Request@262580b3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{loginID=, remoteAdrs=127.0.0.1, remoteHost=127.0.0.1, sessionID=HZ68A88D46625942F9A966C5922DDB1853}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{loginID=, remoteAdrs=127.0.0.1, remoteHost=127.0.0.1, sessionID=HZ68A88D46625942F9A966C5922DDB1853}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{loginID=, remoteAdrs=127.0.0.1, remoteHost=127.0.0.1, sessionID=HZ68A88D46625942F9A966C5922DDB1853}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

ERROR   ~ The web application [/GDS-Test] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@fb9b6ad]) and a value of type [java.util.Hashtable] (value [{activemq.broker=ATC-1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ~ org.apache.catalina.loader.WebappClassLoader

INFO    ~ Stopping ProtocolHandler ["http-apr-8180"] ~ org.apache.coyote.http11.Http11AprProtocol

INFO    ~ Destroying ProtocolHandler ["http-apr-8180"] ~ org.apache.coyote.http11.Http11AprProtocol