"Currently no messaging engines in bus running" exception when trying to open a topic connection

[deryk....@icesoft.com]

Finally, after weeks of running through the WAS jungle, I feel that I'm close to taming the beast.

I have a WAS ND cluster consisting of four nodes: 1 deployment manager, 2 managed app servers, and one database server. I have an application successfully deployed to the cluster and I have the SIB configured and running with a messaging engine and a datastore and I have created a TopicConnectionFactory (jms/ConnectionFactory) and three topics (jms/topic1, jms/topic2, jms/topic3).

I can successfully lookup up the connection factory and topics via JNDI from the application, but when I try to create a topic connection like so:

topicConnection = topicConnectionFactory.createTopicConnection();

I get the following exception:

javax.jms.JMSException: CWSIA0241E: An exception was received during the call to the method JmsManagedConnectionFactoryImpl.createConnection: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0088E: There are currently no messaging engines in bus MyBus running.

The root stack trace is:

Caused by: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0088E: There are currently no messaging engines in bus MyBus running
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl2.createConnection(TrmSICoreConnectionFactoryImpl2.java:227)
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl2.createConnection(TrmSICoreConnectionFactoryImpl2.java:116)
at com.ibm.ws.sib.api.jmsra.impl.JmsJcaManagedConnectionFactoryImpl.createManagedConnection(JmsJcaManagedConnectionFactoryImpl.java:415)
at com.ibm.ejs.j2c.poolmanager.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1632)
at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java:1424)
at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java:2080)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:868)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:617)
at com.ibm.ws.sib.api.jmsra.impl.JmsJcaConnectionFactoryImpl.createConnection(JmsJcaConnectionFactoryImpl.java:249)
at com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:206)
.. 35 more

So I go to the admin console and look at Buses > MyBus > Messaging engines and it shows that the status of myCluster.000-MyBus is fine (the solid green arrow that supposedly indicates that the messaging engine has started.

The odd thing is that if I go through a slightly different path in the admin console, Buses > MyBus > Bus members > Messaging engines, the messaging engine status is "unknown" (the question mark with a circle around it).

I'm a bit stumped. Almost everything points to the messaging engine running in the SIB but there is an apparent conflict of information in the console.

[David Currie]

Hi Deryk,

The definitive mechanism to determine the state of the messaging engine
would be to check the application server log files. On both cluster
members you should see then message "Messaging engine
myCluster.000-MyBus is in state Joined" then on one of the two members
the messaging engine should then go through the states "Starting" and
"Started".

Assuming that the messaging engine is indeed started correctly, then the
next question is why can't your application find it? If the instance of
the application is running in the same cluster member as the messaging
engine then I would expect all to be well (as it is just making local,
in-process, calls to the messaging engine. If it is when the application
is invoked in the other cluster member that it can't find a messaging
engine then this implies that the state information is not being shared
correctly between the two servers. In this case, it is worth checking
the logs for errors relating to DCS.

Regards,
David

[deryk....@icesoft.com]

Thanks for responding.

I restarted everything and checked the app server logs and saw what you mentioned. "App Server 1 on Host A" started up and I got the following log output (I've trimmed out everything except the SibMessage entries):

[10/6/06 14:42:01:493 MDT] 0000000a SibMessage A [:] CWSIC2001I: Messaging connections are being accepted.
[10/6/06 14:42:43:068 MDT] 0000001c SibMessage I [MyBus:myCluster.000-MyBus] CWSID0016I: Messaging engine myCluster.000-MyBus is in state Joined.
[10/6/06 14:42:43:102 MDT] 0000001f SibMessage I [MyBus:myCluster.000-MyBus] CWSID0016I: Messaging engine myCluster.000-MyBus is in state Starting.

[10/6/06 14:42:45:780 MDT] 00000024 SibMessage I [MyBus:myCluster.000-MyBus] CWSIS1538I: The messaging engine, ME_UUID=BCF757B8D4A3BED5, INC_UUID=29d33ef41f5d1c89, is attempting to obtain an exclusive lock on the data store.
[10/6/06 14:42:46:159 MDT] 00000025 SibMessage I [MyBus:myCluster.000-MyBus] CWSIS1543I: No previous owner was found in the messaging engines data store.
[10/6/06 14:42:46:191 MDT] 00000024 SibMessage I [MyBus:myCluster.000-MyBus] CWSIS1537I: The messaging engine, ME_UUID=BCF757B8D4A3BED5, INC_UUID=29d33ef41f5d1c89, has acquired an exclusive lock on the data store.
[10/6/06 14:42:48:858 MDT] 0000001f SibMessage I [MyBus:myCluster.000-MyBus] CWSID0016I: Messaging engine myCluster.000-MyBus is in state Started.

"App Server 2 on Host B" only had the Join message.

So I pointed my browser at the app running on App Server 1 on Host A (where the messaging engine should be based on the logs) and saw the same thing. I was able to retrieve the TopicConnectionFactory and the Topics via JNDI but when I execute:

TopicConnection tc = ((TopicConnectionFactory)connectionFactory).createTopicConnection();

It throws this:

javax.jms.JMSException: CWSIA0241E: An exception was received during the call to the method JmsManagedConnectionFactoryImpl.createConnection: com.
ibm.websphere.sib.exception.SIResourceException: CWSIT0088E: There are currently no messaging engines in bus MyBus running.

at com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:240)
at com.ibm.ws.sib.api.jms.impl.JmsManagedTopicConnectionFactoryImpl.createTopicConnection(JmsManagedTopicConnectionFactoryImpl.java:149)
at com.ibm.ws.sib.api.jms.impl.JmsManagedTopicConnectionFactoryImpl.createTopicConnection(JmsManagedTopicConnectionFactoryImpl.java:113)

...etc

I get the exact same thing if I try to run the application from the server that is not running the messaging engine. :-(

[Charly Oz]

Deryk,

To me that looks like the ME is starting up ok. However, if it had started
up previously, I would expect to see an entry for the previous owner (i.e.
succesfully running ME) in the datastore.

Are there any entries in the SIB database ME tables?

Have you tried using the SIB Explorer tool?
http://www.alphaworks.ibm.com/tech/sibexplorer

<deryk....@icesoft.com> wrote in message
news:707628113.1160169666...@ltsgwas010.sby.ibm.com...

[David Currie]

Hmm, as Charly says, the messaging engine looks to have started
successfully. Have you modified any of the properties on the connection
factory that would cause it to look for a specific messaging engine in a
specific place? For example the provider endpoints, target type, target
and target proximity?

Regards,
David

[deryk....@icesoft.com]

I don't think so. I mostly just accepted the defaults where possible.

Bus name = MyBus
Target = [blank]
Target type = Bus member name
Target significance = Preferred
Target inbound transport chain = [blank]
Provider endpoints = [blank]
Connection proximity = Bus

In fact, other than the Name, JNDI name, and Bus name entries, I don't think I touched anything. Maybe there's something there that I should change?

[deryk....@icesoft.com]

I cleaned out the tables and had them re-created (along with the logs) so I could get a "fresh" copy of everything which is probably why there is no entry for a previous owner.

I followed the link and downloaded the SIBExplorer, ran it, and successfully connected to the Deployment Manager. It shows a running ME. Not sure what I should do to test it. If I select "Destinations" in the tree and right-click the Default.Topic.Space, the context menu has an entry for "Put Single Test Message..." but it greyed-out. The SIBExplorer doesn't seem to come with a lot of docs so any tips would be appreciated.

I checked the database and there are entries there. For example, the SIBOWNER table returned this:

ME_UUID INC_UUID VERSION MIGRATION_VERSION
---------------- ---------------- ---------------------- ----------------------
BCF757B8D4A3BED5 29d33ef41f5d1c89 1 0

The mystery deepens.

[David Currie]

SIBExplorer only lets you send test messages to queue destinations, not
topic spaces. May be worth creating one just to see if this works. I
think you're going to need to turn on some trace to identify the steps
that are leading up to the error message and, if an option for you,
raise a service request to get IBM to take a look at them.

Regards,
David

[deryk....@icesoft.com]

Been away for a bit. Decided to wrestle with WebLogic for a change of pace.

I took you advice and created a couple of Queues - one through the Administrative console and one through the SIBExplorer. I tried sending a single test message to both and was treated to the following stack trace courtesy of the SIBExplorer. I wasn't sure what the Message Client settings should be. I pointed it at both the Deployment Manager and the App server running the JMS but with pretty much the same result. From what I ended up reading, since I'm running the SIBExplorer from a remote machine, I have to set the Provider Endpoints for the connection factory (hostA.mydomain, hostB.mydomain - not sure what the correct delimiter is).

com.ibm.ws.sib.tools.explorer.helper.MessagingHelper$MessagingServiceNotAvailableException: javax.jms.JMSException: CWSIA0241E: An exception was received during the call to the method JmsManagedConnectionFactoryImpl.createConnection: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0006E: It is not possible to connect to bus MyBus because the following bootstrap servers could not be contacted [deploymentManager.myDomain:7276:BootstrapBasicMessaging] and the following bootstrap servers returned an error condition []. See previous messages for the reason for each bootstrap server failure..
at com.ibm.ws.sib.tools.explorer.helper.JMSHelper.getConnectionToME(JMSHelper.java:611)
at com.ibm.ws.sib.tools.explorer.helper.JMSHelper._putMessage(JMSHelper.java:304)
at com.ibm.ws.sib.tools.explorer.helper.JMSHelper.access$3(JMSHelper.java:298)
at com.ibm.ws.sib.tools.explorer.helper.JMSHelper$2.run(JMSHelper.java:218)
Caused by: javax.jms.JMSException: CWSIA0241E: An exception was received during the call to the method JmsManagedConnectionFactoryImpl.createConnection: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0006E: It is not possible to connect to bus MyBus because the following bootstrap servers could not be contacted [deploymentManager.myDomain:7276:BootstrapBasicMessaging] and the following bootstrap servers returned an error condition []. See previous messages for the reason for each bootstrap server failure..
at com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:240)
at com.ibm.ws.sib.api.jms.impl.JmsQueueConnectionFactoryImpl.createQueueConnection(JmsQueueConnectionFactoryImpl.java:152)
at com.ibm.ws.sib.tools.explorer.helper.JMSHelper.getConnectionToME(JMSHelper.java:588)
... 3 more
Caused by: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0006E: It is not possible to connect to bus MyBus because the following bootstrap servers could not be contacted [deploymentManager.myDomain:7276:BootstrapBasicMessaging] and the following bootstrap servers returned an error condition []. See previous messages for the reason for each bootstrap server failure.
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl2.remoteBootstrap(TrmSICoreConnectionFactoryImpl2.java:497)
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl2.createConnection(TrmSICoreConnectionFactoryImpl2.java:209)
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl2.createConnection(TrmSICoreConnectionFactoryImpl2.java:114)
at com.ibm.ws.sib.api.jmsra.impl.JmsJcaConnectionFactoryImpl.createCoreConnection(JmsJcaConnectionFactoryImpl.java:499)
at com.ibm.ws.sib.api.jmsra.impl.JmsJcaConnectionFactoryImpl.createCoreConnection(JmsJcaConnectionFactoryImpl.java:435)
at com.ibm.ws.sib.api.jmsra.impl.JmsJcaConnectionFactoryImpl.createConnection(JmsJcaConnectionFactoryImpl.java:278)
at com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:206)
... 5 more

[David Currie]

Hi Deryk,

The connection process for a remote client involves to steps. The first
step is to connect to an application server running the SIB service.
This service then communicates with WebSphere workload management to
determine where there is a running messaging on the requested bus. The
client is then re-direct to that messaging engine. The SIB service is
enabled by default when you add an application server to a bus or you
enable it explicitly on other application servers.

The provider endpoints are a comma separated list of colon separated
triples. Each triple contains a host name, port number and transport
chain. There are defaults for each of these which are
localhost:7276:BootstrapBasicMessaging. My guess is that, in your case
it isn't working because you're not using the default port. So you want
to use the host name of an application server that is a member of one or
more buses (or has the SIB service enabled) followed by the port number
which corresponds to the SIB_ENDPOINT_ADDRESS on that server. You can
give a comma separated list for high availability purposes.

Regards,
David

[deryk....@icesoft.com]

Thanks, David.

You are quite the masochist for hanging in this long ;-).

1) I used the admin console to check the ports of both of the app servers and they both list SIB_ENDPOINT_ADDRESS as 7276.

2) I turned on all tracing of SIB messages for:

the Deployment Manager [*=info: com.ibm.ws.sib.*=all]
both App servers [*=info: com.ibm.websphere.sib.*=all]
and both Node Agents [*=info: com.ibm.*=info: com.ibm.ws.sib.*=all]

3) I then added the provider endpoints to my TestQConnectionFactory like so:

appServerHostA.mydomain:7276:BootstrapBasicMessaging, appServerHostB.mydomain:7276:BootstrapBasicMessaging

4) I then stopped all the nodes, cleaned out all the logs, and restarted everything.

5) Configured SIBExplorer to use a Messaging Client Configuration of

appServerHostA.mydomain:7276:BootstrapBasicMessaging

and tried to send a single test message to the queue.

Turns out that I still can't connect with SIBExplorer (same stack trace as before). There is also no valuable trace output in any of the logs. There is output relating to the SIB indicating general success in setting up the bus and the messaging engine but no errors or warnings when I try to connect with SIBExplorer.

I guess I'm going to have to try and hit up IBM for help. Hopefully I can point them at this forum thread as a head start to what I've already tried.

[deryk....@icesoft.com]

Thanks, David.

You are quite the masochist for hanging in this long ;-).

1) I used the admin console to check the ports of both of the app servers and they both list SIB_ENDPOINT_ADDRESS as 7276.

2) I turned on all tracing of SIB messages for:

the Deployment Manager [*=info: com.ibm.ws.sib.*=all]
both App servers [*=info: com.ibm.websphere.sib.*=all]
and both Node Agents [*=info: com.ibm.*=info: com.ibm.ws.sib.*=all]

3) I then added the provider endpoints to my TestQConnectionFactory like so:

appServerHostA.mydomain:7276:BootstrapBasicMessaging, appServerHostB.mydomain:7276:BootstrapBasicMessaging

4) I then stopped all the nodes, cleaned out all the logs, and restarted everything.

5) Configured SIBExplorer to use a Messaging Client Configuration of

appServerHostA.mydomain:7276:BootstrapBasicMessaging

and tried to send a single test message to the queue.

Turns out that I still can't connect with SIBExplorer (same stack trace as before). There also seems to be no valuable trace output in any of the logs. There is output relating to the SIB indicating general success in setting up the bus and the messaging engine but no errors or warnings when I try to connect with SIBExplorer. The lack of useful debugging/troubleshooting information has become extremely frustrating.

I guess I'm going to have to try and hit up IBM for help. Hopefully I can point them at this forum thread as a head start to what I've already tried. Again, thanks for hanging on this long.

Deryk

[David Currie]

Just don't like it when things don't work as they should (and as I know
they have for me)! Unfortunately there's only so much you can do via a
newsgroup so, as you say, I think it's time to get IBM support involved.
FWIW: [SIB*=all] will give you all of the SIB trace or try
[SIBTrm=all=enabled:WLM=all=enabled] in particular for determining what
is going on during the bootstrapping process.

Regards,
David

[watn...@mac.com]

Just wanted to provide some closure to this issue in the hopes it saves somebody else some pain.

It turns out that the culprit was static routing being enabled. Apparently it's not a default setting but if it somehow gets turned on (and I don't know how it did for my configuration), then it could cause this type of behaviour.

To get rid of it, IBM support instructed me to delete the various *.wsrttbl files from the deployment manager and app server profiles.

A couple of links that may prove valuable on this topic were also forwarded to me and I'll provide them here:

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/urun_rwlm_cluster_svr.html

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/trun_wlm_cluster_routetable.html

[jyoti....@fmr.com]

Hi DMS..I am struggling with the same error .I get the error when I create a queueconnection in a stand alone client. I am using websphere 6.0.2.
--------------------------------------------------------------------------
QueueConnection queueConnection = queueConnectionFactory.createQueueConnection();
-------------------------------------------------------------------------

I wasn't able to get to the clusters setting from admin console. Is it under some section within Server1 settings?

Also, a search on '*.wsrttbl' found no files.

Any suggestions? Thanks!

[Stephen Cocks]

The problem found in this thread is specific to the use of clusters and
static routing in a WAS ND environment. The symptom message also occurs
under other conditions, some of which are caused by incorrect or missing
configuration settings. If you don't see clusters in your console you're
probably running against a standalone server (not a deployment manager) and
this particular thread won't apply.

I suggest you start a new thread in this newsgroup and fully describe your
environment - how many machines, nodes, etc are involved, what type of
profiles, what is the bus configuration (which servers/clusters are bus
members, etc), where does your client connect to for JNDI lookups (eg
PROVIDER_URL setting), what are the settings on your
QueueConnectionFactory - then we may be able to help spot what's wrong.

--
Stephen Cocks
WESB/WPS System Administration

<jyoti....@fmr.com> wrote in message
news:466080607.1167772761...@ltsgwas010.sby.ibm.com...