Deployment manager extremely poor performance

[Ricky]

Hello,
I have run into some serious problems with our dmgr (5.0.2.16). It runs
on a seprate server (SLES8) and there is 5 separate nodes federated
into the cell. The performance have been slow for a couple of months
but in the last days it has gone from poor performance to no
performance, no changes in configuration have been performed.. Deploys
are out of the question and synchronization can´t be performed. The
load has been as high as 40 and we are forced to keep dmgr down, when
we bring it up again the problem are back. All servers have been booted
and applications on nodes perform as expected. There is no memory or
CPU-problems on the dmgr server as far as we can tell. We havent seen
anything out of the ordinary when it comes to I/O or traffic. We have
reported the problem to IBM but if anyone recognizes the problem,
please let me know.

Regards,
Rickard

[Dexthor]

In the dmgr's SystemOut.log and in the Dmgr's ffdc/ folder, do you see
any changes or logs when it is spinning the load up and churning CPU ?

Also take a threaddump on the dmgr process and see if anything
significant is happening in the process.

-Dexthor.

[adw...@pulpjava.com]

These issues usually boil down to database connections, or session
management, but the symptoms do seem peculiar.

When IBM solves the problem, please don't leave this as a mystery. Let
us know what the solution to the problem was.

Cheers!

-Cameron McKenzie

Author of What is WebSphere?

www.pulpjava.com
Free Multimedia J2EE and Java Tutorials: www.mcnz.com
Free Sun Certification Exams and Tutorials: www.scja.com

[rswartwood]

I have seen very poor dmgr performance as the result of networking
issues. In one case there were 3 DNS servers in the resolv.conf. The
first one was no longer responding. So every time there was a DNS
lookup there would be a delay equivalent to the TCP timeout. The issue
was resolved by changing the order of the DNS servers in the list. We
could see this by just doing a ping on the hostname.

So you might want to look at your etc/hosts and etc/resolv.conf for any
irregularities.

The following is a java program that tests the getLocalHost function:

import java.net.*;
import java.io.*;

public class checkdns
{
public static void main(String args[])
{
try
{
long start = System.currentTimeMillis();
InetAddress local= InetAddress.getLocalHost();
System.out.println("InetAddress = " + local + ", using
getLocalHost()");

InetAddress local2 = InetAddress.getByName(local.getHostAddress());
System.out.println("InetAddress = " + local2 + ", using
getByName(local.getHostAddress()");

if (local2.getHostName().equals(local2.getHostAddress()))
System.out.println("=====> Reverse DNS problem! " + " WebSphere
performance will suffer.");
else
System.out.println("DNS configuration appears correct.");
long elapsed = System.currentTimeMillis() - start;
System.out.println("elapsed time=" + elapsed);
}
catch (Exception e )
{
System.out.println("Exception: " + e + " caught!");
e.printStackTrace();

[Ricky]

Hello,
Thanx for your input.The problem turned out to be something we had
expected before but had tried to fix the wrong way:
Information from files under xxx/xxx/DeploymentManager/config is sent
out to the different nodes. We found out that that the
xxx/xxx/DeploymentManager/config/temp was too big so we removed it
causing dmgr to create a new temp-directory without unnecesarry
information. The problem was that we saved an old copy of temp under
xxx/xxx/DeploymentManager/config and it turned out that everything
under the config directory was sent out to the nodes even our big
TEMP.backup-file. After removing everything the dmgr runs like a dream.
Strange though, that everything under config is sent out and that the
temp directory needs to be removed once in a while (old configurations
is not removed I think). Maybe this is a known issue(?).

/Rickard

rswartwood skrev: