Problem starting Websphere Application Server 7 on Linux AS5 Machine

[sju...@temenos.com]

Hi,

Finally I am here to share this problem with all Websphere GEEKS and its a wiered one.

I have downloaded a trial 90 days version of Websphere Application Server 7 for Linux and installed it on my AS5 machine. After the installation I apply the following command to start my server.

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin$*./startServer.sh server1*

and it got started normally.I signed in; install a resource adapter and my enterprise application sucessfully. When i select the newly installed application to 'Start' it hanged for long time so i just applied './stopServer.sh server1' command to stop it but it failed.

*I think the next step was not good.... I killed the process using 'kill -9 Process_Id'*. Now after that when I am trying to start the server again its failing with the following message on screen. I am copying all the logs I have from 'logs/server1/' directory. *Not a single log file giving me any clue to resolve the problem or even stating any problem.*

*_Messages On Screen_*

hml-rh5xeon-~/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin: ./startServer.sh server1
ADMU0116I: Tool information is being logged in file
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/startServer.log
ADMU0128I: Starting tool with the AppSrv01 profile
ADMU3100I: Reading configuration for server: server1
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3011E: Server launched but failed initialization. startServer.log,
SystemOut.log(or job log in zOS) and other log files under
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1 should
contain failure information.

*_native_stderr.log_*

************ Start Display Current Environment ************
WebSphere Platform 7.0.0.0 [BASE 7.0.0.0 r0835.03] running with process name hml-rh5xeonNode01Cell\hml-rh5xeonNode01\server1 and process id 30481
Host Operating System is Linux, version 2.6.18-128.1.1.el5xen
Java version = 1.6.0, Java Compiler = j9jit24, Java VM name = IBM J9 VM
was.install.root = /opt/IBM/WebSphere/AppServer
user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
Java Home = /opt/IBM/WebSphere/AppServer/java/jre
ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
Java Library path = /opt/IBM/WebSphere/AppServer/java/jre/lib/i386:/opt/IBM/WebSphere/AppServer/bin::/usr/lib
************* End Display Current Environment *************

*_native_stdout.log_*

************ Start Display Current Environment ************
WebSphere Platform 7.0.0.0 [BASE 7.0.0.0 r0835.03] running with process name hml-rh5xeonNode01Cell\hml-rh5xeonNode01\server1 and process id 30481
Host Operating System is Linux, version 2.6.18-128.1.1.el5xen
Java version = 1.6.0, Java Compiler = j9jit24, Java VM name = IBM J9 VM
was.install.root = /opt/IBM/WebSphere/AppServer
user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
Java Home = /opt/IBM/WebSphere/AppServer/java/jre
ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
Java Library path = /opt/IBM/WebSphere/AppServer/java/jre/lib/i386:/opt/IBM/WebSphere/AppServer/bin::/usr/lib
************* End Display Current Environment *************

*_startServer.log_*

************ Start Display Current Environment ************
Host Operating System is Linux, version 2.6.18-128.1.1.el5xen
Java version = J2RE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260-20080816_22093 (JIT enabled, AOT enabled)
J9VM - 20080816_022093_lHdSMr
JIT - r9_20080721_1330ifx2
GC - 20080724_AA, Java Compiler = j9jit24, Java VM name = IBM J9 VM
was.install.root = /opt/IBM/WebSphere/AppServer
user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
Java Home = /opt/IBM/WebSphere/AppServer/java/jre
ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
Java Library path = /opt/IBM/WebSphere/AppServer/java/jre/lib/i386:/opt/IBM/WebSphere/AppServer/bin::/usr/lib
Current trace specification = *=info
************* End Display Current Environment *************
[4/9/09 12:11:55:834 BST] 00000000 ManagerAdmin I TRAS0017I: The startup trace state is *=info.
[4/9/09 12:11:55:894 BST] 00000000 AdminTool A ADMU0128I: Starting tool with the AppSrv01 profile
[4/9/09 12:11:55:895 BST] 00000000 AdminTool A ADMU3100I: Reading configuration for server: server1
[4/9/09 12:11:56:138 BST] 00000000 AdminTool A ADMU3200I: Server launched. Waiting for initialization status.
[4/9/09 12:12:08:618 BST] 00000000 AdminTool A ADMU3011E: Server launched but failed initialization. startServer.log, SystemOut.log(or job log in zOS) and other log files under /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1 should contain failure information.

*_SystemErr.log_*

************ Start Display Current Environment ************
WebSphere Platform 7.0.0.0 [BASE 7.0.0.0 r0835.03] running with process name hml-rh5xeonNode01Cell\hml-rh5xeonNode01\server1 and process id 30481
Host Operating System is Linux, version 2.6.18-128.1.1.el5xen
Java version = 1.6.0, Java Compiler = j9jit24, Java VM name = IBM J9 VM
was.install.root = /opt/IBM/WebSphere/AppServer
user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
Java Home = /opt/IBM/WebSphere/AppServer/java/jre
ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
Java Library path = /opt/IBM/WebSphere/AppServer/java/jre/lib/i386:/opt/IBM/WebSphere/AppServer/bin::/usr/lib
************* End Display Current Environment *************
[4/9/09 12:12:08:542 BST] 0000000b SystemErr R PI with the name 'xml' can occur only in the beginning of the document.

*_SystemOut.log_*

Its quit big so I am attching a file to make this post readable.

*If any one can help me on this please.*

Thanks and Regards

[Ken Hygh]

sju...@temenos.com wrote:
> Hi,
>
> Finally I am here to share this problem with all Websphere GEEKS and its a wiered one.
>
> I have downloaded a trial 90 days version of Websphere Application Server 7 for Linux and installed it on my AS5 machine. After the installation I apply the following command to start my server.
>
> /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin$*./startServer.sh server1*
>
> and it got started normally.I signed in; install a resource adapter and my enterprise application sucessfully. When i select the newly installed application to 'Start' it hanged for long time so i just applied './stopServer.sh server1' command to stop it but it failed.
>
> *I think the next step was not good.... I killed the process using 'kill -9 Process_Id'*. Now after that when I am trying to start the server again its failing with the following message on screen. I am copying all the logs I have from 'logs/server1/' directory. *Not a single log file giving me any clue to resolve the problem or even stating any problem.*
>
> *_Messages On Screen_*

[snip]

> *_SystemErr.log_*
>
> ************ Start Display Current Environment ************
> WebSphere Platform 7.0.0.0 [BASE 7.0.0.0 r0835.03] running with process name hml-rh5xeonNode01Cell\hml-rh5xeonNode01\server1 and process id 30481
> Host Operating System is Linux, version 2.6.18-128.1.1.el5xen
> Java version = 1.6.0, Java Compiler = j9jit24, Java VM name = IBM J9 VM
> was.install.root = /opt/IBM/WebSphere/AppServer
> user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
> Java Home = /opt/IBM/WebSphere/AppServer/java/jre
> ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
> Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
> Java Library path = /opt/IBM/WebSphere/AppServer/java/jre/lib/i386:/opt/IBM/WebSphere/AppServer/bin::/usr/lib
> ************* End Display Current Environment *************
> [4/9/09 12:12:08:542 BST] 0000000b SystemErr R PI with the name 'xml' can occur only in the beginning of the document.
>
> *_SystemOut.log_*
>
> Its quit big so I am attching a file to make this post readable.
>
> *If any one can help me on this please.*
>
> Thanks and Regards

I think that line from SystemErr.log is pointing to one of the .xml
files in the profile's configuration being damaged. Unless you have a
recent backup, it's probably easiest to create another profile.

Ken

[sju...@temenos.com]

Any one...

[Brian S Paskin]

Hi, Which OS are you using (Red Hat, Suse, etc). I see Xen at the end of the kernel name. Are you running it under a Xen partition?

Regards,
Brian

[sju...@temenos.com]

Yes i am using Red Hat Linux and i think yes i was using Xen but i the mean time of this conversation the server has been moved to another one where i installed the Application server but didnt tested yet. But the new server dont have Xen anymore. If i had the same problem again i post it again here. By the way what seems to be wrong here????

[Brian S Paskin]

Hi, One issue could be that WebSphere v7 is not supported on Xen Server. Without doing any research, most of the time that error is because a port that is needed is already in use.

Regards,
Brian

[Brian S Paskin]

Hi, i should clairfy my previous statement. As long as Red Hat ES and Suse ES are running on top of Xen Server then it is supported, but is not supported on the base Xen Server instance.

Regards,
Brian

[sju...@temenos.com]

Hi Brian,

Thanks for the info, So we can say its because of the port allocation error...because as i mentioned earlier It get started for the first time and i installed my resource adapter and EE application.

So which port is the problem I need to look into it????

Appriciated your response.

[Brian S Paskin]

Hi, usually it would say the port name in the SystemOut.log or in the FFDC log. Then you have to refer back the name to the port in the WebSphere configuration.

Regards,
Brian

[sju...@temenos.com]

Hi,

I think I had same issue on my new platfrom. As I have copied all the log earlier. New AS5 machine had the same issue got started once but when i tried to start my EE application after sucessfuly installation it got stuck so have to kill it using kill command then now again its not getting started and failing with the same error message as above...
please help :)

[sju...@temenos.com]

Hi,

Any thoughts on the above problem.

Where usually we define the server pors which it need to get start on.... I am usinf sample server server1 with no extra settings..???

Thanks again

[kh...@us.ibm.com]

I don't see anything in your logs about ports.

What is the exact error message you're getting, and when are you getting it? If you uninstall your application does the server come up cleanly?

Ken

[sju...@temenos.com]

Hi Ken,

I was trying with base-trial version. I dont know but when i unistalled it goes smooth. Then I tried to work with ND [full version] and its get started. Now i am testing it if i get any problem I'll post here again. by the way Websphere default port is 9080. but this time i dont on my machine its something else. where can i find the port information of websphere???? which file i mean???

Thanks again

[kh...@us.ibm.com]

Actually, WebSphere uses many ports, far more than just the one default_host port.

/profiles/ /config/cells/ /nodes//serverindex.xml

Ken

[sju...@temenos.com]

There you Ken,

Again same thing happend with Network Deployment version 7. I installed my resource adaptor and my EE application. everything goes smoothly and get installed. When i select my EE [.war] installation from EE application menu and click 'Start' the websphere icon appears and it stuck there for too long and application didn't get started. I try to navigate to other links on adminstration console but browser page crashes with 'Failed To Connect' message.

Then I thought of restarting the app server for that i have to stop my running server first, so i applied following command and get the results [Failed to start; same problem as I have mentioned with Trial Version above so I killed the process last time but than failed to start again which I do not want to do it this time];

*_Screen Output_*

*[root@ukhml-devas5 bin]# ./stopServer.sh server1*
*ADMU0116I: Tool information is being logged in file*
*/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/logs/server1/stopServer.log*
*ADMU0128I: Starting tool with the AppSrv01 profile*
*ADMU3100I: Reading configuration for server: server1*
*Realm/Cell Name: *
*Username: wasadm*
*Password:*
*ADMU3201I: Server stop request issued. Waiting for stop status.*
*ADMU3111E: Server stop requested but failed to complete.*

*ADMU0111E: Program exiting with error:*
*com.ibm.websphere.management.exception.AdminException: ADMU3060E:*
*Timed out waiting for server shutdown.*
*ADMU1211I: To obtain a full trace of the failure, use the -trace option.*
*ADMU0211I: Error details may be seen in the file:*
*/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/logs/server1/stopServer.log*

*_Active Processes [ps -ef | grep java] _*

*[root@ukhml-devas5 bin]# ps -ef | grep java*
*root 7220 1 2 12:06 pts/2 00:03:39 /home/tecapp/IBM/WebSphere7/AppServer/java/bin/java -Declipse.security -Dwas.status.socket=48722 -Dosgi.install.area=/home/tecapp/IBM/WebSphere7/AppServer -Dosgi.configuration.area=/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/configuration -Djava.awt.headless=true -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M -Xbootclasspath/p:/home/tecapp/IBM/WebSphere7/AppServer/java/jre/lib/ext/ibmorb.jar:/home/tecapp/IBM/WebSphere7/AppServer/java/jre/lib/ext/ibmext.jar -classpath /home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/properties:/home/tecapp/IBM/WebSphere7/AppServer/properties:/home/tecapp/IBM/WebSphere7/AppServer/lib/startup.jar:/home/tecapp/IBM/WebSphere7/AppServer/lib/bootstrap.jar:/home/tecapp/IBM/WebSphere7/AppServer/lib/jsf-nls.jar:/home/tecapp/IBM/WebSphere7/AppServer/lib/lmproxy.jar:/home/tecapp/IBM/WebSphere7/AppServer/lib/urlprotocols.jar:/home/tecapp/IBM/WebSphere7/AppServer/deploytool/itp/batchboot.jar:/home/tecapp/IBM/WebSphere7/AppServer/deploytool/itp/batch2.jar:/home/tecapp/IBM/WebSphere7/AppServer/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms50m -Xmx256m -Xcompressedrefs -Dws.ext.dirs=/home/tecapp/IBM/WebSphere7/AppServer/java/lib:/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/classes:/home/tecapp/IBM/WebSphere7/AppServer/classes:/home/tecapp/IBM/WebSphere7/AppServer/lib:/home/tecapp/IBM/WebSphere7/AppServer/installedChannels:/home/tecapp/IBM/WebSphere7/AppServer/lib/ext:/home/tecapp/IBM/WebSphere7/AppServer/web/help:/home/tecapp/IBM/WebSphere7/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/home/tecapp/IBM/WebSphere7/AppServer/derby -Dcom.ibm.itp.location=/home/tecapp/IBM/WebSphere7/AppServer/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/home/tecapp/IBM/WebSphere7/AppServer -Dpython.cachedir=/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01 -Dcom.ibm.security.jgss.debug=off -Dcom.ibm.security.krb5.Krb5Debug=off -Djava.security.auth.login.config=/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/properties/wsjaas.conf -Djava.security.policy=/home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/properties/server.policy com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /home/tecapp/IBM/WebSphere7/AppServer/profiles/AppSrv01/config ukhml-devas5Node01Cell ukhml-devas5Node01 server1*
*root 19750 6876 0 14:08 pts/2 00:00:00 grep java*

Let me know if you any other info from any other log file to solve this problem. I am not going to touch this machine for now until i debug the problem here.

[sju...@temenos.com]

I found following message in the log

*_SystemOut.log_*

[5/14/09 12:42:12:359 BST] 00000010 ThreadMonitor W WSVR0605W: Thread "Default : 0" (0000001e) has been active for 759076 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:77)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.jbase.jremote.io.inflow.RequestListenerService.runNIOServer(RequestListenerService.java:314)
at com.jbase.jremote.io.inflow.RequestListenerService.run(RequestListenerService.java:244)
at com.ibm.ejs.j2c.work.WorkProxy.run(WorkProxy.java:399)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1527)
[5/14/09 12:42:12:363 BST] 00000010 ThreadMonitor W WSVR0605W: Thread "WebContainer : 2" (00000015) has been active for 760356 milliseconds and may be hung. There is/are 2 thread(s) in total in the server that may be hung.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:196)
at java.lang.Thread.join(Thread.java:615)
at java.lang.ApplicationShutdownHooks.run(ApplicationShutdownHooks.java:90)
at java.lang.Shutdown.runHooks(Shutdown.java:100)
at java.lang.Shutdown.sequence(Shutdown.java:144)
at java.lang.Shutdown.exit(Shutdown.java:189)
at java.lang.Runtime.exit(Runtime.java:101)
at java.lang.System.exit(System.java:275)
at com.temenos.t24browser.xml.XMLTemplate.createXMLDocument(XMLTemplate.java:243)
at com.temenos.t24browser.xml.XMLTemplate. (XMLTemplate.java:195)
at com.temenos.t24browser.xml.XMLRequestManager. (XMLRequestManager.java:155)
at com.temenos.t24browser.utils.PropertyManager. (PropertyManager.java:51)
at com.temenos.t24browser.utils.BrowserParameters.(BrowserParameters.java:51)
at com.temenos.t24browser.servlets.BrowserServlet.readParameters(BrowserServlet.java:683)
at com.temenos.t24browser.servlets.BrowserServlet.init(BrowserServlet.java:140)
at javax.servlet.GenericServlet.init(GenericServlet.java:241)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:325)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:165)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.initialize(ServletWrapper.java:1588)
at com.ibm.wsspi.webcontainer.extension.WebExtensionProcessor.createServletWrapper(WebExtensionProcessor.java:98)
at com.ibm.ws.webcontainer.webapp.WebApp.getServletWrapper(WebApp.java:936)
at com.ibm.ws.webcontainer.webapp.WebApp.getServletWrapper(WebApp.java:857)
at com.ibm.ws.webcontainer.webapp.WebApp.initializeTargetMappings(WebApp.java:538)
at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinish(WebApp.java:360)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:292)
at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:99)
at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:167)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:722)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:607)
at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:376)
at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:668)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1162)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1313)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:611)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:938)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:740)
at com.ibm.ws.runtime.component.ApplicationMgrImpl$1.run(ApplicationMgrImpl.java:1312)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:4279)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:4367)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1317)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2089)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:437)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:122)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:380)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:644)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:613)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1215)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:45)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:599)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:36)
at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:599)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:243)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1085)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:848)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:773)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1313)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1206)
at com.ibm.ws.management.commands.AdminServiceCommands$InvokeCmd.execute(AdminServiceCommands.java:251)
at com.ibm.ws.console.core.mbean.MBeanHelper.invoke(MBeanHelper.java:239)
at com.ibm.ws.console.appdeployment.ApplicationDeploymentCollectionAction.execute(ApplicationDeploymentCollectionAction.java:564)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1486)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:528)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1443)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1384)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:104)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:852)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:785)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:443)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:325)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1070)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:273)
at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:455)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:319)
at com.ibm.isclite.container.controller.InformationController.processForwardConfig(InformationController.java:207)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:279)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1486)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:528)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1443)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1384)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.setUpCommandAssistence(WSCUrlFilter.java:927)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState(WSCUrlFilter.java:494)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter(WSCUrlFilter.java:315)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:852)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:785)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:443)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:859)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1557)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:173)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:272)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1016)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:639)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1772)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:202)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:766)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:896)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1527)
[5/14/09 13:59:32:885 BST] 0000001a ArchiveDeploy W ADMA0142W: Trying to close the archive that is already closed.
[5/14/09 14:49:43:092 BST] 00000022 AdminHelper A ADMN1020I: An attempt is made to stop the server1 server. (User ID = defaultWIMFileBasedRealm/wasadm)

[Ken Hygh]

sju...@temenos.com wrote:
> I found following message in the log
>
> *_SystemOut.log_*
>
> [5/14/09 12:42:12:359 BST] 00000010 ThreadMonitor W WSVR0605W: Thread "Default : 0" (0000001e) has been active for 759076 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:77)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> at com.jbase.jremote.io.inflow.RequestListenerService.runNIOServer(RequestListenerService.java:314)
> at com.jbase.jremote.io.inflow.RequestListenerService.run(RequestListenerService.java:244)
> at com.ibm.ejs.j2c.work.WorkProxy.run(WorkProxy.java:399)
> at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1527)
> [5/14/09 12:42:12:363 BST] 00000010 ThreadMonitor W WSVR0605W: Thread "WebContainer : 2" (00000015) has been active for 760356 milliseconds and may be hung. There is/are 2 thread(s) in total in the server that may be hung.

[snip]

> [5/14/09 13:59:32:885 BST] 0000001a ArchiveDeploy W ADMA0142W: Trying to close the archive that is already closed.
> [5/14/09 14:49:43:092 BST] 00000022 AdminHelper A ADMN1020I: An attempt is made to stop the server1 server. (User ID = defaultWIMFileBasedRealm/wasadm)

I'm guessing that you have an init-on-startup servlet with an init()
method that is never returning, hanging the startup process, and since
there's a blocked thread the stopServer.sh command will never be able to
shutdown the server.

There's definitely something in your application that's being called on
application startup that's hanging.

Ken