Very long deployment time after FixPack 15 + Web Service FP install

[alexis....@sophis.net]

Hello,

I recently upgraded my RAD test server (base_v61) to 6.1.0.15 with the Web Services Feature Pack and since then, I experience very long deployment times (when publishing directly from RAD) and very long resource access time (for example, when modifying a resource JNDI path). Very long as in 5-6 minutes even for the most simple web project (1 JDBC reference, 1 JSF page, 1 Page class).

Process Explorer shows the WebSphere Application Server Java process consuming 100% CPU and lots of I/O reading activity at a rate of 20-40 Mb/s for several minutes (the hard drive led does not show any specific hard drive activity though, so it is in-memory, or network related).

Do you know of anything that could explain this behaviour ?

I have the following log when deploying, that was not occurring before:

wtp W org.eclipse.jst.j2ee.commonarchivecore.internal.util.ArchiveUtil createTempDirectory Warning: createTempDirectory: deleted pre-existing file [ D:\IBM2\SDP70\runtimes\base_v61\profiles\AppSrvWSFP01\wstemp\wstemp\app_1197a998d62\mrg\SphParty.war44656.tmp ] with basename [ D:\IBM2\SDP70\runtimes\base_v61\profiles\AppSrvWSFP01\wstemp\wstemp\app_1197a998d62\mrg\SphParty.war ] and parent directory [ D:\IBM2\SDP70\runtimes\base_v61\profiles\AppSrvWSFP01\wstemp\wstemp\app_1197a998d62\mrg ]

I also have the following FFDC exception log when deploying:

------Start of DE processing------ = [23/04/08 10:24:48:026 BST] , key = java.lang.NoClassDefFoundError com.ibm.ws.management.application.AppManagementImpl.getExtensionInfo 942
Exception = java.lang.NoClassDefFoundError
Source = com.ibm.ws.management.application.AppManagementImpl.getExtensionInfo
probeid = 942
Stack Dump = java.lang.NoClassDefFoundError: org.apache.tools.ant.BuildException
at java.lang.J9VMInternals.verifyImpl(Native Method)
at java.lang.J9VMInternals.verify(J9VMInternals.java:66)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:127)
at java.lang.Class.newInstanceImpl(Native Method)
at java.lang.Class.newInstance(Class.java:1300)
at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:157)
at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:759)
at org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:243)
at org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:51)
at com.ibm.ws.management.application.ExtensionHelper.getExtensionInfo(ExtensionHelper.java:958)
at com.ibm.ws.management.application.ExtensionHelper.getProvidersAndWeights(ExtensionHelper.java:791)
at com.ibm.ws.management.application.ExtensionHelper.getExtensionProviders(ExtensionHelper.java:631)
at com.ibm.ws.management.application.ExtensionHelper.processClientInstallExtensions(ExtensionHelper.java:106)
at com.ibm.websphere.management.application.AppManagementFactory.collectInstallTaskInfo(AppManagementFactory.java:150)
at com.ibm.websphere.management.application.AppManagementFactory.readPartialArchive(AppManagementFactory.java:179)
at com.ibm.ws.management.application.EditApplication.getApplicationInfo(EditApplication.java:331)
at com.ibm.ws.management.application.AppManagementImpl._getApplicationInfo(AppManagementImpl.java:608)
at com.ibm.ws.management.application.AppManagementImpl.getApplicationInfo(AppManagementImpl.java:573)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:615)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:62)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:615)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:265)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1089)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:971)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:231)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:238)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:833)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:802)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1081)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:974)
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java:139)
at com.ibm.ws.management.connector.rmi.RMIConnectorService.invoke(RMIConnectorService.java:179)
at com.ibm.ws.management.connector.rmi._RMIConnectorService_Tie.invoke(_RMIConnectorService_Tie.java:395)
at com.ibm.ws.management.connector.rmi._RMIConnectorService_Tie._invoke(_RMIConnectorService_Tie.java:160)
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:613)
at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:466)
at com.ibm.rmi.iiop.ORB.process(ORB.java:503)
at com.ibm.CORBA.iiop.ORB.process(ORB.java:1553)
at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2680)
at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2554)
at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:62)
at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1473)

Another thing which takes an unusual amount of time is when accessing the module resource references in the admin console. It stays 5-6 minutes with the following log line:

0000001f ResourceValid A validateResources

Please tell me if there is anything I could do to trace this issue, thanks a lot

Alexis

[Ben_]

Did you check the WebSphere and Rational support sites ?

There have been a number of fixes for slow deployments.

E.g.:
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg1PK58273
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg1PK54414

If it's another new problem and you entitle for support, you'll want to open
a PMR with IBM Support.

[g.t....@gmail.com]

Hi, I have also recently upgraded to Rational Software Architect 6.1.0.15 with WSFP and experiencing very similar problems.

Very long deployment times with 100% CPU usage and the same FFDC exceptions:

------Start of DE processing------ = 25/04/08 08:57:45:999 BST , key = java.lang.NoClassDefFoundError com.ibm.ws.management.application.AppManagementImpl.getExtensionInfo 942

Exception = java.lang.NoClassDefFoundError

Source = com.ibm.ws.management.application.AppManagementImpl.getExtensionInfo

probeid = 942

Stack Dump = java.lang.NoClassDefFoundError: org.apache.tools.ant.BuildException

+ at java.lang.J9VMInternals.verifyImpl(Native Method)+

+ at java.lang.J9VMInternals.verify(J9VMInternals.java:68)+

+ at java.lang.J9VMInternals.initialize(J9VMInternals.java:129)+

+ at java.lang.Class.newInstanceImpl(Native Method)+

+ at java.lang.Class.newInstance(Class.java:1300)+

+ at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableEx tension(RegistryStrategyOSGI.java:157)+

+ at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension( ExtensionRegistry.java:759)+

+ at org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtensi on(ConfigurationElement.java:243)+

+ at org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableE xtension(ConfigurationElementHandle.java:51)+

+ at com.ibm.ws.management.application.ExtensionHelper.getExtensionInfo(ExtensionHel per.java:958)+

+ at com.ibm.ws.management.application.ExtensionHelper.getProvidersAndWeights(Extens ionHelper.java:791)+

+ at com.ibm.ws.management.application.ExtensionHelper.getExtensionProviders(Extensi onHelper.java:631)+

+ at com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensi ons(ExtensionHelper.java:236)+

+ at com.ibm.ws.management.application.UninstallSchedulerImpl.setupTasks(UninstallSc hedulerImpl.java:145)+

+ at com.ibm.ws.management.application.SchedulerImpl.run(SchedulerImpl.java:250)+

+ at java.lang.Thread.run(Thread.java:810)+

Dump of callerThis =

null

Performing default dump from com.ibm.ws.management.dm.JmxDM = Fri Apr 25 08:57:46 BST 2008

Dump of callerThis =

null

+Data for directive defaultjmx obtained. =

Dump complete for com.ibm.ws.management.dm.JmxDM = Fri Apr 25 08:57:46 BST 2008

We have a support contract so I will open a PMR and get back to you with any news.

[alexis....@sophis.net]

gTh,

I did not read your last line about support. I am not entitled to such support and would be very glad indeed to hear what IBM have to say about this problem !

Thanks,

Alexis

[alexis....@sophis.net]

Ben,

Many thanks for your reply. My problem seems to be a combination of both issues (multiple compilations of the binding classes, and mine contains quite a few !) plus a problem with the deploying directory not being used (there is a warning message, although not very explicit). What is strange is that all these fixes are supposed to be included in FP15, and they actually appeared with it in my case...

gTh,

This is indeed the same problem. FYI, I worked around this problem by re-enabling the "mnimize application files copied to the server" in the server preferences, that I had disabled for another problem. My deployment time is back to reasonable times now.

Kind regards, and thanks to all for your answers,

Alexis

[g.t....@gmail.com]

Thanks for your quick respone.

I have now opened a PMR.

In the meantime I will investigate what effect the "minimize application files copied to the server" setting has.

Do you know what affect this will have upon deploying applications e.g. if I make small changes to the code, will it still auto-deploy or will I need to 'manually' re-deploy the application to see the changes?

Best regards

Gareth

[alexis....@sophis.net]

I think this "minimal" mode is the standard mode. Everything works fine for me with this mode, including automatic deployment.

regards,

Alexis

[hema...@yahoo.com]

Hi,

Iam facing exactly the same problem after upgrading to Fix pack 15 and webservice FP install.

Any approaches or solutions to the problem will be great help

I tried changing the websphere setting , but no effect

thanks

[Ben_]

There have a been a number of fixes for such behavior (e.g. PK54414 but you
have it already in CF15).

Try with the latest Fix Pack.