org.xadisk.filesystem.exceptions.XASystemBootFailureException

[Kannan Vairavan]

Hi Nitin,
We encountered the followinig exception while trying to deploy our
application in the glassfish server. What could be the issue?

Thanks,
Kannan Vairavan

Jan 23, 2012 6:36:11 PM : The XADisk instance has encoutered a critial
issue and could not be booted. Such a condition is very rare. If you
think you have setup everything right for XADisk to work, please
consider discussing in XADisk forums, or raising a bug with details.

org.xadisk.filesystem.exceptions.XASystemBootFailureException: The
XADisk instance has encoutered a critial issue and could not be
booted. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details.
at
org.xadisk.filesystem.NativeXAFileSystem.<init>(NativeXAFileSystem.java:
186)
at
org.xadisk.filesystem.NativeXAFileSystem.bootXAFileSystem(NativeXAFileSystem.java:
202)
at
org.xadisk.connector.XADiskResourceAdapter.start(XADiskResourceAdapter.java:
38)
at
com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.startResourceAdapter(ActiveOutboundResourceAdapter.java:
156)
at
com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundResourceAdapter.java:
128)
at
com.sun.enterprise.connectors.inbound.ActiveInboundResourceAdapterImpl.init(ActiveInboundResourceAdapterImpl.java:
86)
at
com.sun.enterprise.connectors.ActiveRAFactory.instantiateActiveResourceAdapter(ActiveRAFactory.java:
130)
at
com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(ActiveRAFactory.java:
101)
at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:
216)
at
com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(ConnectorRuntime.java:
337)
at
com.sun.enterprise.connectors.module.ConnectorDeployer.load(ConnectorDeployer.java:
174)
at
com.sun.enterprise.connectors.module.ConnectorDeployer.load(ConnectorDeployer.java:
82)
at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:175)
at
org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:
216)
at
com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:
338)
at
com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:
183)
at
org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:
272)
at com.sun.enterprise.v3.admin.CommandRunnerImpl
$1.execute(CommandRunnerImpl.java:305)
at
com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:
320)
at
com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:
1176)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access
$900(CommandRunnerImpl.java:83)
at com.sun.enterprise.v3.admin.CommandRunnerImpl
$ExecutionContext.execute(CommandRunnerImpl.java:1235)
at com.sun.enterprise.v3.admin.CommandRunnerImpl
$ExecutionContext.execute(CommandRunnerImpl.java:1224)
at
com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:
365)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:
204)
at
com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
166)
at
com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
100)
at
com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:
245)
at
com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
791)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:
693)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
at
com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:
170)
at
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:
135)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
102)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
88)
at
com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:
76)
at
com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:
53)
at
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:
57)
at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
at com.sun.grizzly.util.AbstractThreadPool
$Worker.doWork(AbstractThreadPool.java:330)
at com.sun.grizzly.util.AbstractThreadPool
$Worker.run(AbstractThreadPool.java:309)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.resource.spi.work.WorkRejectedException:
java.lang.InterruptedException, error code: 0
at
com.sun.enterprise.connectors.work.WorkCoordinator.setException(WorkCoordinator.java:
315)
at
com.sun.enterprise.connectors.work.WorkCoordinator.lock(WorkCoordinator.java:
365)
at
com.sun.enterprise.connectors.work.CommonWorkManager.startWork(CommonWorkManager.java:
268)
at
com.sun.enterprise.connectors.work.WorkManagerProxy.startWork(WorkManagerProxy.java:
94)
at
org.xadisk.filesystem.NativeXAFileSystem.<init>(NativeXAFileSystem.java:
182)
... 41 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at
com.sun.enterprise.connectors.work.WorkCoordinator.lock(WorkCoordinator.java:
346)
... 44 more

[Kannan Vairavan]

To add more info for this issue, 

our development environment is CentOS/Glassfish v3.01/JDK1.6_23

our QA environment is RHEL/Glassfish v3.01/JDK1.6_23

The application was working fine in our dev environment. When we deployed in our QA servers, we got this error.

I also tried deploying the XADisk.rar file as a standalone resource module in development and QA environments.

It worked fine in development but failed with the same exception in QA.

Thanks,

Kannan Vairavan

[Nitin Verma]

Hi Kannan,

In general, deploying xadisk over RHEL/Glassfish should not be a
problem, more so because it goes well with CentOS/Glassfish and
Windows/Glassfish (where I had tested). I mean, there should not be
anything OS specific going on here. I think it can be something to do
with Glassfish configuration.

I tried if I could figure out something from the stack-trace. The root
exception is an "InterruptedException" encountered by the work-manager
of Glassfish. I also briefly looked at the relevant glassfish source,
but couldn't figure out. I would suggest to do some more diagnosis and
(glassfish-specific) information gathering. Let me know if I can help
somewhere. Right now, you can try some options like:

-you can download the xadisk source. This exception is arising during
xadisk bootup (i.e. when xadisk.rar gets deployed). During bootup,
xadisk submits a couple of "Work" objects to the "Work-Manager" of the
JavaEE server (these "Work-*" are JCA jargons, think of Work/Work-
Manager as Runnable/ThreadPool where individual Work instances are
submitted for execution to the Work-Manager).

-in the above trace, as indicated by
"org.xadisk.filesystem.NativeXAFileSystem.<init>(NativeXAFileSystem.java:
182", the submission of work object "gatheringDiskWriter" is failing,
but the other submissions before it were successful. I guess the above
exception should not be related to this particular work object. May be
you can try deploying multiple times and see if the trace line number
of "NativeXAFileSystem.java" remains the same (#182).

-a trivial option is to try some clean-ups possible with the glassfish
installation and restart it.

I hope this could help. Let me know how it goes.

Thanks,
Nitin

[Kannan Vairavan]

Hi Nitin,

As you said, the issue is with the glassfish configuration. In our QA ENV, we have configured the ORB Thread pool - MAX_THREAD_POOL_SIZE to be 5.

When we increased this to 20, we were able to deploy the app. In our DEV ENV, we had the default MAX_THREAD_POOL_SIZE for all the Thread Pools as 200.

Thanks,

Kannan Vairavan