Exception on medium-to-large Transactions

[supah]

Hi all,

i just had the issue with an Exception thrown after running a long XA
transaction (sending a lot of files to a remote XADisk instance wiht
approx. 1000 files and 500MB in total). I will investigate further and
report back some more detailed stack traces when i have time to.

Just wanted to give a short indication as it seems the underlying NIO
socket channel is closed for some reason?

Btw: I want to point out that XADisk is such a great library! Saved me
a lot of trouble! Thanks Nitin :)

Cheers,
Daniel

[Daniel Scheibe]

Hi again,

i filed a bug in the JIRA: http://java.net/jira/browse/XADISK-105

Cheers,
Daniel

[Nitin Verma]

Hi Daniel,

In the bug you have mentioned:

"There are two threads running on a single XAFileSystem instance, but
each thread has gotten it's own XASession, maybe i'm missing some
threading issue i should be aware of?".

Actually, in case of remote xadisk, the XAFileSystem reference
(pointing to the remote xadisk instance) and subsequently derived
objects like Session cannot be used by multiple threads concurrently.
Such remote XAFileSystem objects keep their own communication channel
until they are disconnected.

I would make a note to document this. Thanks for letting everyone know
about it.

Thanks,
Nitin

[Daniel Scheibe]

Hi Nitin,

thanks for you response. I took a minute and changed the way my
application uses XADisk. Now i'm using a separate XADiskFileSystem/
XASession for each thread. This turned out to work a lot better but
after a while another Exception popped up:

Exception in thread "pool-1-thread-5"
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:
488)
at
org.xadisk.filesystem.workers.observers.CriticalWorkersListener.workCompleted(CriticalWorkersListener.java:
31)
at org.xadisk.filesystem.standalone.StandaloneWorkManager
$WorkRunnable.run(StandaloneWorkManager.java:81)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.resource.spi.work.WorkException:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
... 4 more
Caused by:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:
488)
at
org.xadisk.filesystem.workers.TransactionTimeoutDetector.doWorkOnce(TransactionTimeoutDetector.java:
42)
at org.xadisk.filesystem.workers.TimedWorker.run(TimedWorker.java:42)
at org.xadisk.filesystem.standalone.StandaloneWorkManager
$WorkRunnable.run(StandaloneWorkManager.java:75)
... 3 more
Caused by:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:
488)
at org.xadisk.filesystem.NativeSession.rollback(NativeSession.java:
934)
at
org.xadisk.filesystem.NativeSession.rollbackPrematurely(NativeSession.java:
121)
at
org.xadisk.filesystem.NativeSession.rollbackAsynchronously(NativeSession.java:
112)
at
org.xadisk.filesystem.workers.TransactionTimeoutDetector.doWorkOnce(TransactionTimeoutDetector.java:
38)
... 5 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:289)
at
org.xadisk.filesystem.workers.GatheringDiskWriter.ensureLogFileCapacity(GatheringDiskWriter.java:
340)
at
org.xadisk.filesystem.workers.GatheringDiskWriter.forceWrite(GatheringDiskWriter.java:
317)
at
org.xadisk.filesystem.workers.GatheringDiskWriter.transactionCompletes(GatheringDiskWriter.java:
232)
at org.xadisk.filesystem.NativeSession.rollback(NativeSession.java:
927)
... 8 more

I re-ran my application a few times and it seems to also throw another
Exception sometimes:

Exception in thread "pool-1-thread-7"
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:
488)
at
org.xadisk.filesystem.workers.observers.CriticalWorkersListener.workCompleted(CriticalWorkersListener.java:
31)
at org.xadisk.filesystem.standalone.StandaloneWorkManager
$WorkRunnable.run(StandaloneWorkManager.java:81)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.resource.spi.work.WorkException:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
... 4 more
Caused by:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:
488)
at
org.xadisk.bridge.server.conversation.ConversationGateway.run(ConversationGateway.java:
77)
at org.xadisk.filesystem.standalone.StandaloneWorkManager
$WorkRunnable.run(StandaloneWorkManager.java:75)
... 3 more
Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:182)
at
org.xadisk.bridge.server.conversation.ConversationContext.updateWithConversation(ConversationContext.java:
81)
at
org.xadisk.bridge.server.conversation.ConversationGateway.run(ConversationGateway.java:
68)
... 4 more
Exception in thread "pool-1-thread-5"
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:
488)
at
org.xadisk.filesystem.workers.observers.CriticalWorkersListener.workCompleted(CriticalWorkersListener.java:
31)
at org.xadisk.filesystem.standalone.StandaloneWorkManager
$WorkRunnable.run(StandaloneWorkManager.java:81)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.resource.spi.work.WorkException:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
... 4 more
Caused by:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:
488)
at
org.xadisk.filesystem.workers.TransactionTimeoutDetector.doWorkOnce(TransactionTimeoutDetector.java:
42)
at org.xadisk.filesystem.workers.TimedWorker.run(TimedWorker.java:42)
at org.xadisk.filesystem.standalone.StandaloneWorkManager
$WorkRunnable.run(StandaloneWorkManager.java:75)
... 3 more
Caused by:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeSession.checkIfCanContinue(NativeSession.java:
1092)
at org.xadisk.filesystem.NativeSession.rollback(NativeSession.java:
850)
at
org.xadisk.filesystem.NativeSession.rollbackPrematurely(NativeSession.java:
121)
at
org.xadisk.filesystem.NativeSession.rollbackAsynchronously(NativeSession.java:
112)
at
org.xadisk.filesystem.workers.TransactionTimeoutDetector.doWorkOnce(TransactionTimeoutDetector.java:
38)
... 5 more
Caused by: javax.resource.spi.work.WorkException:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at org.xadisk.filesystem.standalone.StandaloneWorkManager
$WorkRunnable.run(StandaloneWorkManager.java:81)
... 3 more
Caused by:
org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The
XADisk instance has encoutered a critial issue and is no more
available. Such a condition is very rare. If you think you have setup
everything right for XADisk to work, please consider discussing in
XADisk forums, or raising a bug with details
at
org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:
488)
at
org.xadisk.bridge.server.conversation.ConversationGateway.run(ConversationGateway.java:
77)
at org.xadisk.filesystem.standalone.StandaloneWorkManager
$WorkRunnable.run(StandaloneWorkManager.java:75)
... 3 more
Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:182)
at
org.xadisk.bridge.server.conversation.ConversationContext.updateWithConversation(ConversationContext.java:
81)
at
org.xadisk.bridge.server.conversation.ConversationGateway.run(ConversationGateway.java:
68)
... 4 more

I tried to track it down and it seems that inside the class
org.xadisk.bridge.server.conversation.ConversationGateway at line 42
'ByteBuffer buffer = ByteBuffer.allocate(1000);' a buffer is
allocated, which could be accessed beyond it's size. Of course my
assumption is based on a quick look and can be completely wrong but
1000 is roughly the amount of files/calls to XADisk inside the
transaction for each thread. Interestingly i never run into trouble
before hitting the 1000 files limit. Is there some sort of maximum
transaction size? Anything you could recommend to fix this?

Thanks in advance!

Cheers,
Daniel

[Nitin Verma]

Hi Daniel,

No, there is no transaction limit. Do you get these two exceptions
even when using a single thread and single session? Can you please
tell in brief if there are any specific steps to reproduce the
exceptions?

Thanks,
Nitin

[Nitin Verma]

Hi Daniel,

I created a test-case (described below) and found it working fine in
repeated executions; I can't reproduce the two exceptions
(ClosedChannelException and BufferOverflowException) you have reported
in your latest post.

I have a jvm running one xadisk instance and another jvm running two
threads as clients to this xadisk instance. Both of these threads have
created their own remote reference of XAFileSystem (it is required as
we discussed above) and perform this operation: start a session and
create 1100 files of size 250KB each. For each of these files, a
XAFileOutputStream is opened after creating them and a small text
string is written in a loop (leading to 250KB of data in each file).

Do you see some differences among this and your test-case which could
possibly generate the above two exceptions.

Just wanted to add something regarding the first exception
(ClosedChannelException). Briefly visiting the source code, there does
not seem to be any reason for this ClosedChannelException except the
moment when the xadisk instance is shutdown while some io operations
are running.

Thanks,
Nitin

[Daniel Scheibe]

Hi Nitin,

sorry for not coming back to you earlier, i was busy with some stuff. I
will create and send you a test-case in the next couple of days which
should reproduce the exceptions. Maybe i'm doming something else wrong.
The only difference i see with your test case is that i had ~1100 files
of ~400kb in size instead of 250kb but that shouldn't be the matter.

Thanks for keeping me posted on this, really appreciated.

Cheers,
Daniel