ClosedChannelException on LRU cache under load

[Dave Lamy]

Hello--

We are using MapDB 0.9.9 as an LRU disk-based cache:

contextDataDb = DBMaker.newFileDB(storageFile)

  .closeOnJvmShutdown()

  .cacheLRUEnable()

  .make();

contextDataMap = contextDataDb.getHashMap("workflowContextData");

We are using transactions to persist changes as they happen (originally we were not, but were seeing corruption issues):

<snippet>

contextDataMap.put(storageKey, objectMapper.writeValueAsBytes(json));

contextDataDb.commit();

</snippet>

When the system gets under load, we are seeing the whole thing come crashing down with a nio ClosedChannelException:

java.nio.channels.ClosedChannelException

                at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)

                at sun.nio.ch.FileChannelImpl.truncate(FileChannelImpl.java:319)

                at org.mapdb.Volume$FileChannelVol.tryAvailable(Volume.java:683)

                at org.mapdb.Volume.ensureAvailable(Volume.java:58)

                at org.mapdb.StoreWAL.logAllocate(StoreWAL.java:310)

                at org.mapdb.StoreWAL.put(StoreWAL.java:225)

                at org.mapdb.EngineWrapper.put(EngineWrapper.java:54)

                at org.mapdb.Caches$LRU.put(Caches.java:46)

                at org.mapdb.HTreeMap.putInner(HTreeMap.java:559)

                at org.mapdb.HTreeMap.put(HTreeMap.java:458)

                at com.levelsbeyond.plugin.workflow.context.WorkflowContextDataServiceImpl.updateContextDataValue(WorkflowContextDataServiceImpl.java:208)

                at com.levelsbeyond.plugin.workflow.context.WorkflowContextDataServiceImpl.updateContextDataValue(WorkflowContextDataServiceImpl.java:187)

                at com.levelsbeyond.plugin.workflow.step.VideoConversionStepExecution.monitorMediaConversion(VideoConversionStepExecution.java:398)

                at com.levelsbeyond.plugin.workflow.step.VideoConversionStepExecution.execute(VideoConversionStepExecution.java:112)

                at com.levelsbeyond.plugin.workflow.execution.WorkflowStepExecutor$StepCallable.call(WorkflowStepExecutor.java:237)

                at com.levelsbeyond.plugin.workflow.execution.WorkflowStepExecutor$StepCallable.call(WorkflowStepExecutor.java:185)

                at java.util.concurrent.FutureTask.run(FutureTask.java:262)

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

                at java.util.concurrent.FutureTask.run(FutureTask.java:262)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                at java.lang.Thread.run(Thread.java:744)

The MapDB is only used by the WorkflowContextDataServiceImpl class, though it is heavily multi-threaded, there is potential for 50+ threads to be writing simultaneously.  We definitely only see this issue under load.

I'm frankly unsure what to do to correct/avoid this issue.  Can someone provide any hints or info on things I can try?

Thanks,

Dave Lamy

[Dave Lamy]

<bump>

I could really use at least a "You're doing it wrong" response, in fact I'm hoping that's the case.  Could the problem be with using the global commit in a multithreaded environment?  Should I be using the TxMaker instead?  Perhaps a memory mapped file would be preferable?  What would cause the ClosedChannelException?

I unfortunately don't have a great test bed for this-- I can't reproduce it locally as I don't have the horsepower/throughput needed to make it crash.  We are only seeing this at a customer site which of course makes things much worse, and I can't do a lot of "best guess" deploys.  

I'm unfortunately going to have to consider an alternative solution if we can't figure this out.  I'm very excited about this product but this is a killer for us ATM.

Thanks--

Dave

[Jan Kotek]

Hi,

 

I have no idea why it is throwing this exception.

It looks like there is race condition between log-update and commit, which should not happen in theory.

 

I will have to do code audit around stack trace you send.

 

For start you could workaround it by using memory mapped file:

 

DBMaker...

.mmapFileEnable()

 

Also async write could workaround this problem:

 

DBMaker...

.asyncWriteEnable()

 

Hope this helps,

Jan

--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dave Lamy]

Thanks for the response Jan!  I'll try these and let you know what I find.

Dave

[Jan Kotek]

I believe this is fixed in 0.9.11 which will come out today.

 

Jan

For more options, visit https://groups.google.com/d/optout.

[Dave Lamy]

Woot!  Thanks Jan, looking forward to it.  The changes I made based on your suggestions unfortunately did not change the runtime situation.

I'll be on the lookout for the release and will let you know if there are continuing issues.  Thanks for giving it attention!

Dave

To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+unsubscribe@googlegroups.com.

[Dave Lamy]

Hi Jan, any word on the .11 release?  Is this fix in the current snapshot?  Would you be comfortable with us using it or are there continuing issues that would make that a bad idea?

[Dave Lamy]

Never mind, I see you got the release out.  We’ll get it out there as soon as we get a window.  Sorry for not checking the website first!

Dave

[Jan Kotek]

Hi,

 

the allocation logic has changed a lot, so there is very good chance this is fixed. I have not investigated this too deeply yet.

 

Jan

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.