Use transactional writes on async flush?
Sam Hendley
Peter Blakeley
I found in the past better to throw away the caches(of search indexes) to speed reload of the application be interested to see your result
cheers pb...
Jan Kotek
> cache that is persistent but I don't care if I lose a small number of updates on an unclean shutdown.
In this case leave default settings and commit periodically. You will only loose data since last commit.
>Sounds like the use case for transactionDisable()
`transactionDisable()` is different. It trades all protections for write speed. Not just on modified data, but on entire store. So on unclean shutdown you are most likely to corrupt entire store.
> asyncWriteEnable() but when I combine them I get index out of bound exceptions.
Async write has some overhead, so it depends on scenario if it actually improves performance.
Please send me stack trace. You might get exception after unclean shutdown while reopening. Otherwise it is bug.
There is a bug in asyncWrite being fixed in MapDB 1.0.5, perhaps it is the same.
> Are these features supposed to work together?
Yes, anything with stack trace is a probably a bug. If not there should be some sort of warning or InvalidArgumentException at config time.
> In general are the "transactions" provided by transacationDisable() "user facing" or are they talking about the writes to the disk?
There are three transactions modes:
1) WriteAheadLog: on by default and crash resistant. Single global transaction per store.
2) transactionDisabled: no WAL, changes written directly to files, data gone if JVM crashes.
3) TxMaker: A concurrent transaction with MVCC serializable snapshots.
In short if you are not sure, use WAL.
> For reference this is my config file for a "InMemory cache with best effort persistence". Anything I can do to keep this basic performance and not lose my cache to corrupted indexes?
return DBMaker
.newFileDB(file)
// do not use this, or you will losoe all data
.transactionDisable()
//disable async, unless tests shows better performance
.asyncWriteEnable()
.asyncWriteFlushDelay(500)
.asyncWriteQueueSize(100000)
//good
.mmapFileEnableIfSupported()
//hard ref cache does not have
.cacheSize(20000000)
.cacheHardRefEnable()
//bad idea unless you know what it does
.commitFileSyncDisable()
.make();
BTW HashMap has optional time based eviction...
>I would be interested in plots also in how you find the reload time of your cache, do you load it in back into memory?
No, store on disk is not loaded into memory.
> I found in the past better to throw away the caches(of search indexes) to speed reload of the application be interested to see your result
That is probably best if your crashes are not frequent and data are not important.
Just disable transactions. In case of crash the MapDB will throw an exception while you reopen the store.
So catch the exception, wipe store (files) and start over.
Option to wipe corrupted store will be added in 1.1 in about two months.
Jan
--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sam Hendley
--
You received this message because you are subscribed to a topic in the Google Groups "MapDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mapdb/fCM3iMjpvmE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mapdb+un...@googlegroups.com.
Sam Hendley
|
5M Entry map |
EhCache |
MapDb |
|
File Size |
2500 MB |
150 MB |
|
Start Time |
157 seconds |
1 second |
|
Bootstrap Time |
60 seconds |
10 seconds |
|
Shutdown Time |
120 seconds |
1 second |
|
Gets a second |
600 K/s |
750 K/s |
|
Max Memory |
4.5 GB |
3 GB |
For reference here were my configurations:
EhCache 2.5.4
<diskStore path="/opt/var/db"/>
<defaultCache
eternal="true"
timeToIdleSeconds="0"
timeToLiveSeconds="0"
overflowToDisk="true"
maxEntriesLocalHeap="20000000"
diskPersistent="true"
maxElementsOnDisk="0"
diskExpiryThreadIntervalSeconds="0"
memoryStoreEvictionPolicy="LRU"
/>
MapDb 1.04
DBMaker
.newFileDB(file)
.transactionDisable()
.asyncWriteEnable()
.asyncWriteFlushDelay(500)
.asyncWriteQueueSize(100000)
.mmapFileEnableIfSupported()
.cacheSize(20000000)
.cacheHardRefEnable()
.commitFileSyncDisable()
.make();
--
Jan Kotek
Problem in 1.0.4 seems like something I fixed recently (fix will be in 1.0.5).
I do not thing there is a deadlock in 1.1.0-SNAPSHOT.
I got simple test case to reproduce issue, but no luck (attached)
Code audit shows offending line in writer:
//if conditions are right, slow down writes a bit
if(asyncFlushDelay!=0 &&
!commitLock.isWriteLocked() &&
size.get()<maxParkSize){
LockSupport.parkNanos(1000L * 1000L * asyncFlushDelay);
}
This can not cause deadlock. It will wait for asyncFlushDelay miliseconds and than continue to dump write queue content.
I believe that this setting is too high: `.asyncWriteFlushDelay(500)`
The `maxParkSize` is 1/4 of queue size, so it always parks even if there are items to be written.
So either increase commit interval to something comparable to asyncQueueSize,
or set flush delay to zero or something smaller.
Jan
Jan Kotek
Nice, could you make blog post out of it? Or could I use it in testimonials?
I could use some user stories.
I believe GC spikes are caused by hard-reference instance cache. It clears itself when free memory is low. Perhaps leave default cache to improve this.
If you run out of memory, decrease cache size.
Also you might call `db.clearCache()` manually.
> .commitFileSyncDisable()
Hm, you may loose data. What is point of WAL if it is not synced?
MapDB could have some problems to recover in this case.
It is probably better to disable transactions and just discard the store in case of unclean shutdown.
WAL performance should be fixed in 1.1 with append-only store and some more features.
Jan
--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.