DataCorruption while removing elements (mapDB is 2.0-beta13)

Michaël C.

unread,

Feb 23, 2016, 4:47:10 AM2/23/16

to MapDB

Hi all,

My current version of mapDB is 2.0-beta13.

In my case i'm working with two or more process that needs to read in a fresh HTreeMap<K, V>.

Updates are scheduling every minutes and do two operations :

- remove entries (.remove(K key))

- put new entries (.put(K key, V Value)

The exception DataCorruption is raised when it commits these operations while another process is reading the map.

I'm assuming the exception is caused by the remove operation, because if i only put entries there is no problem.

Maker in refresh/create mode

Maker maker = DBMaker 
 .fileDB(new File(mapdbMmapDirectory.get() + this.serializerName)) 
 .fileMmapEnable() 
 .fileMmapCleanerHackEnable() 
 .closeOnJvmShutdown();

Maker in read mode

Maker maker = DBMaker
 .fileDB(new File(mapdbMmapDirectory.get() + this.serializerName))
 .fileMmapEnable()
 .fileMmapCleanerHackEnable()
 .readOnly()
 .cacheLRUEnable()
 .cacheSize(10000)
 .closeOnJvmShutdown();

HTreeMap

HTreeMap<K, V> data = maker.make().hashMapCreate(this.serializerName)
 .keySerializer(this.keySerializer())
 .valueSerializer(this.valueSerializer())
 .makeOrGet();

FYI data contains 158 000 entries and the exception is raised randomly, even if i remove only 1K entries.

Best regards

Jan Kotek

unread,

Feb 23, 2016, 4:48:59 AM2/23/16

to ma...@googlegroups.com

Hi,

would you have a stacktrace?

Jan

--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Message has been deleted

Jan Kotek

unread,

Feb 23, 2016, 7:34:25 AM2/23/16

to ma...@googlegroups.com

This is not necessary data corruption. It is an assertion that serializer read less data than it wrote. It could be caused by your serializer.

What type of serialization are you using?

Jan

On Tuesday, February 23, 2016 01:54:03 Michaël C. wrote:

Hi,

Thanks for your quick answer

org.mapdb.DBException$DataCorruption: Data were not fully read, check your serializer. Read size:391, expected size:396

at org.mapdb.Store.deserialize(Store.java:418)

at org.mapdb.StoreWAL.get2(StoreWAL.java:527)

at org.mapdb.Store.get(Store.java:253)

at org.mapdb.Engine$ReadOnlyWrapper.get(Engine.java:369)

at org.mapdb.HTreeMap.getInner(HTreeMap.java:649)

at org.mapdb.HTreeMap.get(HTreeMap.java:583)

at com.netadge.grinder.dsr.MapDbTest.testLookUp(MapDbTest.java:99)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)

at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Best regards,

Michaël C.

unread,

Feb 23, 2016, 8:10:53 AM2/23/16

to MapDB

Hi,

I'm using a custom serializer like this :

public Serializer<V> valueSerializer() {

                return new Serializer<V>() {

                                                  @Override
                        public void serialize(DataOutput out, V value) throws IOException {
                                out.writeInt(value.XXXXX.intValue());
                                                                   out.writeInt(value.XXXXX.intValue());
                                out.writeInt(value.XXXXX.intValue());
                                out.writeInt(value.XXXXX.intValue());
                                out.writeInt(value.XXXXX.intValue());
                                out.writeInt(value.XXXXX.intValue());
                                out.writeInt(value.XXXXX.intValue());
                                out.writeUTF(value.XXXXX);
                        }

                       @Override
                        public V deserialize(DataInput in, int available) throws IOException {
                                return new V(
                                        in.readInt(),
                                        in.readInt(),
                                        in.readInt(),
                                        in.readInt(),
                                        in.readInt(),
                                        in.readInt(),
                                        in.readInt(),
                                        in.readUTF());
                        }



                       @Override
                        public boolean isTrusted() {
                                return true;
                        }
                };
        }

FYI fields are never NULL or empty.

Best regards,

To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "MapDB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+unsubscribe@googlegroups.com.

Jan Kotek

unread,

Feb 23, 2016, 8:19:50 AM2/23/16

to ma...@googlegroups.com

That looks ok.

What happens if you set 'isTrusted' to false? That will make more strict size checking, and should for sure diagnose data corruption

To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "MapDB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "MapDB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.

Michaël C.

unread,

Feb 23, 2016, 8:24:33 AM2/23/16

to MapDB

Same issue if i set isTrusted to false.

Best Regards,

...

Michaël C.

unread,

Feb 23, 2016, 10:37:39 AM2/23/16

to MapDB

Hi Jan,

During my test i was preloading file content into disk cache using :

Store.forDB(this.db).fileLoad();

If i disable this operation, i get a new error message

org.mapdb.DBException$EngineGetVoid: Recid passed to Engine.get() does not exist. Possible data corruption!

at org.mapdb.StoreDirect.indexValGet(StoreDirect.java:1577)

at org.mapdb.StoreWAL.indexValGet(StoreWAL.java:307)

at org.mapdb.StoreWAL.get2(StoreWAL.java:517)

at org.mapdb.Store.get(Store.java:253)

at org.mapdb.Engine$ReadOnlyWrapper.get(Engine.java:369)

at org.mapdb.HTreeMap.getInner(HTreeMap.java:662)

at org.mapdb.HTreeMap.get(HTreeMap.java:583)

at com.*******.MapDbTest.testLookUp(MapDbTest.java:97)