Deserialization problem

[Jan Mynařík]

Hello,

I've got this problem with MapDB 1.0.7. I can't get some records from map. It hasn't happened during testing, then it has started to appear on one of our production deployments, recently on a second one. We have around 50 production deployments now ... and growing.

There has been no changes in DB/entities and especially not in particular field (mentioned in stacktrace), that field is unchanged for 2 years. It is not caused by previous MapDB upgrades as we do handle our own export-update-import procedure. Actually, this is happening during our export as we cycle through all records in MapDB.

You can find stack trace below. There are some "censored" parts, I can't disclose company's private details (may be possible in private communication).

Can someone help me? I need to get this working properly. And possibly not loosing any data (or maybe just that one record :-)

Best regards,

Jan Mynařík

java.lang.RuntimeException: Could not instantiate class
    at org.mapdb.SerializerPojo.deserializeUnknownHeader(SerializerPojo.java:483)
    at org.mapdb.SerializerBase.deserialize3(SerializerBase.java:1216)
    at org.mapdb.SerializerBase.deserialize(SerializerBase.java:1132)
    at org.mapdb.SerializerBase.deserialize(SerializerBase.java:867)
    at org.mapdb.SerializerPojo.deserialize(SerializerPojo.java:701)
    at org.mapdb.HTreeMap$2.deserialize(HTreeMap.java:135)
    at org.mapdb.HTreeMap$2.deserialize(HTreeMap.java:121)
    at org.mapdb.Store.deserialize(Store.java:297)
    at org.mapdb.StoreDirect.get2(StoreDirect.java:475)
    at org.mapdb.StoreWAL.get2(StoreWAL.java:336)
    at org.mapdb.StoreWAL.get(StoreWAL.java:320)
    at org.mapdb.Caches$HashTable.get(Caches.java:246)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
    at org.mapdb.TxEngine.get(TxEngine.java:192)
    at org.mapdb.TxEngine$Tx.getNoLock(TxEngine.java:427)
    at org.mapdb.TxEngine$Tx.get(TxEngine.java:402)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
    at org.mapdb.TxEngine.get(TxEngine.java:192)
    at org.mapdb.TxEngine$Tx.getNoLock(TxEngine.java:427)
    at org.mapdb.TxEngine$Tx.get(TxEngine.java:402)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
    at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1089)
    at org.mapdb.HTreeMap$HashIterator.findNextLinkedNode(HTreeMap.java:1054)
    at org.mapdb.HTreeMap$HashIterator.<init>(HTreeMap.java:978)
    at org.mapdb.HTreeMap$KeyIterator.<init>(HTreeMap.java:1116)
    at org.mapdb.HTreeMap$KeySet.iterator(HTreeMap.java:818)

...
Caused by: java.lang.IllegalArgumentException: Can not set java.lang.Long field <PACKAGE_NAME>.ContactGroup.parentId to java.lang.String
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
    at sun.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)
    at java.lang.reflect.Field.set(Field.java:758)
    at org.mapdb.SerializerPojo.setFieldValue(SerializerPojo.java:346)
    at org.mapdb.SerializerPojo.deserializeUnknownHeader(SerializerPojo.java:478)
... 41 more

... and in another production deployment caused by:

Caused by: java.lang.IllegalArgumentException: Can not set boolean field <PACKAGE_NAME>.Event.audio to java.lang.Long
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
    at sun.reflect.UnsafeBooleanFieldAccessorImpl.set(UnsafeBooleanFieldAccessorImpl.java:86)
    at java.lang.reflect.Field.set(Field.java:758)
    at org.mapdb.SerializerPojo.setFieldValue(SerializerPojo.java:346)
    at org.mapdb.SerializerPojo.deserializeUnknownHeader(SerializerPojo.java:478)
... 36 more

[Jan Mynařík]

Anybody, please help!

Dne pondělí 4. května 2015 12:05:34 UTC+2 Jan Mynařík napsal(a):

[Christian MICHON]

Are you using a consistent JDK throughout?

And what about the hw specs of your production ?

Do you have a reproducible test case?

[Jan Mynařík]

Hello Christian,

Thanks for you interest.

JDK is the same on production machines, I use JDK8. Production machines are Raspberry Pi model B+.

Unfortunately, I still haven't managed to create a test case. If it were easily reproducible, I would dive into MapDB sources myself or create a workaround.

I was hoping someone had a similar problem.

Regards,

Jan

--
You received this message because you are subscribed to a topic in the Google Groups "MapDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mapdb/3iyJabiisIs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mapdb+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Snellman]

Could be hardware i.e. lack of ECC memory.

[Jan Kotek]

Hi,

 

sorry for delay.

 

There are two cases. It is trying to assign wrong field type.

 

Caused by: java.lang.IllegalArgumentException: Can not set java.lang.Long field <PACKAGE_NAME>.ContactGroup.parentId to java.lang.String

Caused by: java.lang.IllegalArgumentException: Can not set boolean field <PACKAGE_NAME>.Event.audio to java.lang.Long

Is there chance the class structure has changed somehow? Also mapdb tries to use get/set methods, some changes in those could cause problem.

 

I can thing of two mistakes in mapdb:

 

1) class catalog got corrupted. This could be tested by fetching the same data type (Event and ContactGroup) from different key. If that fails, class catalog is corrupted. It could be restored with some work

 

2) some data corruption (only this record fails). It could be workarounded sort of by removing those two fields for now (ContactGroup.parentId and Event.audio).

 

I could provide patched MapDB version to restore remaining data in HashMap (iterator skips failed items).

 

Jan

--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.

[Jan Mynařík]

Hi Jan,

thanks for your answer. And sorry for my late reply, I've got a deadline in another project.

Those two classes haven't changed for a long time (last changes: ContactGroup 2013-11-27, Event 2013-12-05). Problems appear in version that was built 2015-04-03, not with previous versions (although previous versions with MapDB 1.0.7 are not used much).

Currently, I am waiting to get my hands on the real DB from production machines, so that I can try if it is really problem of only one record in those class tables.

I have created a version of our application that skips failing items during export (when reported problem happens), it is simply try-catch inside a for loop. I am waiting to test it against the real DB I am waiting for.

I will let you know.

Thanks!

Jan

--
You received this message because you are subscribed to a topic in the Google Groups "MapDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mapdb/3iyJabiisIs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mapdb+un...@googlegroups.com.

[Jan Mynařík]

Hello Jan,

I've finally got that "bad" production database and have some news.

Our backup is the only place where those exceptions happen. All data is readable during normal operations of our application. Even that map, where it fails during backup, is fully readable. First I thought it was caused by using snapshot for backup, using standard tx didn't help. Then I tried to downgrade from 1.0.7 to 1.0.6 but that didn't help either.

The only difference I could find was that during backup we go through result of db.getAll() and then go through individual values, e.g. map. In backup it fails on map.keySet() during construction of iterator. I suspect there might be a problem with hash function. From my investigations it looks that database is not corrupted, only during backup and traversing a map it can jump to some other place in database and read something somewhere wrong as it is a record of a "problematic" class instance.

We really need this fixed, so that I've got permission from my boss to give you our problematic database and structures (and any other sources needed). But as this data is confidential, I can send it only directly to you as it cannot be published.

Thanks a lot!

Jan