Buffer underflow when reading object

[Giuseppe Villani]

I'm using a library (Paper db) which is based on Kryo. 

These are the dependecies it is using:

+--- com.esotericsoftware:kryo:4.0.1@jar

+--- com.esotericsoftware:reflectasm:1.11.3@jar

+--- com.esotericsoftware:minlog:1.3.0@jar

In order to read the content of the database this method is used:

private <E> E readContent(File originalFile, Kryo kryo) throws FileNotFoundException, KryoException {

        final Input i = new Input(new FileInputStream(originalFile));

        try {

            final PaperTable<E> paperTable = kryo.readObject(i, PaperTable.class);

            return paperTable.mContent;

        } finally {

            i.close();

        }

    }

Now this works fine, but sometimes we get a `com.esotericsoftware.kryo.KryoException` with "Buffer underflow." text,  when `readObject()` is executed. 

com.esotericsoftware.kryo.io.Input.require (Input.java:199)

com.esotericsoftware.kryo.io.Input.readVarInt (Input.java:373)

com.esotericsoftware.kryo.util.DefaultClassResolver.readClass (DefaultClassResolver.java:127)

com.esotericsoftware.kryo.Kryo.readClass (Kryo.java:693)

com.esotericsoftware.kryo.serializers.ObjectField.read (ObjectField.java:118)

com.esotericsoftware.kryo.serializers.FieldSerializer.read (FieldSerializer.java:543)

com.esotericsoftware.kryo.Kryo.readObject (Kryo.java:712)

I understand that the problem might be related to the interaction between the two library, but I would like to try to fix the problem on that library and hopefully you can give me some hints about what can be the cause of the problem.

[Thomas Heigl]

Hi Giuseppe,

Do you serialize and deserialize the class using the exact same version of Kryo with the same configuration?

Please try to extract a minimal roundtrip test-case from your project. It is very difficult to diagnose such issues without a self-contained test-case.

Best regards,

Thomas

[Giuseppe Villani]

Hi, Thomas,

Unfortunately I don't have a lot of info yet. What I can tell so far is that it only happens for some users, all of them with Android 10. But there are also some Android 10 devices that don't have the problem. 

Also the same read operation is executed in different places: in one place the operation always succeed, in the other one I have the problem. The only difference is that in the second case, mistakenly, the operation was done in the main thread. Do you think this can be the source of the problem? If so, why only on some Android 10 devices?

I'll try to create some test-case and see if I can reproduce it.

Thanks,

Giuseppe

[Giuseppe Villani]

And yes, all serialization and deserialization is done with the same version and configuration.

[Joachim Durchholz]

> The only difference is that in the second case, mistakenly, the
> operation was done in the main thread. Do you think this can be the
> source of the problem? If so, why only on some Android 10 devices?

Multithreading is a common source of "sometimes we have a problem". If
you have a race condition that's "very unlikely to happen" (say, once in
a thousand cases), then if you have 1000 users, it is almost certain
that somebody will hit the problem, while your chances of reproducing
the problem are pretty slim.

You need to have some 150% enforced policy in the software that prevents
race conditions. It doesn't matter whether it is enforced through type
system, annotations, programmer discipline, testing, or whatever - but
it needs to be iron-clad and so reliable that you'd trust your family's
life with it.
Without that, you're in for misery for the rest of the app's lifetime.
General rules:
1) Avoid locking. Locks tend to deadlock, which is a nonlocal bug
(potentially far-away parts of the code can contribute to deadlock).
2) To avoid locks, you make threads communicate only read-only data
structures.
3) If you need shared mutable data structures, use the lock-free
collections and queues that the JDK offers. These structures are mutable
but the data items you put in and take out still need to be immutable
"value objects".
4) If you have a big mutable data structure (such as the GUI, or an
internal state machine, etc.), make a single thread responsible for
maintaining it. Never let other threads write to it, that way lies
madness. If other threads absolutely need to read it, you can use a read
lock on the data structure - but then no other thread may ever wait for
the managing thread, otherwise you risk deadlock.

BTW none of this is related to Kryo. If you have multithreading issues,
it is pretty likely that Kryo is merely the victim.

HTH
Jo

Am 26.06.20 um 08:29 schrieb Giuseppe Villani:

> <http://com.esotericsoftware.kryo.io>.Input.require (Input.java:199)
> com.esotericsoftware.kryo.io
> <http://com.esotericsoftware.kryo.io>.Input.readVarInt

> (Input.java:373)
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass
> (DefaultClassResolver.java:127)
> com.esotericsoftware.kryo.Kryo.readClass (Kryo.java:693)
> com.esotericsoftware.kryo.serializers.ObjectField.read
> (ObjectField.java:118)
> com.esotericsoftware.kryo.serializers.FieldSerializer.read
> (FieldSerializer.java:543)
> com.esotericsoftware.kryo.Kryo.readObject (Kryo.java:712)
>
>
> I understand that the problem might be related to the
> interaction between the two library, but I would like to try to
> fix the problem on that library and hopefully you can give me
> some hints about what can be the cause of the problem.
>

> --
> You received this message because you are subscribed to the "kryo-users"
> group.
> http://groups.google.com/group/kryo-users
> ---
> You received this message because you are subscribed to the Google
> Groups "kryo-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kryo-users+...@googlegroups.com
> <mailto:kryo-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kryo-users/1f518bce-58cc-4bc6-900b-916baa93e6f1n%40googlegroups.com
> <https://groups.google.com/d/msgid/kryo-users/1f518bce-58cc-4bc6-900b-916baa93e6f1n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Giuseppe Villani]

Hi Toolforger,

I agree with you, but the fact that we were using the main thread for reading doesn't mean there is automatically a multithreading problem.

Of course, if something happens only in rare cases, it is a good hint that it is a multi-threading issue. However here the problem is happening only in Android 10, 

so I am just wondering if accessing the main thread for reading with Kryo can be a problem, assuming there is no race condition with other read/write.

Giuseppe

[Joachim Durchholz]

It is pretty typical for multithreading issues to become visible under
timing variation - which means almost any kind of variation imaginable.
Operating system differences are definitely one of the things that can
affect timing. Operating systems permanently alter timing, trying to
improve responsiveness and reduce delays, stutter, and lag, which can
obviously affect the timing of everything.

It is pretty atypical for mere data manipulation libraries like Kry to
depend on timing; the list of suspects for random behaviour is
unexpectedly-random data first place, multithreading second place.
Main thread or not does not usually matter.
However, if the library in question does multithreading itself, then it
might have a multithreading bug. (I assume Kryo does not do that though.)

Am 26.06.20 um 10:35 schrieb Giuseppe Villani:

> https://groups.google.com/d/msgid/kryo-users/966b7d8a-d23a-456d-9599-49586e95aa4bn%40googlegroups.com
> <https://groups.google.com/d/msgid/kryo-users/966b7d8a-d23a-456d-9599-49586e95aa4bn%40googlegroups.com?utm_medium=email&utm_source=footer>.