Chill and Kryo 5

[Nicholas Nezis]

I'm trying to update the Twitter Chill library to use Kryo 5. I think I have the code updated, but I'm having trouble updating the spec tests. Would someone be able to help me figure out my issue?

Here is my pull request: https://github.com/twitter/chill/pull/514

[Thomas Heigl]

Hi Nicholas,

What exact problems are you facing?

I'd suggest you run the test against Kryo 5 and update each failing entry in the test-file with the new serialized representation.

Thomas

On Wed, Mar 17, 2021 at 7:02 AM Nicholas Nezis <nichola...@gmail.com> wrote:

I'm trying to update the Twitter Chill library to use Kryo 5. I think I have the code updated, but I'm having trouble updating the spec tests. Would someone be able to help me figure out my issue?

Here is my pull request: https://github.com/twitter/chill/pull/514

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
---
You received this message because you are subscribed to the Google Groups "kryo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kryo-users/374c32d5-327d-4a4c-8992-73b60c3fd498n%40googlegroups.com.

[Nicholas Nezis]

Hi, 

Sorry. My previous message was painfully lacking in details. I've been working through some of the test results and updating the serialized snippets that are being compared. My initial confusion was perhaps related to a non Kryo issue with the way the tests are run depending on which version of Scala is targeted. But here is a specific test error that I suspect must be related to my code changes.

Test output:

[info] - should handle WrappedArray instances *** FAILED ***

[info]   successful serialization roundtrip for ArraySeq((1,1), (2,2), (3,3)) (KryoSpec.scala:168)

The relevant code:
"handle WrappedArray instances" in {
val tests = Seq(
Array((1, 1), (2, 2), (3, 3)).toSeq,
Array((1.0, 1.0), (2.0, 2.0)).toSeq,
Array((1.0, "1.0"), (2.0, "2.0")).toSeq
)
tests.foreach(_ should roundtrip)
}

I'm seeing this stack trace which might be related.

[info] KryoSerializers and KryoDeserializers

[info] - should round trip any non-array object *** FAILED ***

[info]   com.esotericsoftware.kryo.kryo5.KryoException: java.lang.ClassCastException: class [Ljava.lang.String; cannot be cast to class java.lang.String ([Ljava.lang.String; and java.lang.String are in module java.base of loader 'bootstrap')

[info] Serialization trace:

[info] unsafeArray (scala.collection.immutable.ArraySeq$ofRef)

[info] vargs (com.twitter.chill.TestVarArgs)

[info]   at com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:101)

[info]   at com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:107)

[info]   at com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:626)

[info]   at com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:70)

[info]   at com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:107)

[info]   at com.esotericsoftware.kryo.kryo5.Kryo.writeClassAndObject(Kryo.java:695)

[info]   at com.twitter.chill.SerDeState.writeClassAndObject(SerDeState.java:64)

[info]   at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:116)

[info]   at com.twitter.chill.BaseProperties.serialize(BaseProperties.scala:24)

[info]   at com.twitter.chill.BaseProperties.serialize$(BaseProperties.scala:23)

[info]   ...

[info]   Cause: java.lang.ClassCastException: class [Ljava.lang.String; cannot be cast to class java.lang.String ([Ljava.lang.String; and java.lang.String are in module java.base of loader 'bootstrap')

[info]   at com.esotericsoftware.kryo.kryo5.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:156)

[info]   at com.esotericsoftware.kryo.kryo5.Kryo.writeObjectOrNull(Kryo.java:676)

[info]   at com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:79)

[info]   at com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:107)

[info]   at com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:626)

[info]   at com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:70)

[info]   at com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:107)

[info]   at com.esotericsoftware.kryo.kryo5.Kryo.writeClassAndObject(Kryo.java:695)

[info]   at com.twitter.chill.SerDeState.writeClassAndObject(SerDeState.java:64)

[info]   at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:116)

[info]   ...

[Joachim Durchholz]

Seems to be a confusion between String... and String.

Hints 1:

> java.lang.ClassCastException:
> class [Ljava.lang.String; cannot be cast to class java.lang.String

It's trying to cast a String[] to a String.

> ([Ljava.lang.String; and java.lang.String are in module java.base of
> loader 'bootstrap')

It's telling you that both classes are in the same module (java.base) of
the same loader (bootstrap), i.e. you don't have any "class Foo cannot
be cast to class Foo (because they are in different class loaders)"
shenanigans.

> > [info] vargs (com.twitter.chill.TestVarArgs)

Seems like your test code is about varargs, that's why String... is my
top suspect.

HTH

Regards
Jo

Am 18.03.21 um 03:39 schrieb Nicholas Nezis:

> Hi,
>
> Sorry. My previous message was painfully lacking in details. I've been
> working through some of the test results and updating the serialized
> snippets that are being compared. My initial confusion was perhaps
> related to a non Kryo issue with the way the tests are run depending on
> which version of Scala is targeted. But here is a specific test error
> that I suspect must be related to my code changes.
>
> Test output:
> [info] - should handle WrappedArray instances *** FAILED ***
> [info]   successful serialization roundtrip for ArraySeq((1,1), (2,2),
> (3,3)) (KryoSpec.scala:168)
>
> The relevant code

> <https://github.com/twitter/chill/blob/987fdecfa0ee33a105f2fdfc2dd76ccfdc6770f1/chill-scala/src/test/scala/com/twitter/chill/KryoSpec.scala#L162>:

> <https://groups.google.com/d/msgid/kryo-users/374c32d5-327d-4a4c-8992-73b60c3fd498n%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the "kryo-users"
> group.
> http://groups.google.com/group/kryo-users
> ---
> You received this message because you are subscribed to the Google
> Groups "kryo-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kryo-users+...@googlegroups.com

> <mailto:kryo-users+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kryo-users/873f5794-bfdb-4eb7-beee-58071f4650d4n%40googlegroups.com
> <https://groups.google.com/d/msgid/kryo-users/873f5794-bfdb-4eb7-beee-58071f4650d4n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Thomas Heigl]

Hi Nicholas,

Can you try to create an absolutely minimal example that demonstrates this test failure? Ideally without any dependencies on external libraries.

The problem is hard to diagnose from just the stacktrace and I would need to debug this locally.

Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/kryo-users/873f5794-bfdb-4eb7-beee-58071f4650d4n%40googlegroups.com.

[Nicholas Nezis]

I'll try my best to produce a minimal example, but still working through the test cases without a great understanding. So will need to keep working on it for now. But I will most likely have some hopefully specific questions as I work through the updates.

My current question, I updates some of the serialized test results, but now have a test assertion failing due to the `void` type. It is mentioned as a special type here:: https://github.com/nicknezis/chill/blob/6fc09e65371692c5803bb62a847ef0cf974141ed/chill-scala/src/test/scala-2.12-/com/twitter/chill/SerializedExamplesData.scala#L35

I don't know why I'm getting this test failure. It seems that it expects the test result to be item 10, but it came up as 9.

```

[info] With ScalaKryoInstantiators, the registered classes

[info] - should serialize as expected to the correct value (see above for details) *** FAILED ***

[info]   9 did not equal 10 [2] is registered with ID 9, but expected 10 (SerializedExamplesOfStandardDataSpec.scala:85)

[info] - should all be covered by an example *** FAILED ***

[info]   List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148) did not equal List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147) there are approx -1 examples missing for preregistered classes (SerializedExamplesOfStandardDataSpec.scala:57)

[info]   Analysis:

[info]   List(148: 148 -> )
```

[Nicholas Nezis]

Actually, I was able to bump all of the IDs to remove "void" from ID 9. Now I'm continuing the update of the serialized results. Will keep trucking on for now, but I am still curious why ID 9 changed. But at least not blocked for the moment.