Published kryo-shaded 5.x
551 views
Skip to first unread message
Nicholas Nezis
Jan 6, 2021, 9:24:36 PM1/6/21
to kryo-users
Hello,
I'm was looking to update https://github.com/twitter/chill with the newer version of Kryo (5.0.3), but I ran into an issue because they are referencing `kryo-shaded` which doesn't seem to exist in the Sonatype Nexus repository.
`kryo-shaded` latest version was 4.0.2. Should there be newer artifacts, or was kryo-shaded deprecated for a different approach?
thomas...@gmail.com
Jan 7, 2021, 8:48:52 AM1/7/21
to kryo-users
Hi Nicholas,
The shaded artifact has been replaced by a versioned artifact. (See https://github.com/EsotericSoftware/kryo#installation).
This approach is more flexible because it allows different versions of Kryo to be used at the same time.
I'll update the migration guide accordingly.
Best,
Thomas
Nicholas Nezis
Jan 7, 2021, 1:40:26 PM1/7/21
to kryo-users
Oh interesting. That's good to know. And this artifact is shaded in a similar fashion so that the compile dependencies are included in the jar? I guess I didn't see it because it was still using the older "com.esotericsoftware.kryo" instead of "com.esotericsoftware".
I'll try using this. Thanks!
Nicholas Nezis
Jan 7, 2021, 2:03:15 PM1/7/21
to kryo-users
I just realized that the class names are different. I understand the desire to have different versions so that someone could support both Kryo4 and Kryo5 classes at the same time. But is there not still a use case for someone wanting to use the standard Kryo classes, but with a shaded jar that includes the dependencies?
As I look to update the twitter/chill library, would you recommendation be to switch to the Kryo5 class? My fear is that this will impact downstream consumers of this library. Is the path forward going to entail everyone using the versioned class names?
Just want to make sure I understand the design decisions so I can articulate them to others as I advocate for upgrading the project.
Thanks.
Thomas Heigl
Jan 7, 2021, 4:29:44 PM1/7/21
to kryo-...@googlegroups.com
But is there not still a use case for someone wanting to use the standard Kryo classes, but with a shaded jar that includes the dependencies?
I'm not sure. So far nobody has asked for a non-versioned shaded JAR.
As I look to update the twitter/chill library, would you recommendation be to switch to the Kryo5 class? My fear is that this will impact downstream consumers of this library. Is the path forward going to entail everyone using the versioned class names?
You can take a look at https://github.com/EsotericSoftware/kryo/issues/650 for the discussions that lead to this new approach. Chill is explicitly mentioned in this comment.
Does chill expose any Kryo classes directly to clients of the library? Or is Kryo abstracted away completely?
Thomas
--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
---
You received this message because you are subscribed to the Google Groups "kryo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kryo-users/0c22fcfa-27d8-4374-a243-6eb2dc0cb60an%40googlegroups.com.
Nicholas Nezis
Jan 7, 2021, 11:17:12 PM1/7/21
to kryo-users
Awesome, I think I'm starting to get a better understanding. I can work with Johny, the author of the comment you linked to. He commented on my PR to their repo. I'll get the answer to your question about if the Kryo classes are exposed. Hopefully they aren't. I shared your v5 Migration Wiki page with Johny which described the approach for reading v4 and v5 at the same time. I might try to update my Chill Pull Request to do this. That should satisfy his concerns. Although he did seem slightly more open to the idea that we could put a disclaimer in the README and warn others to not use Chill + Kryo for long term storage.
I'll report back with future questions or final resolution. Thanks again!
Nicholas Nezis
Jan 8, 2021, 1:51:30 AM1/8/21
to kryo-users
Looking at the Spark code, it seems they do reference the Kryo classes directly. Chill is only providing a couple of supplemental imports.
So I would need to update both sets of code. Are these the two options available for this upgrade?
1. Make a kryo-shaded with non versioned class names, update chill. Users would need to know that upgrading Chill is a breaking change
2. Update Chill and Spark to use the new Kryo 5 versions. Perhaps with both having support to read v4 in addition to the new v5 classes.
Thomas Heigl
Jan 12, 2021, 11:02:45 AM1/12/21
to kryo-...@googlegroups.com
So I would need to update both sets of code. Are these the two options available for this upgrade?
1. Make a kryo-shaded with non versioned class names, update chill. Users would need to know that upgrading Chill is a breaking change
2. Update Chill and Spark to use the new Kryo 5 versions. Perhaps with both having support to read v4 in addition to the new v5 classes.
Correct.
As far as I understand (the decision to create a versioned Kryo artifact was made long before I joined the project), #2 is the preferred option for libraries.
The only issue with #2 is if the library exposes Kryo classes to clients, they will have to update their code (custom serializers etc) to use the versioned packages.
I'll try to get feedback from Martin Grotzke, who originally introduced the versioned artifact.
Thomas
To view this discussion on the web visit https://groups.google.com/d/msgid/kryo-users/de49d68d-1ea6-4924-be92-35c2326497c2n%40googlegroups.com.
Martin Grotzke
Jan 12, 2021, 2:06:25 PM1/12/21
to kryo-...@googlegroups.com, Thomas Heigl
On 1/12/21 12:02 PM, Thomas Heigl wrote:
> So I would need to update both sets of code. Are these the two
> options available for this upgrade?
>
>
>
> 1. Make a kryo-shaded with non versioned class names, update chill.
> Users would need to know that upgrading Chill is a breaking change
> 2. Update Chill and Spark to use the new Kryo 5 versions. Perhaps
> with both having support to read v4 in addition to the new v5 classes.
>
>
> Correct.
>
> As far as I understand (the decision to create a versioned Kryo artifact
> was made long before I joined the project), #2 is the preferred option
> for libraries.
I agree. I would not recommend option 1 (according to thoughts shared in
> So I would need to update both sets of code. Are these the two
> options available for this upgrade?
>
>
>
> 1. Make a kryo-shaded with non versioned class names, update chill.
> Users would need to know that upgrading Chill is a breaking change
> 2. Update Chill and Spark to use the new Kryo 5 versions. Perhaps
> with both having support to read v4 in addition to the new v5 classes.
>
>
> Correct.
>
> As far as I understand (the decision to create a versioned Kryo artifact
> was made long before I joined the project), #2 is the preferred option
> for libraries.
#650).
> The only issue with #2 is if the library exposes Kryo classes to
> clients, they will have to update their code (custom serializers etc) to
> use the versioned packages.
Kryo, then the application should have an explicit dependency on Kryo
(I'd tend to recommend the unversioned Kryo for the application - if
shading would be needed for this we'd need an issue/PR :-)). Maybe I'd
need to see a concrete example to better understand the use case to give
a better recommendation.
Cheers,
Martin
>
> https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo5
>
> --
> You received this message because you are subscribed to the
> "kryo-users" group.
> http://groups.google.com/group/kryo-users
> <http://groups.google.com/group/kryo-users>
> ---
> You received this message because you are subscribed to the Google
> Groups "kryo-users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kryo-users+...@googlegroups.com
> <mailto:kryo-users+...@googlegroups.com>.
> --
> You received this message because you are subscribed to the
> "kryo-users" group.
> http://groups.google.com/group/kryo-users
> <http://groups.google.com/group/kryo-users>
> ---
> You received this message because you are subscribed to the Google
> Groups "kryo-users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kryo-users+...@googlegroups.com
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kryo-users/de49d68d-1ea6-4924-be92-35c2326497c2n%40googlegroups.com
> <https://groups.google.com/d/msgid/kryo-users/de49d68d-1ea6-4924-be92-35c2326497c2n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the "kryo-users"
> group.
> http://groups.google.com/group/kryo-users
> <http://groups.google.com/group/kryo-users>
> ---
> You received this message because you are subscribed to the Google
> Groups "kryo-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kryo-users+...@googlegroups.com
> <mailto:kryo-users+...@googlegroups.com>.
> --
> You received this message because you are subscribed to the "kryo-users"
> group.
> http://groups.google.com/group/kryo-users
> <http://groups.google.com/group/kryo-users>
> ---
> You received this message because you are subscribed to the Google
> Groups "kryo-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kryo-users+...@googlegroups.com
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kryo-users/CANjQhG-vFLd%2BcJPDyyvzJDj%2BCPatYRRCby__RjQTxfrEwRXAtw%40mail.gmail.com
> <https://groups.google.com/d/msgid/kryo-users/CANjQhG-vFLd%2BcJPDyyvzJDj%2BCPatYRRCby__RjQTxfrEwRXAtw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
inoio gmbh - http://inoio.de
Kampstraße 15, 20357 Hamburg
Amtsgericht Hamburg, HRB 123031
Geschäftsführer: Dennis Brakhane, Martin Grotzke, Ole Langbehn
Nicholas Nezis
Jan 12, 2021, 6:17:54 PM1/12/21
to kryo-users
The main example I can provide is my Spark and Chill PRs that are driving my question. I was trying to upgrade Spark to use the newer Kryo.
but first I need to update Chill:
I think I'm close to having Chill updated, but I don't know how to add the support for v4 and v5. In the Kryo migration guide, it mentions an if/else, but it's not clear to me what that conditional should be driven by. Is the version information somehow stored in the serialized data? Or would someone need to create a migration process that is configured to read v4 and write out v5? It seems like this type of migration would not make sense in the library, right?
Once I have Chill building local artifacts, I'll try updating Spark to use it with an explicit Kryo dependency. I'm torn on the versioned vs unversioned namespaces. Part of me feel like we should just move to using the versioned namespaces if that helps with understanding binary compatibility. What are the benefits of the application staying with the unversioned while libraries use the versioned namespace?
Ultimately, I'm happy to go either route as long as everyone is in agreement.
Martin Grotzke
Jan 12, 2021, 10:09:37 PM1/12/21
to kryo-...@googlegroups.com
I quickly scanned the PRs, the one for chill LGTM so far.
The spark one misses the actual update, which was already pointed out - at least KryoSerializer would have to be changed. Because users with custom serializers would be affected also https://spark.apache.org/docs/latest/tuning.html should point the binary incompatibility.
I'd definitely expect some spark users to complain about this change, but I still think that this is the right direction and it pays off in the long run to allow easier updates of kryo major versions in the whole ecosystem as I understood from discussions during the last years. And I hope that I'm not wrong 🙂
The change should be communicated accordingly of course, so that people reading the spark release notes at least are not surprised 😉
Regarding v4 and v5 upgrade path in the context of chill/spark I have no idea TBH... (I've also never used spark)
Cheers,
Martin
Message has been deleted
Message has been deleted
Vishv Garg
Jan 13, 2023, 2:17:48 PM1/13/23
to kryo-users
Vulnerabilities have been reported on kryo shaded 4.0.2. And possible solution quotes upgrading to kryo 5.x.
But spark still uses kryo shaded internally and processes breaks if kryo-shaded is replaced.
Any solutions ?
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/pool/KryoPool
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1082)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.__invoke(StandardContextValve.java:97)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:41002)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1722)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/pool/KryoPool
at org.apache.spark.serializer.SerializerManager.<init>(SerializerManager.scala:43)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:278)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1082)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.__invoke(StandardContextValve.java:97)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:41002)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1722)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/pool/KryoPool
at org.apache.spark.serializer.SerializerManager.<init>(SerializerManager.scala:43)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:278)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
Thomas Heigl
Jan 13, 2023, 4:07:09 PM1/13/23
to kryo-...@googlegroups.com
Hi,
What vulnerabilities have been reported? Can you share a link?
I'm not aware of any vulnerability in Kryo 4 that can be resolved by upgrading to 5.
Thomas
You received this message because you are subscribed to a topic in the Google Groups "kryo-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kryo-users/hVlwHmBE_-s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kryo-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kryo-users/0f4e3a0d-d5cb-4c27-b284-b01eab1b7203n%40googlegroups.com.
Nicholas Nezis
Jul 7, 2023, 7:49:33 PM7/7/23
to kryo-users
I was trying to push the Spark upgrade to Kryo 5. I hit a roadblock when trying to upgrade a dependency Twitter/Chill, but there might be some movement on that PR soon (fingers crossed). https://github.com/twitter/chill/pull/747
Also on a related note, I've been following this effort to upgrade Apache Flink from 2.24.0 to 5.5.0. There is a full PR with tests passing, but there is some internal debate about Flink's use of Kryo. Sharing for those interested to follow along and contribute.
Reply all
Reply to author
Forward
0 new messages