Kryo and ASM 4.0

[Dirk]

Hi everybody,

we love kryo for its speed and would like to use it in our project.
But the dependency to asm 4.0 is a problem for us, since other
librarys (e.g. cglib) need asm 3.x.

I tried the older 1.x kryo, but this implementation can't serialize
our instances (maybe due to generics?).

There is no possibilty to get Kryo 2.x to work with asm 3.x, is it?

Thanks in advance for any feedback,
Dirk

[Nate]

The ASM dependency is for ReflectASM. You could revert ReflectASM to when it used ASM 3.x, then port the recent changes that added construction support. Or you could just not use ReflectASM.

-Nate

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users

[Martin Grotzke]

Hi,

you can also exclude org.ow2.asm:asm from kryo if you're using maven and
use your preferred version of asm - as long as it's working with reflectasm.

Cheers,
Martin

--
inoio gmbh
Breitenfelder Str. 13c, 20251 Hamburg
Amtsgericht Hamburg, HRB 123031
Geschäftsführer: Dennis Brakhane, Martin Grotzke, Ole Langbehn

[Nate]

I checked and there is some issue with ReflectASM and ASM 3.x. Didn't dig into what exactly the problem is, but it isn't just a drop in replacement.

-Nate

[Niall Gallagher]

Hi Dirk,

You could try building a shaded jar for Kryo. That would be Kryo and all of the classes from Kryo's own dependencies merged into Kryo's main jar, but Kryo's dependency classes would be relocated to different packages in the Kryo jar. That way Kryo could use ASM 4.0 (in the renamed packages in its own jar) and your application could use ASM 3.0 (in the regular asm jar) at the same time.

Take a look at the maven-shade-plugin: http://maven.apache.org/plugins/maven-shade-plugin/

Example snippet in a pom.xml for something similar here (line 156): http://code.google.com/p/mobility-rpc/source/browse/mobility-rpc/trunk/pom.xml?spec=svn145&r=138#156

You need to check out Kryo from source and add a snippet like that to Kryo's own pom.xml, then run mvn package.

Niall

[Dirk]

Hi everybody!

Thanks for your quick advices!! I can't downgrade to reflectasm/asm
3.0 due to this bug:
http://code.google.com/p/kryo/issues/detail?id=55

Further more, i already tried the maven dependency management, but asm
4.0 is not compatible
with 3.x :-/.

But the idea with the shaded plugin sound great. I will try that!

@Nate: What do you mean with "don't use reflectasm"? Is there an
option to disable this component?

cu,
Dirk

On Jun 5, 3:40 am, Niall Gallagher <ni...@npgall.com> wrote:
> Hi Dirk,
>
> You could try building a shaded jar for Kryo. That would be Kryo and all of
> the classes from Kryo's own dependencies merged into Kryo's main jar, but
> Kryo's dependency classes would be relocated to different packages in the
> Kryo jar. That way Kryo could use ASM 4.0 (in the renamed packages in its
> own jar) and your application could use ASM 3.0 (in the regular asm jar) at
> the same time.
>
> Take a look at the
> maven-shade-plugin:http://maven.apache.org/plugins/maven-shade-plugin/
>
> Example snippet in a pom.xml for something similar here (line

> 156):http://code.google.com/p/mobility-rpc/source/browse/mobility-rpc/trun...

[Dirk]

Hi Niall,

that was a great hint. maven-shade-plugin rocks. Works like a sharm!!

Thanks,
Dirk

On 5 Jun., 03:40, Niall Gallagher <ni...@npgall.com> wrote:
> Hi Dirk,
>
> You could try building a shaded jar for Kryo. That would be Kryo and all of
> the classes from Kryo's own dependencies merged into Kryo's main jar, but
> Kryo's dependency classes would be relocated to different packages in the
> Kryo jar. That way Kryo could use ASM 4.0 (in the renamed packages in its
> own jar) and your application could use ASM 3.0 (in the regular asm jar) at
> the same time.
>
> Take a look at the
> maven-shade-plugin:http://maven.apache.org/plugins/maven-shade-plugin/
>
> Example snippet in a pom.xml for something similar here (line

> 156):http://code.google.com/p/mobility-rpc/source/browse/mobility-rpc/trun...

[Martin Grotzke]

Perhaps we should publish this uber jar to maven central by default as a separate artifact (with different classifier)?

http://maven.apache.org/plugins/maven-shade-plugin/examples/attached-artifact.html

Cheers,
Martin

[Niall Gallagher]

The config I liked to above, does attach the shaded artifact actually with classifier "all", so if left in Kryo's pom it would publish the regular jar AND the shaded/uber jar to maven central.

If shadedArtifactAttached is set to true, it is supposed to do this (deploy both jars). However- if it's set to false, it seems to clobber the regular jar in the target/ directory and replace it with the shaded one, which I didn't want either. In my case I want to deploy the regular jar to central, and make the uber jar available from the Downloads tab on my site.

I think uber jars in maven central have limited value if the pom they're associated with still declares the transitive dependencies (as is the case in my project). If we add a dependency to an application, regardless of the classifier, I think Maven will still pull in the transitive dependencies. So deploying the shaded jar to maven central just with a different classifier doesn't seem to be that useful, the only way to use it without pulling in the (unnecessary) transitive deps would be to exclude them manually where the dep is declared.

So I'd say some solutions are:

(1) deploy the regular jar without the uber jar to central, along with the regular pom (i.e. the current setup), or..

(2) deploy the uber jar without the regular jar, with a modified pom which omits the transitive dependencies

I just did a bit more digging, take a look at - http://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html - it looks like the plugin does support these use cases after all, the options are just a bit buried in the detail on that page.

For option 1: To have it generate the uber jar without clobbering the regular jar AND to *not* deploy the uber jar to maven central, you could try setting shadedArtifactAttached to false, and set finalName in the plugin's config to something else.

For option 2: To *only* deploy the uber jar to central, the pom that gets deployed to central should omit Kryo's transitive dependencies. It looks like the "createDependencyReducedPom" option could be useful for this.

Let me know which config you end up going with, as I want to fix this same issue in my own project too!

Niall

http://code.google.com/p/mobility-rpc/

[Martin Grotzke]

Hi,

I think the easiest would be to push only the uber jar to maven central
(option 2). The uber jar should at least have the asm dependency
relocated, reflectasm and minlog may be relocated but don't have to.

Can you provide the pom snippet of your preference so that Nate can
update the pom.xml - as long as there are not other opinions?

Cheers,
Martin

> <mailto:h3rm...@googlemail.com>>:

>
> Hi Niall,
>
> that was a great hint. maven-shade-plugin rocks. Works like a
> sharm!!
>
> Thanks,
> Dirk
>
> On 5 Jun., 03:40, Niall Gallagher <ni...@npgall.com> wrote:
> > Hi Dirk,
> >
> > You could try building a shaded jar for Kryo. That would be
> Kryo and all of
> > the classes from Kryo's own dependencies merged into Kryo's
> main jar, but
> > Kryo's dependency classes would be relocated to different
> packages in the
> > Kryo jar. That way Kryo could use ASM 4.0 (in the renamed
> packages in its
> > own jar) and your application could use ASM 3.0 (in the
> regular asm jar) at
> > the same time.
> >
> > Take a look at the
> >
> maven-shade-plugin:http://maven.apache.org/plugins/maven-shade-plugin/
> <http://maven.apache.org/plugins/maven-shade-plugin/>
> >
> > Example snippet in a pom.xml for something similar here (line
> >
> 156):http://code.google.com/p/mobility-rpc/source/browse/mobility-rpc/trun.

> <http://code.google.com/p/mobility-rpc/source/browse/mobility-rpc/trun.>..

> >
> > You need to check out Kryo from source and add a snippet like
> that to
> > Kryo's own pom.xml, then run mvn package.
> >
> > Niall
> >
> >
> >
> >
> >
> >
> >
> > On Monday, June 4, 2012 2:27:55 PM UTC+1, Dirk wrote:
> >
> > > Hi everybody,
> >
> > > we love kryo for its speed and would like to use it in our
> project.
> > > But the dependency to asm 4.0 is a problem for us, since other
> > > librarys (e.g. cglib) need asm 3.x.
> >
> > > I tried the older 1.x kryo, but this implementation can't
> serialize
> > > our instances (maybe due to generics?).
> >
> > > There is no possibilty to get Kryo 2.x to work with asm 3.x,
> is it?
> >
> > > Thanks in advance for any feedback,
> > > Dirk
>
> --
> You received this message because you are subscribed to the
> "kryo-users" group.
> http://groups.google.com/group/kryo-users

> <http://groups.google.com/group/kryo-users>

>
> --
> You received this message because you are subscribed to the "kryo-users"
> group.
> http://groups.google.com/group/kryo-users

--
inoio gmbh - http://inoio.de

[Niall Gallagher]

Hi Martin/Dirk/Nate,

Pushing only the uber jar sounds good to me.

It takes some trial and error to configure the maven-shade-plugin to do all of the relocations, as the roots of the packages to be relocated need to be listed in the config. I think we might as well put all of Kryo's deps in the uber jar, including minlog (i.e. all or nothing)? But it's Nates call ultimately.

Dirk- Would you mind sharing the configuration for the maven-shade-plugin that you ended up with to build your shaded jar for Kryo? We can use this as a basis for our config.

Thanks,

Niall

>     <mailto:h3rmanns@googlemail.com>>:

[badger]

I know I'm joining the party a bit late here, and I may be
misunderstanding, but...

Can I add my vote _against_ publishing only the uber jar? I'm
generally against uber jars unless you really have to, because it
makes it harder to see what libraries you're using, and it increases
the size of your distribution. If you can only publish one jar, I'd
prefer the standard jar, those few people in the world who come up
against this problem could shade their own jar if really necessary.

Firstly, our application has a high number of libraries, if every
library included all its libs shaded into its own jar then the size of
our distribution would explode. Secondly, I suspect Kryo is used by a
number of commercial products. Products come with warranties. Being
able to warrant a piece of software means knowing what code is in it
(shading makes it harder to know). Equally, if a bug is found in ASM3
in a customer's specific environment, then steps can be taken to
ensure all versions of asm in the product are upgraded (sure, in some
cases this means forking libraries that depend on ASM, but that's
life). If people start bundling ASM into their own jars, this becomes
more difficult to spot, and even more difficult to fix.

These are just my thoughts, I'm not trying to tell anyone what to do.

Btw I'd prefer if both jars are available if it helps people - I've
been trying to work out how to publish both jars with the correct
dependencies using some magic in profile activation based on
classifier, but my knowledge of maven hits a brick wall fairly
quickly.

Presumably it would be possible to publish the shaded jar as a
_separate_ artifact, which can depend on the main kryo artifact, do
the shading, and exclude asm - not quite as neat but may work?

-badger

> >http://maven.apache.org/plugins/maven-shade-plugin/examples/attached-...
> > >     <
> >http://maven.apache.org/plugins/maven-shade-plugin/examples/attached-...>
>
> > >     Cheers,
> > >     Martin
>
> > >     Am 06.06.2012 14:55 schrieb "Dirk" <h3rma...@googlemail.com
> > >     <mailto:h3rma...@googlemail.com>>:

> > inoio gmbh -http://inoio.de

[Niall Gallagher]

Hi Badger,

I think that is a reasonable argument. It was just a case of the right way, versus the easy way. My reason for supporting the idea was just that it would reduce the amount of effort for Martin to deploy to central, plus that ASM 3.0 vs 4.0 conflict issue, but that was just in this one instance. I'm easy either way.

Sooo.... assuming we now want the regular jar in central(?), options:

(1) deploy both regular and shaded jar to central. By default maven will take the regular jar unless you specify a classifier. If you want to use the shaded jar, specify classifier and then manually exclude Kryo's dependencies where you declare the dependency. Something I thought was a bit hacky above, but hey it would work.

(2) deploy just the regular jar to central. Nate could make the shaded jar available for download on the Downloads page(?).

(3) deploy just the regular jar to central. No further changes :) People who want the shaded jar would have to build it themselves.

Whatever the case it would be useful to get the config for building a shaded jar into Kryo's pom. I could help with this.

If Dirk doesn't respond to my question earlier, if someone could instead provide a list of the root package names (com.foo.project1, org.bar.project2 etc) used by each of Kryo's dependencies I could cobble together the config.

Niall

[Martin Grotzke]

(1) +1

Cheers,
Martin