KafkaAvroDeserializer Dependencies / POM file

[Avi Flax]

Hi all!

I’m using the `KafkaAvroDeserializer` class that’s included in
the Maven artifact `io.confluent:kafka-avro-serializer` 2.0.0 by
“plugging” it in to the new Consumer
(`org.apache.kafka.clients.consumer.KafkaConsumer`) that’s included
in the Maven artifact `org.apache.kafka:kafka-clients` 0.9.0.0 and
I encountered some trouble, so I thought I’d report what I
experienced just in case other people encounter the same troubles.

When I ran `require` on my Ruby file that
references both classes I got a `java.lang.NoClassDefFoundError`
for `kafka/utils/VerifiableProperties`. I took a look at the Kafka
repo, and it makes sense that `VerifiableProperties` wouldn’t be
included in the `kafka-clients` artifact, since that artifact seems
to correspond to the root-level dir `clients`, while
`VerifiableProperties` is part of the source tree rooted in the
root-level dir `core`.

I’m not particularly proficient with Maven or Gradle, but it seems
to me that if the classes in the artifact `kafka-avro-serializer`
depend on classes in the Kafka “core” artifact, then shouldn’t
`kafka-avro-serializer` explicitly declare that in its dependencies
list in its `pom` file, so that it’d be automatically
included in my project by Maven?

So I took a look at the Maven `pom` file for `kafka-avro-serializer`
(the version in my local Maven repo) and it looks like maybe there’s
a bug in the packaging routine for the project? Because all the
`version` elements in the file seem to contain unrendered string
interpolations, like so:

```
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_${kafka.scala.version}</artifactId>
<version>${kafka.version}</version>
<scope>provided</scope>
</dependency>
```

Like I said, I’m not a Maven expert, but I don’t believe I’ve ever
seen stuff like that in `pom` files before, and that syntax reminds
me of Groovy string interpolation — although it’s been a few years
since I’ve used Groovy.

I was able to work around this by explicitly adding
`org.apache.kafka:kafka_2.11` (0.9.0.0) to my project’s dependencies
(since my project uses JRuby, it uses JBundler to specify Java
dependencies).

So:

* Confluent folks: you might want to look at the packaging for the
schema registry; it seems like it might have a bug.
* Anyone else experiencing similar issues: try specifying the
dependencies of `kafka-avro-serializer` as dependencies of your
project

If I got any of this wrong, which is quite possible, I’d definitely
appreciate being schooled as to what’s actually going on.

Thanks!
Avi

[Mark Davis]

In maven it's common practice to store version numbers as variables or "properties" at the top section of the POM file or perhaps in some parent POM.  Trace thru the pom.xmls and you should see a setting somewhere like:

<properties>
   <kafka.scala.version>2.10</kafka.scala.version>
   <kafka.version>0.9.0</kafka.version>
</properties>

Values might differ but this is the idea, those properties can be referenced in the pom or any child poms via ${property.name} expression.

[Avi Flax]

On Fri, Feb 12, 2016 at 8:53 AM, Mark Davis <mark....@gmail.com> wrote:
> In maven it's common practice to store version numbers as variables or
> "properties" at the top section of the POM file or perhaps in some parent
> POM. Trace thru the pom.xmls and you should see a setting somewhere like:
>
> <properties>
> <kafka.scala.version>2.10</kafka.scala.version>
> <kafka.version>0.9.0</kafka.version>
> </properties>
>
> Values might differ but this is the idea, those properties can be
> referenced in the pom or any child poms via ${property.name} expression.

Ah, thanks Mark! I see it now — `kafka-avro-serializer` defines
`kafka-schema-registry-parent` as its parent, and
`kafka-schema-registry-parent-2.0.0.pom` includes a `properties` element
that defines all those properties.

So I’m back to square one in figuring out why the `kafka_2.11` artifact
wasn’t properly included in my project by JBundler — I wouldn’t be surprised
if it’s a bug in JBundler or in the ruby-maven gem — and at least in the
meantime I have a workaround.

So, never mind then! And thanks!

[Ewen Cheslack-Postava]

The dependency is marked as provided. This is intentional for 2 reasons.

First, one jar provides both old and new (de)serializers. The old ones depend on the core jar because everything used to be bundled into a single jar. The new ones only require kafka-clients. If you're using the new clients (which we encourage), it's helpful not to have to pull in all the core dependencies (including scala!) since they won't be referenced anyway.

Second, since you need to use classes from the clients/core jar anyway, your project can just provide the dependency explicitly. With a scala dependency this is a good idea anyway because if you're using Scala in your own project, you may need to specify the right version. Additionally, since we maintain compatibility between versions of the serializer APIs but the broker/clients may need specific versions, it's good practice to explicitly specify the version of Kafka clients you want to use in your own project instead of implicitly choosing whichever version we happen to build the serializers against.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CABSFgAnDJW1G5BfBpJwgJ-9apDRa7xc%3DBjSFmKa5EaEyQ7Pc8A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Avi Flax]

On Tue, Feb 16, 2016 at 10:26 AM, Ewen Cheslack-Postava
<ew...@confluent.io> wrote:
> The dependency is marked as provided.

Ah, I see. You and Mark are really schooling me on Maven here, I appreciate it!

> This is intentional for 2 reasons.
>
> First, one jar provides both old and new (de)serializers. The old ones
> depend on the core jar because everything used to be bundled into a single
> jar. The new ones only require kafka-clients. If you're using the new
> clients (which we encourage), it's helpful not to have to pull in all the
> core dependencies (including scala!) since they won't be referenced anyway.

Ah, I see! That makes sense.

However, I’m not sure you’ve fully achieved your goal of enabling projects using
the new consumer and producer to omit kafka “core” and Scala, etc from their
dependency tree, because AbstractKafkaAvroDeserializer references
kafka.utils.VerifiableProperties which is in “core”. So it seems like maybe that
dependency might have snuck in by accident and compromised that goal.
(That is, unless I’m missing something, which is frequently the case.)

> Second, since you need to use classes from the clients/core jar anyway, your
> project can just provide the dependency explicitly. With a scala dependency
> this is a good idea anyway because if you're using Scala in your own
> project, you may need to specify the right version. Additionally, since we
> maintain compatibility between versions of the serializer APIs but the
> broker/clients may need specific versions, it's good practice to explicitly
> specify the version of Kafka clients you want to use in your own project
> instead of implicitly choosing whichever version we happen to build the
> serializers against.

Makes sense, I agree 100%.

Thank you!
Avi

[Ewen Cheslack-Postava]

On Wed, Feb 17, 2016 at 11:25 AM, Avi Flax <a...@aviflax.com> wrote:

On Tue, Feb 16, 2016 at 10:26 AM, Ewen Cheslack-Postava
<ew...@confluent.io> wrote:
> The dependency is marked as provided.

Ah, I see. You and Mark are really schooling me on Maven here, I appreciate it!

> This is intentional for 2 reasons.
>
> First, one jar provides both old and new (de)serializers. The old ones
> depend on the core jar because everything used to be bundled into a single
> jar. The new ones only require kafka-clients. If you're using the new
> clients (which we encourage), it's helpful not to have to pull in all the
> core dependencies (including scala!) since they won't be referenced anyway.

Ah, I see! That makes sense.

However, I’m not sure you’ve fully achieved your goal of enabling projects using
the new consumer and producer to omit kafka “core” and Scala, etc from their
dependency tree, because AbstractKafkaAvroDeserializer references
kafka.utils.VerifiableProperties which is in “core”. So it seems like maybe that
dependency might have snuck in by accident and compromised that goal.
(That is, unless I’m missing something, which is frequently the case.)

This is just a Java-specific thing that can be a bit unintuitive -- that reference to VerifiableProperties won't cause any problems unless we actually execute that code. Since it's only invoked via the KafkaAvroDecoder, which already requires the core jar, we're guaranteed it will be available when needed. If you're using the new consumer (and the new Deserializer interface), VerifiableProperties is not used and the core jar can safely be omitted.

-Ewen

 

> Second, since you need to use classes from the clients/core jar anyway, your
> project can just provide the dependency explicitly. With a scala dependency
> this is a good idea anyway because if you're using Scala in your own
> project, you may need to specify the right version. Additionally, since we
> maintain compatibility between versions of the serializer APIs but the
> broker/clients may need specific versions, it's good practice to explicitly
> specify the version of Kafka clients you want to use in your own project
> instead of implicitly choosing whichever version we happen to build the
> serializers against.

Makes sense, I agree 100%.

Thank you!
Avi

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CABSFgAkvTQNrX%2BAs0HVWCsLReGpepbyMfCxLy2sEKsC2mYRu0g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Avi Flax]

On Wed, Feb 17, 2016 at 10:19 PM, Ewen Cheslack-Postava
<ew...@confluent.io> wrote:
> This is just a Java-specific thing that can be a bit unintuitive -- that
> reference to VerifiableProperties won't cause any problems unless we
> actually execute that code. Since it's only invoked via the
> KafkaAvroDecoder, which already requires the core jar, we're guaranteed it
> will be available when needed. If you're using the new consumer (and the new
> Deserializer interface), VerifiableProperties is not used and the core jar
> can safely be omitted.

Hmmm, well, in my current project I’m _trying_ to use the new consumer
and the new
Deserializer interface, but if I omit the core dependency from my
project then I get that
NoClassDefFoundError for VerifiableProperties. So either I’m doing
something wrong,
or maybe it has something to do with how JRuby invokes/loads Java classes maybe?

My JRuby code that creates my consumer is here:

https://gist.github.com/aviflax/43e206015f208129aef4

If you have a minute to take a quick look and let me know if I’m doing something
wrong here, that’d be great!

Thanks,
Avi

[Avi Flax]

On Thursday, February 18, 2016 at 4:32:00 PM UTC-5, Avi Flax wrote:

Hmmm, well, in my current project I’m _trying_ to use the new consumer
and the new Deserializer interface, but if I omit the core dependency from my
project then I get that NoClassDefFoundError for VerifiableProperties. So
either I’m doing something wrong, or maybe it has something to do with how
JRuby invokes/loads Java classes maybe?

To follow up on this: I created a pure Java test project, and it doesn’t exhibit the same problem. So I guess there is indeed something about JRuby, or JBundler, or how I’m using them, that’s causing this issue.

Thanks again Ewen for your help!

Avi