Kafka + Avro serialization - io.confluent.kafka.serializers.KafkaAvroSerializer - maven dependency?

[Neville Sequeira]

Maven repository http://packages.confluent.io/maven/ does not seem to be working.

I have these two blocks in my pom file...

<repositories>
    <repository>
        <id>confluent</id>
        <url>http://packages.confluent.io/maven/</url>
    </repository>
</repositories>

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>1.0</version> <!-- Also tried 2.0-SNAPSHOT  -->
</dependency>

Any work around for this? Any other Serializer implementation I can used when using Avro with Kafka?

Thanks,

-Neville

[Neville Sequeira]

Attaching full pom file - file named pom.xml

Thanks in advance!

[Jason Gustafson]

Hi Neville,

I tried your pom and there doesn't seem to be anything wrong with it. It might help if you send us the output from the failed build. Here are a couple other things to check:

1. Can you access http://packages.confluent.io? 

2. Do you have an .m2/setting.xml installed which could be overriding the pom's repositories?

Thanks,

Jason

[Neville Sequeira]

Good guess. It turns out to be something in my .m2/Settings.xml

Took that out of the picture and now the build is fine.

However, now, on running the my producer application, I get the following exception...

DEBUG 2015-08-05 20:05:09,962 [main] (RestUtils.java:105) - Sending POST with input {"schema":"{\"type\":\"record\",\"name\":\"Event\",\"namespace\":\"events.avro\",\"fields\":[{\"name\":\"type\",\"type\":\"string\"},{\"name\":\"message\",\"type\":[\"string\",\"null\"]}]}"} to http://localhost:8081/subjects/test-value/versions

Exception in thread "main" org.apache.kafka.common.errors.SerializationException: Error serializing Avro message

Caused by: com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input

 at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@51fb3f37; line: 1, column: 1]

        at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:164)

        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3095)

        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3036)

        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2192)

        at io.confluent.kafka.schemaregistry.client.rest.utils.RestUtils.httpRequest(RestUtils.java:144)

        at io.confluent.kafka.schemaregistry.client.rest.utils.RestUtils.registerSchema(RestUtils.java:174)

        at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:51)

        at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:89)

        at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:49)

        at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:67)

        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:326)

        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248)
        .....
        .....

[Jason Gustafson]

The exception looks like it's being thrown from code handling a response error from the schema-registry. Dumb question, but do you have the schema-registry service running?

-Jason

[Neville Sequeira]

Not a dumb question.

I do not have schema-registry service running. I believe that is confluent specific.

All I am trying to do is use Kafka + Avro. Do I need the full shbang from confluent to use Kafka + Avro? Is there no other way to serialize Avro?

[Gwen Shapira]

Hi,

Our Avro serializers are very efficient, in the sense that they don't send the entire schema with the message (the way Avro will do by default), but rather send the message with a URI of the schema. This saves a lot of network overhead, but means that we need somewhere to store the schema - hence the schema repository.

I can go on and on about why the schema repository is awesome and you really should use it.

If you insist on serializing Avro in a way that doesn't require a schema repository, you will need to figure out your own method and implement your own ser/de. 

Avro documentation covers how to serialize Avro into byte arrays and then de-serialize back. You can use this method with Kafka's default Byte serializer, but note that there will be some overhead for the schema in every message.

Gwen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/64542116-0147-492f-9772-cc8edf2248ef%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Neville Sequeira]

Thanks for the response Gwen!

Our development and deployment are both on Windows. Currently, our research and feasibility study for Kafka is being hampered by lack of official support for Confluent on Windows.

I have managed to run Schema Registry. But any help or pointers into getting Windows versions of the scripts that come with Confluent will be highly appreciated.

Thanks,

-Neville

To post to this group, send email to confluent...@googlegroups.com.

[Gwen Shapira]

If "official support" means you want to pay us, then by all means discuss Windows with your friendly Confluent salesperson :)

If you mean community support, then I don't think this is an issue. Feel free to either open an issue on our Github repository, or even a pull request if you want to contribute scripts that you developed.

Gwen

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/be1820e7-93c7-4335-90e0-df3533ba7177%40googlegroups.com.

[Neville Sequeira]

Gwen,

Out of the 3 options you suggested, I chose to open an issue - https://github.com/confluentinc/schema-registry/issues/211

Thanks,

-Neville

To post to this group, send email to confluent...@googlegroups.com.

[David Chen]

I had the same error as you.  I think it was a very simple error in their documentation.  Change from http to https

Just change http://packages.confluent.io/maven/ to https://packages.confluent.io/maven/

<repositories>
    <repository>
        <id>confluent</id>

        <url>https://packages.confluent.io/maven/</url>
    </repository>
</repositories>

On Wednesday, August 5, 2015 at 6:40:39 PM UTC-4, Neville Sequeira wrote:

[廖师虎]

I can't download the avro too, why?

Best Regards

Shifu