Re: Unknown Magic Byte with Kafka Streams and Connect

[Michael Noll]

Hi Artyom,

Sorry to hear that you have been stuck with this error!

Quick question to confirm:

> We even upgraded to 0.10 and 3.0 Confluent to test if that was the problem, still the same issue.

You followed the instructions at [1] to build Kafka 0.10.0.0(-SNAPSHOT) and the corresponding Confluent 3.0.0-SNAPSHOT services, including confluentinc/schema-registry etc.?

Also, if that's possible, would you mind sharing your code example that triggers this problem?

Best,

Michael

[1] https://github.com/confluentinc/examples/tree/master/kafka-streams#apache-kafka-and-kafka-streams

On Wed, May 11, 2016 at 12:18 PM, <artyom....@gmail.com> wrote:

Hi,

So we are having an issue, that is slightly driving us insane.

We are trying to consume message produced by Kafka Streams with a simple Kafka Connect console consumer. We are serializing the key with StringSerialzier and the values with GenericAvroSerialzer from the Confluent examples. The key is a simple string and value is GenericRecord with one subrecord/schema.

We are getting:

[2016-05-11 12:06:58,419] ERROR Task local-file-sink-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:142)
org.apache.kafka.connect.errors.DataException: Failed to deserialize data to Avro: 
    at io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:345)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:226)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

The funny thing is if we by hand deserialize and call toConnectData, everything executes without exceptions. Same if we just use AvroConsoleConsumer.

We even upgraded to 0.10 and 3.0 Confluent to test if that was the problem, still the same issue.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6aed9db3-2ef2-4df2-8f9a-61fd69c7fa90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dominik Benz]

Hi,

just in case somebody else stumbles upon this - I faced the same issue, and the solution was not to use StringSerializer for the key, but rather the same Serializer as for the value (GenericAvroSerializer from the Confluent examples).

Best,
  Dominik

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

[Gabriel Queiroz]

Hi Dominik,

I am facing the same problem as talked in this topic... Getting "Error deserializing Avro message for id -1  Unknown magic byte!".

Which key exactly you were talking about? Is the "KEY_SERDE_CLASS_CONFIG", "KEY_SERIALIZER_CLASS_CONFIG" or another key? I tried to change both to the same Serializer that I am using, but it didn't work...

I've created a topic today with my code examples if you want to check it. I would be really grateful if you could help me with this problem.

Thank you very much for your attention.

Gabriel Queiroz.