Gobblin writer conversion exception on pulling HDFS avro into Kafka

[Zhu Wayne]

I have a HIVE table data in avro format in HDFS and try to pull it into Kafka. The avro data file is good and I was able to use avro-tool to parse it.

However, I got the following exception:
2016-10-28 16:01:44 PDT INFO  [ForkExecutor-1] gobblin.runtime.Fork  376 - Wrapping writer gobblin.writer.Parti
tionedDataWriter@63db5cfb
2016-10-28 16:01:45 PDT WARN  [ForkExecutor-1] gobblin.writer.RetryWriter$1  83 - Caught exception. This may be
 retried.
org.apache.kafka.common.errors.SerializationException: Can't convert value of class org.apache.avro.generic.Gen
ericData$Record to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serialize
r
2016-10-28 16:02:00 PDT WARN  [ForkExecutor-1] gobblin.writer.RetryWriter$1  83 - Caught exception. This may be
 retried.
org.apache.kafka.common.errors.SerializationException: Can't convert value of class org.apache.avro.generic.Gen
ericData$Record to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serialize
r
2016-10-28 16:02:00 PDT ERROR [ForkExecutor-1] gobblin.runtime.Fork  174 - Fork 0 of task task_AvroToKafka_1477
695688368_0 failed to process data records
java.io.IOException: com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 5
attempts.
        at gobblin.writer.RetryWriter.callWithRetry(RetryWriter.java:128)
        at gobblin.writer.RetryWriter.write(RetryWriter.java:108)
        at gobblin.runtime.Fork.processRecords(Fork.java:404)
        at gobblin.runtime.Fork.run(Fork.java:170)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 5 attempts.
        at com.github.rholder.retry.Retryer.call(Retryer.java:174)
        at com.github.rholder.retry.Retryer$RetryerCallable.call(Retryer.java:318)
        at gobblin.writer.RetryWriter.callWithRetry(RetryWriter.java:126)

[Zhu Wayne]

P.S. I used the following writer.
####################################################
# Writer : Kafka
####################################################

writer.builder.class=gobblin.kafka.writer.KafkaDataWriterBuilder
writer.kafka.topic=chicago_esat
writer.kafka.producerConfig.bootstrap.servers=localhost:9092

[Shirshanka Das]

What did you set the key and value serializer to? 

writer.kafka.producerConfig.value.serializer= writer.kafka.producerConfig.key.serializer=

These should be set to serializers that can work with Avro records.

There are a few examples on the Kafka sink page:

http://gobblin.readthedocs.io/en/latest/sinks/Kafka/#steps

Shirshanka

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-users+unsubscribe@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/d1f7956f-0cce-4ac3-8023-658a0c21dded%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ravi pillala]

Hi Zhu, Are you able to solve your issue. if yes can yo give the same .pull file? I need to solve the same issue.

[Shirshanka Das]

Hi Ravi,

  I think all you have to do is start up the confluent schema registry (assuming you are using confluent's kafka). 

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-users+unsubscribe@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/25a1e9cb-76ed-48f1-abc1-4e89a5012a7e%40googlegroups.com.