kafka connect AVRO using file sink

[adam.klein...@adp.com]

Hi,

I'm using org.apache.kafka.connect.file.FileStreamSinkConnector

consuming AVRO encoded messages and outputting them to a file.

I would expect the messages to be outputted as JSON format, but what I see in the file is:

"

org.apache.kafka.connect.data.Struct@ca1bf85a

org.apache.kafka.connect.data.Struct@c298db6a

org.apache.kafka.connect.data.Struct@44108fbd

"

So basically the name of the class with probably the pointer.

Can anyone please help?

BTW, When I use value.converter=org.apache.kafka.connect.storage.StringConverter and I send strings, then I can see them in the file.

The configuration:

bootstrap.servers=kafka:9092

key.converter=org.apache.kafka.connect.storage.StringConverter

value.converter=io.confluent.connect.avro.AvroConverter

value.converter.schema.registry.url=http://avro-schema-registry:8081

internal.key.converter=org.apache.kafka.connect.json.JsonConverter

internal.value.converter=org.apache.kafka.connect.json.JsonConverter

internal.key.converter.schemas.enable=false

internal.value.converter.schemas.enable=false

offset.storage.file.filename=/tmp/connect.offsets.auth

The sink:

name=my-file-sink

connector.class=org.apache.kafka.connect.file.FileStreamSinkConnector

tasks.max=1

file=auth.sink.txt

topics=test

Thanks

[gerard...@dizzit.com]

If you look at the io.confluent.connect.avro.AvroConverterTest, you see there is a Struct Object 'expected', and 'original' in the value part of the SchemaAndValue:

byte[] converted = converter.fromConnectData(TOPIC, original.schema(), original);
SchemaAndValue schemaAndValue = converter.toConnectData(TOPIC, converted);
assertEquals(expected, schemaAndValue.value());

[michael...@gmail.com]

I'm hitting this same issue.. did you ever get a resolution?

[Shikhar Bhushan]

Hi,

The File[Source|Sink]Connector's are toy examples that only work on String's.

I have opened a ticket to improve this experience with the FileSinkConnector, https://issues.apache.org/jira/browse/KAFKA-4068

Best,

Shikhar

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/8a1a3f18-50dc-4511-9b9d-762e925ec35d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.