Kafka Connect Transformation - Struct to Flat Data

[Senthil Nayagan]

I am working on a proof-of-concepts on Kafka Connect  (using Standalone for now) where my use case is to extract data from Kafka and write them into a file as flat data with pipe delimited .

I am using File-sink connector for the same. Since my data in the Kafka topic is encoded in Avro (with schema), I am using avro converter to deserialise the avro data and writing them into a flat file.

As Converter returns the whole data as Struct object with keys-values in it, I couldn't able to store them as flat data, for instance, 1010|John|30 (delimited with pipe or something).

For example, data comes as Struct{empid=1010, name=John, age=30}, but I want them to be as 1010|John|30 using Transformation. 

I tried converting the Struct data into flat data using Connect's  Transformations, but didn't work in the way I wanted. Not sure it's possible to do that.

Here is what my transformation goes into my standalone worker properties:

transforms=FlattenData transforms.FlattenData.type=org.apache.kafka.connect.transforms.Flatten$Value transforms.FlattenData.delimiter=| Any suggestion/help to get the Struct data as flat ones using Transformation would be highly appreciated.

[saravanan shanmugam]

Hi Senthil,

Hope you are doing good. I also have the similar requirement, I am trying to a do a POC and happen to find your post.where you able to find solution. 

Thanks,

Saravanan