Binary file transfer using kafka connect - message size changes

[sunil yadav]

Hi all,

I am transferring binary file using Kafka connect by breaking them into chunk 1 KB bytes. The source record value schema is Bytes and content is byte[]. when I consume message in sink it does not automatically type case to byte[](complains it is a string) and the size is also more than 1kb, which results in corrupt data.

I have used Json converter , with schema false.  I really appreciate any help or pointers.

Thanks

[Ewen Cheslack-Postava]

JSON doesn't have any built-in type that represents byte[]. On the source side, the converter is base64 encoding the data and storing it as a string, but then on the sink side there is no information about the schema which can be used to determine that we should reverse the base64 encoding and turn it back into a byte[].

This is just a fundamental limitation of JSON -- it loses schema information, so for schema types it does not natively support, there will be some mismatch between source and sink if the schema is not somehow carried over to the sink.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a7d9f056-5f7c-48ea-95c2-604ddf970a1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.