Onyx-kafka serializer function

[aman shah]

Hi,

Is  :kafka/serializer-fn always necessary when we are writing the segments on Kafka topic?
What does it actually do?

[Mike Drogalis]

Hi,

Yes, this configuration option is always required. It corresponds to the Kafka Producer value.serializer option, which takes an object and transforms it into a byte array. Ideally this would be called kafka/value-serializer-fn, but for historical reasons (I didn't understand how keys worked in Kafka when I wrote the plugin years ago :) it remains for compatibility.

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+unsubscribe@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/36918c1e-5326-43ff-91ec-2670d0b6837f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[aman shah]

Thanks Mike. It helps. Just one more thing, do we need to write our own serializer function?
In the Onxy-kafka documentation you have mentioned something like :kafka/serializer-fn :my-ns/serializer-fn . What does it actually mean? Do we need to give the namespace where this serializer function is. If yes in which namespace where we can find it? Or do we need to give an object to it so it can transform it to byte-array.

[Mike Drogalis]

Per the docs:

A keyword that represents a fully qualified namespaced function to serialize a record's key. Takes one argument - the segment

You'd generally write this yourself, but there's a few helpers here: https://github.com/onyx-platform/onyx-kafka/blob/0.12.x/src/onyx/tasks/kafka.clj#L64-L74

--

You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+unsubscribe@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/71bf3164-b9e4-4f04-b999-fc0626c6daac%40googlegroups.com.