Confluent-Kafka-Python: What serializer to use for producing ProtoBuf messages?

[Ananya]

Few questions:

1) What happens if I don't specify any serialization at all? This is a python script so does that read it as string and send as a byte array?

2) What's the importance of using serializer and how can I implement one for ProtoBuf records?

3) Does the serializer need to be implemented in Java if I'm using Confluent-Kafka-Python client?

Thank you!

[Ewen Cheslack-Postava]

Because of the nature of python, it doesn't need to have pluggable serializers in the same way as many other clients do -- it's trivial to add the serialization step by overriding send() on the producer, e.g. see how the Avro support is implemented:

https://github.com/confluentinc/confluent-kafka-python/blob/master/confluent_kafka/avro/__init__.py#L62

The expectation if you use the raw producer/consumer is that you handle serialization yourself. So it is expected that the data is already serialized as a byte array (the exact type depends on if you're in python 2 or python 3).

For protobuf records, you'd handle serialization by using the protocol buffers python library and probably just call the SerializeToString() method on the objects passed to the producer -- serializers are always implemented by invoking the relevant method for the serialization format that convert the data to a byte array. (I am just basing this on looking up how to use the python protobuf library: https://developers.google.com/protocol-buffers/docs/pythontutorial)

Serializers are language specific, so you need an implementation in python to work with confluent-kafka-python.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f4a4c337-f030-49c1-8dff-2cf0eb732461%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ananya]

Thank you, Ewen! This is really helpful.

On Friday, March 3, 2017 at 11:35:38 AM UTC-8, Ewen Cheslack-Postava wrote:

Because of the nature of python, it doesn't need to have pluggable serializers in the same way as many other clients do -- it's trivial to add the serialization step by overriding send() on the producer, e.g. see how the Avro support is implemented:

https://github.com/confluentinc/confluent-kafka-python/blob/master/confluent_kafka/avro/__init__.py#L62

The expectation if you use the raw producer/consumer is that you handle serialization yourself. So it is expected that the data is already serialized as a byte array (the exact type depends on if you're in python 2 or python 3).

For protobuf records, you'd handle serialization by using the protocol buffers python library and probably just call the SerializeToString() method on the objects passed to the producer -- serializers are always implemented by invoking the relevant method for the serialization format that convert the data to a byte array. (I am just basing this on looking up how to use the python protobuf library: https://developers.google.com/protocol-buffers/docs/pythontutorial)

Serializers are language specific, so you need an implementation in python to work with confluent-kafka-python.

-Ewen

On Thu, Mar 2, 2017 at 1:13 PM, Ananya <sneha.j...@gmail.com> wrote:

Few questions:

1) What happens if I don't specify any serialization at all? This is a python script so does that read it as string and send as a byte array?

2) What's the importance of using serializer and how can I implement one for ProtoBuf records?

3) Does the serializer need to be implemented in Java if I'm using Confluent-Kafka-Python client?

Thank you!

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.