Kafka Consumer origin stops consuming when a large message is encountered

[TJ Brown]

Kafka consumers have a property (max.partition.fetch.bytes) to define the largest size of a message that it can consume.  When a Kafka Consumer origin encounters a message that exceeds the size specified in the max.partition.fetch.bytes property, the consumer stops consuming messages from the Kafka topic.  The Kafka Consumer origin, however, does not give any indication that it has encountered a message that it cannot consume.  It appears as if it has consumed all of the messages in the topic (even sending empty batches through the pipeline) when really there are still messages to be consumed.  My expectation would be that the origin would indicate in some way that it has encountered a message that it cannot consume and either stop the pipeline and give an error message or create an error record and continue consuming subsequent messages.  What is the intended behavior in this situation? 

Thanks,

T.J. Brown

[Jeff Evans]

Hi TJ,

My understanding is that is the intended behavior of Kafka.  A consumer stops consuming messages if the next one is too large for it to handle (by way of this, or any of the similar configuration parameters).

https://kafka.apache.org/documentation/#newconsumerconfigs

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why?

--
You received this message because you are subscribed to the Google Groups "sdc-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdc-user+unsubscribe@streamsets.com.
Visit this group at https://groups.google.com/a/streamsets.com/group/sdc-user/.

--

Jeff Evans

je...@streamsets.com

[TJ Brown]

Thanks for the explanation.   

To unsubscribe from this group and stop receiving emails from it, send an email to sdc-user+u...@streamsets.com.

Visit this group at https://groups.google.com/a/streamsets.com/group/sdc-user/.

--

Jeff Evans

je...@streamsets.com