--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b987789-9121-4b68-95a8-3f33234d7852%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAFLW6WTZoFFQorYfHCQya0MOkYaR48HKG_sLJkMB074xvwQEiQ%40mail.gmail.com.
I would suggest handling this at the Kafka level -- if you want data produced to Kafka that is snappy compressed, just configure the producers and consumers in Connect to use snappy compression. You can override any client settings as described here: http://docs.confluent.io/2.0.1/connect/userguide.html#overriding-producer-consumer-settings-Ewen
On Tue, May 17, 2016 at 9:50 AM, Liquan Pei <liq...@confluent.io> wrote:
Hi Shrk,Kafka connect don't provide compression related code. Although Kafka Connect depends on the clinets jar which includes Compressor, but I don'd encourage you to use that may not meet the specific needs of your connector.Thanks,Liquan
On Tue, May 17, 2016 at 2:30 AM, shrk <nkravi...@gmail.com> wrote:
Hi ,Is there any direct method for setting compression codec preferably snappy compression in Kafka connect - Sink Connector .Or what is the best way to set the compression methodology in kafka connect to store the kafka messages to the local file system disk which are snappy compressed.Do we have direct method to set the compression or should we use snappy library externally to compress the data and push to the file ( snappy format )Regards,shrk
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b987789-9121-4b68-95a8-3f33234d7852%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--Liquan Pei | Software Engineer | Confluent | +1 413.230.6855Download Apache Kafka and Confluent Platform: www.confluent.io/download
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAFLW6WTZoFFQorYfHCQya0MOkYaR48HKG_sLJkMB074xvwQEiQ%40mail.gmail.com.
--Thanks,
Ewen
Hi Ewen,I tried setting compression codec at the producer level , but even after overwriting the properties in Kafka Connect worker properties the records fetched from the SinkTask put method is not the snappy compressed data . Can I know how can I fetch the snappy data ?Thanks,Shrk
On Thursday, 19 May 2016 09:45:46 UTC+5:30, Ewen Cheslack-Postava wrote:
I would suggest handling this at the Kafka level -- if you want data produced to Kafka that is snappy compressed, just configure the producers and consumers in Connect to use snappy compression. You can override any client settings as described here: http://docs.confluent.io/2.0.1/connect/userguide.html#overriding-producer-consumer-settings-Ewen
On Tue, May 17, 2016 at 9:50 AM, Liquan Pei <liq...@confluent.io> wrote:
Hi Shrk,Kafka connect don't provide compression related code. Although Kafka Connect depends on the clinets jar which includes Compressor, but I don'd encourage you to use that may not meet the specific needs of your connector.Thanks,Liquan
On Tue, May 17, 2016 at 2:30 AM, shrk <nkravi...@gmail.com> wrote:
Hi ,Is there any direct method for setting compression codec preferably snappy compression in Kafka connect - Sink Connector .Or what is the best way to set the compression methodology in kafka connect to store the kafka messages to the local file system disk which are snappy compressed.Do we have direct method to set the compression or should we use snappy library externally to compress the data and push to the file ( snappy format )Regards,shrk
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b987789-9121-4b68-95a8-3f33234d7852%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--Liquan Pei | Software Engineer | Confluent | +1 413.230.6855Download Apache Kafka and Confluent Platform: www.confluent.io/download
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAFLW6WTZoFFQorYfHCQya0MOkYaR48HKG_sLJkMB074xvwQEiQ%40mail.gmail.com.
--
--Thanks,
Ewen
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/eb30ef20-c133-49d6-b02c-a297ec5007cb%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b987789-9121-4b68-95a8-3f33234d7852%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--Liquan Pei | Software Engineer | Confluent | +1 413.230.6855Download Apache Kafka and Confluent Platform: www.confluent.io/download
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAFLW6WTZoFFQorYfHCQya0MOkYaR48HKG_sLJkMB074xvwQEiQ%40mail.gmail.com.
--Thanks,
Ewen
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/eb30ef20-c133-49d6-b02c-a297ec5007cb%40googlegroups.com.
--Thanks,
Ewen
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b987789-9121-4b68-95a8-3f33234d7852%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--Liquan Pei | Software Engineer | Confluent | +1 413.230.6855Download Apache Kafka and Confluent Platform: www.confluent.io/download
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAFLW6WTZoFFQorYfHCQya0MOkYaR48HKG_sLJkMB074xvwQEiQ%40mail.gmail.com.
--Thanks,
Ewen
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/eb30ef20-c133-49d6-b02c-a297ec5007cb%40googlegroups.com.
--
--Thanks,
Ewen
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/99f6aafc-c82f-48e3-a189-aecb1ebf530e%40googlegroups.com.