kafka-hadoop

[Daniel]

hi,

we are using camus for long time,

and we have issue with duplicated messages.

our flow is publish events to kafka - >  consume & write to hdfs with camus

after the whole process finishes, we have noticed that there are events that written to hadoop more than once.

anyone had the same issue or have any idea what could be the cause ? 

thanks in advance.

[Félix GV]

You need to use the camus-sweeper module to dedupe.

You should also note that Camus is not actively maintained anymore.

Confluent is now supporting Kafka Connect.

LinkedIn is now supporting the open-source Gobblin project, which had a migration guide from Camus to Gobblin here: https://github.com/linkedin/gobblin/blob/master/gobblin-docs/miscellaneous/Camus-to-Gobblin-Migration.md

-F

--
You received this message because you are subscribed to the Google Groups "Camus - Kafka ETL for Hadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camus_etl+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel]

thanks!