Kafka Connect - Performance

[Saravanan Tirugnanum]

Hi ,

How do we measure the performance of Kafka which is publishing data using Connect -  throughput  ( messages per second) and other relavent parameters.

I want to measure and tune all the parameters to achieve desired output. Any recommendations.

Regards

Saravanan

[gerard...@dizzit.com]

For our project we have created a project-specific test jar, which starts a producer and a consumer, and measures the latency, amongst other things. Maybe that's an option for your project as well. It gives you full control on what happens. I used the default kafka performance test class, org.apache.kafka.tools.ProducerPerformance.

[Ewen Cheslack-Postava]

We'd like to add per-connector/task metrics to Kafka Connect itself, but they aren't there quite yet. This gets a bit tricky because in distributed mode you might see connectors/tasks bouncing around workers -- not regularly, but during a rebalance it will happen -- so the metrics will bounce around workers as well. To make it easy to consume we'll have to get them reported somewhere (presumably a Kafka topic) in an easy to parse format.

If you're trying to get some stats out now for performance tuning where you can run controlled experiments with only one connector per worker, all the normal metrics for producer/consumer are still there since we just use the producer/consumer internally. Those are a good starting point to collect performance numbers.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5bb8c71d-9992-40b6-b391-09e877710635%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[VidyaSagar Pallikonda]

Ewen,

Can you provide some links or documentation to know how to get this statistics of kafka connect consumer performance.

Thanks

Sagar