Kafka Connect 0.10.1.0-rc1: High CPU usage

Christian Karlsen

unread,

Oct 10, 2016, 10:31:39 PM10/10/16

to Confluent Platform

I noticed much higher CPU usage after upgrading from 0.10.0.1.

It seems like org.apache.kafka.connect.runtime.distributed.WorkerGroupMember.ensureActive() is constantly using one CPU core.

Happens even if there are no running connectors.

Attached is a screenshot from a simple 1m profiling recording with Java Mission Control, running kafka connect distributed from command line, with default configuration and no connectors. 1 kafka broker with default configs on local machine.

Dustin Cote

unread,

Oct 11, 2016, 8:29:51 AM10/11/16

to confluent...@googlegroups.com

Hi Christian,

This looks a bit like https://issues.apache.org/jira/browse/KAFKA-4247

Can you make sure you aren't running the worker from the root directory or a directory with a large number of files? You'll also want to make sure the CLASSPATH variable is not empty.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/52a55589-12bd-4f16-97a9-6673ea835efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Dustin Cote

Customer Operations Engineer | Confluent

Follow us: Twitter | blog

Christian Karlsen

unread,

Oct 11, 2016, 9:57:55 AM10/11/16

to Confluent Platform

Hi,

https://issues.apache.org/jira/browse/KAFKA-4290 was created 30 minutes after this post :)

I've applied the patch and everything looks good!

Best regards,

Christian Karlsen

On Tuesday, October 11, 2016 at 2:29:51 PM UTC+2, Dustin Cote wrote:

Hi Christian,

This looks a bit like https://issues.apache.org/jira/browse/KAFKA-4247

Can you make sure you aren't running the worker from the root directory or a directory with a large number of files? You'll also want to make sure the CLASSPATH variable is not empty.

On Mon, Oct 10, 2016 at 10:31 PM, Christian Karlsen <ckar...@gmail.com> wrote:

I noticed much higher CPU usage after upgrading from 0.10.0.1.

It seems like org.apache.kafka.connect.runtime.distributed.WorkerGroupMember.ensureActive() is constantly using one CPU core.

Happens even if there are no running connectors.

Attached is a screenshot from a simple 1m profiling recording with Java Mission Control, running kafka connect distributed from command line, with default configuration and no connectors. 1 kafka broker with default configs on local machine.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/52a55589-12bd-4f16-97a9-6673ea835efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Shikhar Bhushan

unread,

Oct 11, 2016, 2:03:16 PM10/11/16

to Confluent Platform

Indeed, KAFKA-4290 was raised when investigating your report :-) Thanks for testing out the RC!

On Tue, Oct 11, 2016 at 6:57 AM Christian Karlsen <ckar...@gmail.com> wrote:

Hi,

https://issues.apache.org/jira/browse/KAFKA-4290 was created 30 minutes after this post :)

I've applied the patch and everything looks good!

Best regards,
Christian Karlsen

On Tuesday, October 11, 2016 at 2:29:51 PM UTC+2, Dustin Cote wrote:
Hi Christian,

This looks a bit like https://issues.apache.org/jira/browse/KAFKA-4247

Can you make sure you aren't running the worker from the root directory or a directory with a large number of files? You'll also want to make sure the CLASSPATH variable is not empty.
On Mon, Oct 10, 2016 at 10:31 PM, Christian Karlsen <ckar...@gmail.com> wrote:
I noticed much higher CPU usage after upgrading from 0.10.0.1.

It seems like org.apache.kafka.connect.runtime.distributed.WorkerGroupMember.ensureActive() is constantly using one CPU core.

Happens even if there are no running connectors.

Attached is a screenshot from a simple 1m profiling recording with Java Mission Control, running kafka connect distributed from command line, with default configuration and no connectors. 1 kafka broker with default configs on local machine.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/52a55589-12bd-4f16-97a9-6673ea835efd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Dustin Cote
Customer Operations Engineer | Confluent
Follow us: Twitter | blog

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/269ed5f6-01c6-4f90-9d0f-4a8dc5726aca%40googlegroups.com.

Reply all

Reply to author

Forward