Kafka connect in distributed mode worker re-balance issue

4,148 views
Skip to first unread message

Adarsh Lal

unread,
Nov 16, 2016, 11:51:12 PM11/16/16
to Confluent Platform
Hi,

I am running Kafka 10.1.0 in a 3 node cluster.
Trying to run Kafka connect in distributed mode.

There are 3 workers in Connect cluster each running in each of the Kafka node.

When I add or edit a configuration via the REST API, one node in the cluster shows an error in the log. 

[2016-11-16 08:47:02,979] INFO 127.0.0.1 - - [16/Nov/2016:13:47:02 +0000] "POST /connectors/mqtt/tasks HTTP/1.1" 409 113  2 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-11-16 08:47:03,229] INFO SourceConnectorConfig values: 
connector.class = com.evokly.kafka.connect.mqtt.MqttSourceConnector
key.converter = null
name = mqtt
tasks.max = 6
value.converter = null
 (org.apache.kafka.connect.runtime.SourceConnectorConfig:180)
[2016-11-16 08:47:03,230] ERROR Unexpected error during connector task reconfiguration:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:804)
[2016-11-16 08:47:03,230] ERROR Task reconfiguration for mqtt failed unexpectedly, this connector will not be properly reconfigured unless manually triggered. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:805)
[2016-11-16 08:47:03,233] INFO 127.0.0.1 - - [16/Nov/2016:13:47:03 +0000] "POST /connectors/mqtt/tasks?forward=false HTTP/1.1" 409 113  1 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-11-16 08:47:03,234] INFO 127.0.0.1 - - [16/Nov/2016:13:47:03 +0000] "POST /connectors/mqtt/tasks?forward=true HTTP/1.1" 409 113  2 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-11-16 08:47:03,234] INFO 127.0.0.1 - - [16/Nov/2016:13:47:03 +0000] "POST /connectors/mqtt/tasks HTTP/1.1" 409 113  3 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-11-16 08:47:03,234] ERROR Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder:867)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot complete request because of a conflicting operation (e.g. worker rebalance)
at org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.run(DistributedHerder.java:864)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-11-16 08:47:03,234] ERROR Failed to reconfigure connector's tasks, retrying after backoff: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:793)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot complete request because of a conflicting operation (e.g. worker rebalance)
at org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.run(DistributedHerder.java:864)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-11-16 08:47:03,436] INFO Rebalance started (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1030)
[2016-11-16 08:47:03,437] INFO Stopping connector mqtt (org.apache.kafka.connect.runtime.Worker:218)
[2016-11-16 08:47:03,437] INFO Stop the MqttSourceConnector (com.evokly.kafka.connect.mqtt.MqttSourceConnector:92)
[2016-11-16 08:47:03,440] INFO Stopped connector mqtt (org.apache.kafka.connect.runtime.Worker:229)
[2016-11-16 08:47:03,440] INFO Stopping task mqtt-2 (org.apache.kafka.connect.runtime.Worker:341)
[2016-11-16 08:47:03,440] INFO Stoping the MqttSourceTask (com.evokly.kafka.connect.mqtt.MqttSourceConnector:142)
[2016-11-16 08:47:03,450] INFO [mqtt.client_id2paho20601603858025] Disconnected from Broker. (com.evokly.kafka.connect.mqtt.MqttSourceConnector:147)
[2016-11-16 08:47:03,450] INFO Stopping task mqtt-5 (org.apache.kafka.connect.runtime.Worker:341)
[2016-11-16 08:47:03,450] INFO Stoping the MqttSourceTask (com.evokly.kafka.connect.mqtt.MqttSourceConnector:142)
[2016-11-16 08:47:03,452] INFO [mqtt.client_id5paho20601603872172] Disconnected from Broker. (com.evokly.kafka.connect.mqtt.MqttSourceConnector:147)
^C[2016-11-16 08:47:08,116] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:68)
[2016-11-16 08:47:08,116] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:154)
[2016-11-16 08:47:08,134] INFO Stopped ServerConnector@1755e85b{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:306)
[2016-11-16 08:47:08,143] INFO Stopped o.e.j.s.ServletContextHandler@2f05be7f{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:865)
[2016-11-16 08:47:08,145] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:165)
[2016-11-16 08:47:08,146] INFO Herder stopping (org.apache.kafka.connect.runtime.distributed.DistributedHerder:351)
[2016-11-16 08:47:08,453] ERROR Graceful stop of task mqtt-2 failed. (org.apache.kafka.connect.runtime.Worker:358)
[2016-11-16 08:47:08,453] ERROR Graceful stop of task mqtt-5 failed. (org.apache.kafka.connect.runtime.Worker:358)
[2016-11-16 08:47:08,453] INFO Finished stopping tasks in preparation for rebalance (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1054)


The other two nodes' log has no error.
The error is shown in anyone of the three node each time. Even restarting the connect does not help.

Thanks in advance.

Ewen Cheslack-Postava

unread,
Nov 21, 2016, 1:55:37 PM11/21/16
to Confluent Platform
Is there more in the log? It's possible we should be logging those at WARN level instead. It is sometimes possible for requests between brokers to fail because a rebalance has started (at which point it is not safe for the leader to do things like write updated config values). However, as the second ERROR notes, it's going to retry after backing off, allowing the rebalance that is in progress to complete. Presumably this happened in the remainder of the log (or there were no errors and nothing needed to be logged). Did you observe something actually fail permanently (e.g. when you check its status via the REST API) or are you just concerned about the ERROR-level message?

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4df3b41b-feb2-4882-9df6-3dbdfb05d28d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Adarsh Lal

unread,
Nov 21, 2016, 11:42:26 PM11/21/16
to Confluent Platform
Hi,

Thanks for the reply.

I see the same error repeatedly in the log forever. In the log all I can see is this repeating continuously.
I am concerned about this line specifically

2016-11-16 08:47:03,230] ERROR Unexpected error during connector task reconfiguration:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:804)
[2016-11-16 08:47:03,230] ERROR Task reconfiguration for mqtt failed unexpectedly, this connector will not be properly reconfigured unless manually triggered. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:805)

I restarted the connect but could not see any change.
Out my six tasks, two are running in this worker. So I assume only 4 tasks are working now.

I see this error after upgrading into 10.1.0.


Thanks,

Adarsh
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

Adarsh Lal

unread,
Nov 28, 2016, 8:22:43 AM11/28/16
to Confluent Platform
Hi,

Is there any solution?

Thanks and Regards,
Adarsh 

Ewen Cheslack-Postava

unread,
Nov 29, 2016, 4:18:33 PM11/29/16
to Confluent Platform
Adarsh,

And there's no exception between those two log lines? In your original output there was an exception, but it just indicates the other node was still trying to process a rebalance. Now it looks like it's just empty? Without info about the exception it will be difficult to track down why it's getting stuck in this state.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/62eaabde-619b-40bd-b700-76fffd2d6f38%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Lakshmi G.S

unread,
Nov 30, 2016, 6:05:53 AM11/30/16
to Confluent Platform
Hi Ewen,

We are concerned about the error message. So is that a handled error which we shouldn't be concerned of? Can you please provide us with a link to refer where we can understand what is happening behind this? The REST call to get configurations work fine while we see this error in logs. We are a bit concerned to ignore an error without a satisfactory justifiction as we are moving to production shortly.

Any light on this would be appreciated :)

Regards,
Lakshmi G. S.


On Wednesday, November 30, 2016 at 2:48:33 AM UTC+5:30, Ewen Cheslack-Postava wrote:
Adarsh,

And there's no exception between those two log lines? In your original output there was an exception, but it just indicates the other node was still trying to process a rebalance. Now it looks like it's just empty? Without info about the exception it will be difficult to track down why it's getting stuck in this state.

-Ewen
On Mon, Nov 28, 2016 at 5:22 AM, Adarsh Lal <adars...@gmail.com> wrote:
Hi,

Is there any solution?

Thanks and Regards,
Adarsh 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

Ewen Cheslack-Postava

unread,
Dec 1, 2016, 5:32:24 PM12/1/16
to Confluent Platform
Yes, that reconfiguration has a built-in retry process -- it'll backoff for a bit and then try again. We should probably reword that and maybe drop the log level to WARN for certain types of exceptions that we know we can expect to happen occasionally.

-Ewen

On Wed, Nov 30, 2016 at 3:05 AM, Lakshmi G.S <lakshm...@gmail.com> wrote:
Hi Ewen,

We are concerned about the error message. So is that a handled error which we shouldn't be concerned of? Can you please provide us with a link to refer where we can understand what is happening behind this? The REST call to get configurations work fine while we see this error in logs. We are a bit concerned to ignore an error without a satisfactory justifiction as we are moving to production shortly.

Any light on this would be appreciated :)

Regards,
Lakshmi G. S.

On Wednesday, November 30, 2016 at 2:48:33 AM UTC+5:30, Ewen Cheslack-Postava wrote:
Adarsh,

And there's no exception between those two log lines? In your original output there was an exception, but it just indicates the other node was still trying to process a rebalance. Now it looks like it's just empty? Without info about the exception it will be difficult to track down why it's getting stuck in this state.

-Ewen
On Mon, Nov 28, 2016 at 5:22 AM, Adarsh Lal <adars...@gmail.com> wrote:
Hi,

Is there any solution?

Thanks and Regards,
Adarsh 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/87955050-dca4-4138-8001-e629904c7974%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Ewen Cheslack-Postava

unread,
Dec 1, 2016, 5:33:32 PM12/1/16
to Confluent Platform

To post to this group, send email to confluent-platform@googlegroups.com.



--
Thanks,
Ewen



--
Thanks,
Ewen

Lakshmi G.S

unread,
Dec 2, 2016, 8:19:46 AM12/2/16
to Confluent Platform
Thanks Ewen. But the interesting thing is that the error is happening in one worker after all 3 workers have finished re-balancing and the source and sink connectors start performing their task. 


On Friday, December 2, 2016 at 4:03:32 AM UTC+5:30, Ewen Cheslack-Postava wrote:
On Thu, Dec 1, 2016 at 2:32 PM, Ewen Cheslack-Postava <ew...@confluent.io> wrote:
Yes, that reconfiguration has a built-in retry process -- it'll backoff for a bit and then try again. We should probably reword that and maybe drop the log level to WARN for certain types of exceptions that we know we can expect to happen occasionally.

-Ewen
On Wed, Nov 30, 2016 at 3:05 AM, Lakshmi G.S <lakshm...@gmail.com> wrote:
Hi Ewen,

We are concerned about the error message. So is that a handled error which we shouldn't be concerned of? Can you please provide us with a link to refer where we can understand what is happening behind this? The REST call to get configurations work fine while we see this error in logs. We are a bit concerned to ignore an error without a satisfactory justifiction as we are moving to production shortly.

Any light on this would be appreciated :)

Regards,
Lakshmi G. S.

On Wednesday, November 30, 2016 at 2:48:33 AM UTC+5:30, Ewen Cheslack-Postava wrote:
Adarsh,

And there's no exception between those two log lines? In your original output there was an exception, but it just indicates the other node was still trying to process a rebalance. Now it looks like it's just empty? Without info about the exception it will be difficult to track down why it's getting stuck in this state.

-Ewen
On Mon, Nov 28, 2016 at 5:22 AM, Adarsh Lal <adars...@gmail.com> wrote:
Hi,

Is there any solution?

Thanks and Regards,
Adarsh 

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen



--
Thanks,
Ewen
Message has been deleted

Adarsh Lal

unread,
Dec 2, 2016, 8:45:02 AM12/2/16
to Confluent Platform
This is the error we see repeatedly in the log :

2016-12-02 08:37:18,147 226103 [qtp1381713434-24] DEBUG o.e.jetty.io.AbstractConnection - FILLING-->FILLING_FILL_INTERESTED HttpConnection@2387b437[FILLING_FILL_INTERESTED,SelectChannelEndPoint@43ca7c50{/127.0.0.1:1150<->8084,Open,in,out,-,-,1/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of -1},g=HttpGenerator{s=START},c=HttpChannelOverHttp@47f5e0f1{r=809,c=false,a=IDLE,uri=}]
2016-12-02 08:37:18,147 226103 [qtp1381713434-24] DEBUG o.e.jetty.io.AbstractConnection - FILLING_FILL_INTERESTED-->FILL_INTERESTED HttpConnection@2387b437[FILL_INTERESTED,SelectChannelEndPoint@43ca7c50{/127.0.0.1:1150<->8084,Open,in,out,-,-,1/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of -1},g=HttpGenerator{s=START},c=HttpChannelOverHttp@47f5e0f1{r=809,c=false,a=IDLE,uri=}]
2016-12-02 08:37:18,147 226103 [qtp1381713434-56 - /connectors/mqtt/tasks?forward=true] DEBUG o.a.k.c.r.r.e.ConnectExceptionMapper - Uncaught exception in REST call: 
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot complete request because of a conflicting operation (e.g. worker rebalance)
at org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229) ~[connect-runtime-0.10.1.0.jar:na]
at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:249) ~[connect-runtime-0.10.1.0.jar:na]
at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:278) ~[connect-runtime-0.10.1.0.jar:na]
at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.putTaskConfigs(ConnectorsResource.java:192) ~[connect-runtime-0.10.1.0.jar:na]
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.22.2.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.22.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.22.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.22.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.22.2.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.22.2.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.22.2.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.22.2.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.22.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.22.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.22.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.22.2.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
2016-12-02 08:37:18,147 226103 [qtp1381713434-24] DEBUG o.e.jetty.io.SelectChannelEndPoint - Local interests updating 0 -> 1 for SelectChannelEndPoint@43ca7c50{/127.0.0.1:1150<->8084,Open,in,out,R,-,0/30000,HttpConnection}{io=1,kio=0,kro=1}
2016-12-02 08:37:18,148 226104 [qtp1381713434-24] DEBUG o.eclipse.jetty.io.SelectorManager - Queued change org.eclipse.jetty.io.SelectChannelEndPoint$1@333f74eb
2016-12-02 08:37:18,148 226104 [qtp1381713434-20-selector-ServerConnectorManager@105fece7/2] DEBUG o.eclipse.jetty.io.SelectorManager - Selector loop woken up from select, 0/1 selected

Adarsh Lal

unread,
Dec 5, 2016, 10:43:56 AM12/5/16
to Confluent Platform

Hi Ewen,

We are facing another issue.
We are testing connect in a 3 node cluster. After starting, we are trying to register a connector via REST API. Now in all the workers, the task gets started and then it stops after a second. This happens again and again and the tasks never gets initialized. We can see the tasks getting started and stopped in the log. This never stops. We tried waiting some time. 

Tried restarting the workers several times. No use.

We tried using different connectors(source and sink).

Our development activities are actually blocked by this. 

We tested the same connector in stand alone mode, everything was working fine.

Thanks in advance,

Adarshlal

Saeed Shariati

unread,
Jul 16, 2018, 7:08:50 AM7/16/18
to Confluent Platform
Hi,

It seems we need to send request into the leading broker.

Cheers,
Saeed
Reply all
Reply to author
Forward
0 new messages