Distributed worker

210 views

Skip to first unread message

Peng Liu

unread,

Aug 9, 2016, 4:41:03 PM8/9/16

to Confluent Platform

Trying to run distributed workers on Mesos, but after having 10+ jobs running, the REST API for the workers seems to stop responding, preventing us from adding/removing/modifying jobs. Please advise. Thank you :)

[2016-08-09 16:18:59,507] DEBUG Uncaught exception in REST call: (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:35)

org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Request timed out

at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:265)

at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:273)

at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.listConnectors(ConnectorsResource.java:83)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:49

at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)

at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)

at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)

at org.glassfish.jersey.internal.Errors.process(Errors.java:315)

at org.glassfish.jersey.internal.Errors.process(Errors.java:297)

at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)

at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)

at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)

at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)

at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:38

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:22

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:499)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

Peng Liu

unread,

Aug 9, 2016, 4:54:55 PM8/9/16

to Confluent Platform

To add to this, a month ago while testing deployment, I noticed that when this happens with multiple distributed workers, killing all but one would allow you to call the REST API of the final, single worker. Again, not sure why.

Ewen Cheslack-Postava

unread,

Aug 9, 2016, 11:41:36 PM8/9/16

to Confluent Platform

The part at the top of the stack trace referencing ConnectorsResource.completeOrForwardRequest indicates that the worker handling the request is trying to forward it to the leader (some requests must be forwarded to the leader for handling). It seems that operation is timing out. I suspect you don't need to kill N-1 instances, it was probably just the leader that was getting stuck and by eventually (at random) killing it, it allowed the rest of the cluster to continue functioning normally.

One way to validate this if you get into this state again is to hit an API endpoint that doesn't require forwarding, e.g. GET /connector-plugins. If that continues to work but other endpoints hang, it's probably the leader hanging. At that point, to track down the real source of the issue you'd probably want to check the logs for the last assignment made for the cluster (should get logged with something like "Joined group and got assignment:"). Then use that to check the logs of the leader for any hints about why it is hanging.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ca6786e1-cf73-4f8f-b59f-e498cee3bdfa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Thanks,
Ewen

Peng Liu

unread,

Aug 10, 2016, 12:37:51 AM8/10/16

to Confluent Platform

Thanks. Will try this and get back to you! :)

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ca6786e1-cf73-4f8f-b59f-e498cee3bdfa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Thanks,
Ewen

Shao Wang

unread,

Dec 9, 2016, 11:30:23 PM12/9/16

to Confluent Platform

We are facing the similiar issue. The way to reproduce this issue is to create a connector and then delete it. After that the request to /connectors/ will always return 500 timeout error. But we could still get connectors/{connector}/status response for the connector we just sent a DELETE request to, which status became "UNASSIGNED" while the task status showed "RUNNING". We are using Confluent Kafka Connect version 3.1.1 with only one worker.

Shao Wang

unread,

Dec 10, 2016, 12:05:58 AM12/10/16

to Confluent Platform

This is the debug log when executing listConnectors request

[2016-12-10 05:03:23,124] DEBUG Updated cluster metadata version 15 to Cluster(id = lRs4JGziQneqK0TrPZKqGQ, nodes = [kafka1:9092 (id: 1 rack: null), kafka3:9092 (id: 1001 rack: null), kafka2:9092 (id: 1002 rack: null)], partitions = [Partition(topic = _confluent-monitoring, partition = 0, leader = 1002, replicas = [1002,], isr = [1002,])]) (org.apache.kafka.clients.Metadata)

[2016-12-10 05:03:38,123] DEBUG Publish clientType: PRODUCER

clientId: "producer-6"

session: "2cb5df39-74ad-4843-a1fc-840dfe51c5ce"

sequence: 257

timestamp: 1481346218123

topic: "dptask_101"

samplePeriod: 15000

type: HEARTBEAT

minWindow: -1

maxWindow: -1

clusterId: "lRs4JGziQneqK0TrPZKqGQ"

(io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)

[2016-12-10 05:03:38,124] DEBUG Publish clientType: PRODUCER

clientId: "producer-6"

session: "2cb5df39-74ad-4843-a1fc-840dfe51c5ce"

sequence: 294

timestamp: 1481346218123

topic: "dptask_101.kf.cdsgus"

samplePeriod: 15000

type: HEARTBEAT

minWindow: -1

maxWindow: -1

clusterId: "lRs4JGziQneqK0TrPZKqGQ"

(io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)

[2016-12-10 05:03:38,531] DEBUG Queued change org.eclipse.jetty.io.SelectorManager$ManagedSelector$Accept@75623f89 (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,531] DEBUG Selector loop woken up from select, 0/2 selected (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,531] DEBUG Running change org.eclipse.jetty.io.SelectorManager$ManagedSelector$Accept@75623f89 (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,531] DEBUG onOpen SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,0/30000,null}{io=0,kio=0,kro=0} (org.eclipse.jetty.io.AbstractEndPoint)

[2016-12-10 05:03:38,531] DEBUG SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,0/30000,null}{io=0,kio=0,kro=0} idle timeout check, elapsed: 0 ms, remaining: 30000 ms (org.eclipse.jetty.io.IdleTimeout)

[2016-12-10 05:03:38,531] DEBUG new HttpChannelOverHttp@2d07cd84{r=0,c=false,a=IDLE,uri=} -> SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,0/30000,null}{io=0,kio=0,kro=0},null,HttpChannelState@73e7b026{s=IDLE i=true a=null} (org.eclipse.jetty.server.HttpChannel)

[2016-12-10 05:03:38,531] DEBUG New HTTP Connection HttpConnection@1679bfd9[IDLE,SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,0/30000,null}{io=0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@2d07cd84{r=0,c=false,a=IDLE,uri=}] (org.eclipse.jetty.server.HttpConnection)

[2016-12-10 05:03:38,532] DEBUG onOpen HttpConnection@1679bfd9[IDLE,SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,1/30000,HttpConnection}{io=0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@2d07cd84{r=0,c=false,a=IDLE,uri=}] (org.eclipse.jetty.io.AbstractConnection)

[2016-12-10 05:03:38,532] DEBUG fillInterested HttpConnection@1679bfd9[IDLE,SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,1/30000,HttpConnection}{io=0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@2d07cd84{r=0,c=false,a=IDLE,uri=}] (org.eclipse.jetty.io.AbstractConnection)

[2016-12-10 05:03:38,532] DEBUG IDLE-->FILL_INTERESTED HttpConnection@1679bfd9[FILL_INTERESTED,SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,1/30000,HttpConnection}{io=0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@2d07cd84{r=0,c=false,a=IDLE,uri=}] (org.eclipse.jetty.io.AbstractConnection)

[2016-12-10 05:03:38,532] DEBUG Local interests updating 0 -> 1 for SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,R,-,0/30000,HttpConnection}{io=1,kio=0,kro=0} (org.eclipse.jetty.io.SelectChannelEndPoint)

[2016-12-10 05:03:38,532] DEBUG Queued change org.eclipse.jetty.io.SelectChannelEndPoint$1@7666ab72 (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,532] DEBUG Created SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,R,-,0/30000,HttpConnection}{io=1,kio=0,kro=0} (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,532] DEBUG Running change org.eclipse.jetty.io.SelectChannelEndPoint$1@7666ab72 (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,532] DEBUG Key interests updated 0 -> 1 on SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,R,-,0/30000,HttpConnection}{io=1,kio=1,kro=0} (org.eclipse.jetty.io.SelectChannelEndPoint)

[2016-12-10 05:03:38,532] DEBUG Selector loop waiting on select (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,532] DEBUG Selector loop woken up from select, 1/3 selected (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,532] DEBUG Key interests updated 1 -> 0 on SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,R,-,0/30000,HttpConnection}{io=1,kio=0,kro=1} (org.eclipse.jetty.io.SelectChannelEndPoint)

[2016-12-10 05:03:38,532] DEBUG Local interests updating 1 -> 0 for SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,R,-,0/30000,HttpConnection}{io=0,kio=0,kro=1} (org.eclipse.jetty.io.SelectChannelEndPoint)

[2016-12-10 05:03:38,532] DEBUG Queued change org.eclipse.jetty.io.SelectChannelEndPoint$1@7666ab72 (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,532] DEBUG FILL_INTERESTED-->FILLING HttpConnection@1679bfd9[FILLING,SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,0/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@2d07cd84{r=0,c=false,a=IDLE,uri=}] (org.eclipse.jetty.io.AbstractConnection)

[2016-12-10 05:03:38,532] DEBUG Running change org.eclipse.jetty.io.SelectChannelEndPoint$1@7666ab72 (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,532] DEBUG Selector loop waiting on select (org.eclipse.jetty.io.SelectorManager)

[2016-12-10 05:03:38,532] DEBUG HttpConnection@1679bfd9[FILLING,SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,0/30000,HttpConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@2d07cd84{r=0,c=false,a=IDLE,uri=}] onFillable HttpChannelState@73e7b026{s=IDLE i=true a=null} (org.eclipse.jetty.server.HttpConnection)

[2016-12-10 05:03:38,532] DEBUG filled 91 SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,0/30000,HttpConnection}{io=0,kio=0,kro=1} (org.eclipse.jetty.io.ChannelEndPoint)

[2016-12-10 05:03:38,532] DEBUG parseNext s=START HeapByteBuffer@1f361d1a[p=0,l=91,c=16384,r=91]={<<<GET /connectors/ ...Accept: */*\r\n\r\n>>>rcemysql:8083\r\nAc...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG START --> SPACE1 (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG SPACE1 --> URI (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG URI --> SPACE2 (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG SPACE2 --> REQUEST_VERSION (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG REQUEST_VERSION --> HEADER (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER --> HEADER_VALUE (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER_VALUE --> HEADER_IN_VALUE (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER_IN_VALUE --> HEADER (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER --> HEADER_VALUE (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER_VALUE --> HEADER_IN_VALUE (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER_IN_VALUE --> HEADER (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER --> HEADER_IN_VALUE (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER_IN_VALUE --> HEADER (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HEADER --> END (org.eclipse.jetty.http.HttpParser)

[2016-12-10 05:03:38,532] DEBUG HttpChannelOverHttp@2d07cd84{r=1,c=false,a=IDLE,uri=/connectors/} messageComplete (org.eclipse.jetty.server.HttpChannel)

[2016-12-10 05:03:38,532] DEBUG HttpInputOverHTTP@1abae62c EOF (org.eclipse.jetty.server.HttpInput)

[2016-12-10 05:03:38,532] DEBUG HttpChannelOverHttp@2d07cd84{r=1,c=false,a=IDLE,uri=/connectors/} handle enter (org.eclipse.jetty.server.HttpChannel)

[2016-12-10 05:03:38,532] DEBUG HttpChannelState@73e7b026{s=IDLE i=true a=null} handling IDLE (org.eclipse.jetty.server.HttpChannelState)

[2016-12-10 05:03:38,532] DEBUG HttpChannelOverHttp@2d07cd84{r=1,c=false,a=DISPATCHED,uri=/connectors/} action REQUEST_DISPATCH (org.eclipse.jetty.server.HttpChannel)

[2016-12-10 05:03:38,533] DEBUG REQUEST GET /connectors/ on HttpChannelOverHttp@2d07cd84{r=1,c=false,a=DISPATCHED,uri=/connectors/} (org.eclipse.jetty.server.Server)

[2016-12-10 05:03:38,533] DEBUG scope null||/connectors/ @ o.e.j.s.ServletContextHandler@f9d87b{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)

[2016-12-10 05:03:38,533] DEBUG context=||/connectors/ @ o.e.j.s.ServletContextHandler@f9d87b{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)

[2016-12-10 05:03:38,533] DEBUG sessionManager=org.eclipse.jetty.server.session.HashSessionManager@184497d1 (org.eclipse.jetty.server.session)

[2016-12-10 05:03:38,533] DEBUG session=null (org.eclipse.jetty.server.session)

[2016-12-10 05:03:38,533] DEBUG servlet ||/connectors/ -> org.glassfish.jersey.servlet.ServletContainer-3185fa6b@d91e5d0==org.glassfish.jersey.servlet.ServletContainer,-1,true (org.eclipse.jetty.servlet.ServletHandler)

[2016-12-10 05:03:38,533] DEBUG chain=null (org.eclipse.jetty.servlet.ServletHandler)

[2016-12-10 05:03:38,647] DEBUG Sending metadata request {topics=[]} to node 1002 (org.apache.kafka.clients.NetworkClient)

[2016-12-10 05:03:38,648] DEBUG Updated cluster metadata version 17 to Cluster(id = lRs4JGziQneqK0TrPZKqGQ, nodes = [kafka1:9092 (id: 1 rack: null), kafka3:9092 (id: 1001 rack: null), kafka2:9092 (id: 1002 rack: null)], partitions = []) (org.apache.kafka.clients.Metadata)

[2016-12-10 05:03:39,443] DEBUG Scavenging sessions at 1481346219443 (org.eclipse.jetty.server.session)

[2016-12-10 05:03:47,594] DEBUG SelectChannelEndPoint@5e731e91{/10.0.0.28:35172<->8083,Open,in,out,-,-,30000/30000,HttpConnection}{io=0,kio=0,kro=1} idle timeout check, elapsed: 30000 ms, remaining: 0 ms (org.eclipse.jetty.io.IdleTimeout)

[2016-12-10 05:03:47,594] DEBUG SelectChannelEndPoint@5e731e91{/10.0.0.28:35172<->8083,Open,in,out,-,-,30001/30000,HttpConnection}{io=0,kio=0,kro=1} idle timeout expired (org.eclipse.jetty.io.IdleTimeout)

[2016-12-10 05:03:47,594] DEBUG ignored: WriteFlusher@418dcd18{IDLE} {} (org.eclipse.jetty.io.WriteFlusher)

[2016-12-10 05:03:47,594] DEBUG Ignored idle endpoint SelectChannelEndPoint@5e731e91{/10.0.0.28:35172<->8083,Open,in,out,-,-,30001/30000,HttpConnection}{io=0,kio=0,kro=1} (org.eclipse.jetty.io.AbstractEndPoint)

[2016-12-10 05:03:50,730] DEBUG SelectChannelEndPoint@a2f14c7{/10.0.0.28:35173<->8083,Open,in,out,-,-,30000/30000,HttpConnection}{io=0,kio=0,kro=1} idle timeout check, elapsed: 30000 ms, remaining: 0 ms (org.eclipse.jetty.io.IdleTimeout)

[2016-12-10 05:03:50,731] DEBUG SelectChannelEndPoint@a2f14c7{/10.0.0.28:35173<->8083,Open,in,out,-,-,30001/30000,HttpConnection}{io=0,kio=0,kro=1} idle timeout expired (org.eclipse.jetty.io.IdleTimeout)

[2016-12-10 05:03:50,731] DEBUG ignored: WriteFlusher@7027b315{IDLE} {} (org.eclipse.jetty.io.WriteFlusher)

[2016-12-10 05:03:50,731] DEBUG Ignored idle endpoint SelectChannelEndPoint@a2f14c7{/10.0.0.28:35173<->8083,Open,in,out,-,-,30001/30000,HttpConnection}{io=0,kio=0,kro=1} (org.eclipse.jetty.io.AbstractEndPoint)

[2016-12-10 05:03:53,124] DEBUG Publish clientType: PRODUCER

clientId: "producer-6"

session: "2cb5df39-74ad-4843-a1fc-840dfe51c5ce"

sequence: 258

timestamp: 1481346233124

topic: "dptask_101"

samplePeriod: 15000

type: HEARTBEAT

minWindow: -1

maxWindow: -1

clusterId: "lRs4JGziQneqK0TrPZKqGQ"

(io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)

[2016-12-10 05:03:53,124] DEBUG Publish clientType: PRODUCER

clientId: "producer-6"

session: "2cb5df39-74ad-4843-a1fc-840dfe51c5ce"

sequence: 295

timestamp: 1481346233124

topic: "dptask_101.kf.cdsgus"

samplePeriod: 15000

type: HEARTBEAT

minWindow: -1

maxWindow: -1

clusterId: "lRs4JGziQneqK0TrPZKqGQ"

(io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)

[2016-12-10 05:04:05,716] DEBUG Sending metadata request {topics=[]} to node 1 (org.apache.kafka.clients.NetworkClient)

[2016-12-10 05:04:05,717] DEBUG Updated cluster metadata version 17 to Cluster(id = lRs4JGziQneqK0TrPZKqGQ, nodes = [kafka2:9092 (id: 1002 rack: null), kafka1:9092 (id: 1 rack: null), kafka3:9092 (id: 1001 rack: null)], partitions = []) (org.apache.kafka.clients.Metadata)

[2016-12-10 05:04:08,124] DEBUG Publish clientType: PRODUCER

clientId: "producer-6"

session: "2cb5df39-74ad-4843-a1fc-840dfe51c5ce"

sequence: 259

timestamp: 1481346248124

topic: "dptask_101"

samplePeriod: 15000

type: HEARTBEAT

minWindow: -1

maxWindow: -1

clusterId: "lRs4JGziQneqK0TrPZKqGQ"

(io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)

[2016-12-10 05:04:08,125] DEBUG Publish clientType: PRODUCER

clientId: "producer-6"

session: "2cb5df39-74ad-4843-a1fc-840dfe51c5ce"

sequence: 296

timestamp: 1481346248124

topic: "dptask_101.kf.cdsgus"

samplePeriod: 15000

type: HEARTBEAT

minWindow: -1

maxWindow: -1

clusterId: "lRs4JGziQneqK0TrPZKqGQ"

(io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)

[2016-12-10 05:04:08,532] DEBUG SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,30000/30000,HttpConnection}{io=0,kio=0,kro=1} idle timeout check, elapsed: 30000 ms, remaining: 0 ms (org.eclipse.jetty.io.IdleTimeout)

[2016-12-10 05:04:08,532] DEBUG ignored: WriteFlusher@106ce6a9{IDLE} {} (org.eclipse.jetty.io.WriteFlusher)

[2016-12-10 05:04:08,532] DEBUG Ignored idle endpoint SelectChannelEndPoint@5cf7911d{/10.0.0.28:35184<->8083,Open,in,out,-,-,30000/30000,HttpConnection}{io=0,kio=0,kro=1} (org.eclipse.jetty.io.AbstractEndPoint)

[2016-12-10 05:04:09,443] DEBUG Scavenging sessions at 1481346249443 (org.eclipse.jetty.server.session)

[2016-12-10 05:04:13,061] DEBUG Sending metadata request {topics=[]} to node 1001 (org.apache.kafka.clients.NetworkClient)

[2016-12-10 05:04:13,062] DEBUG Updated cluster metadata version 18 to Cluster(id = lRs4JGziQneqK0TrPZKqGQ, nodes = [kafka1:9092 (id: 1 rack: null), kafka2:9092 (id: 1002 rack: null), kafka3:9092 (id: 1001 rack: null)], partitions = []) (org.apache.kafka.clients.Metadata)

[2016-12-10 05:04:17,594] DEBUG SelectChannelEndPoint@5e731e91{/10.0.0.28:35172<->8083,Open,in,out,-,-,30000/30000,HttpConnection}{io=0,kio=0,kro=1} idle timeout check, elapsed: 30000 ms, remaining: 0 ms (org.eclipse.jetty.io.IdleTimeout)

[2016-12-10 05:04:17,594] DEBUG ignored: WriteFlusher@418dcd18{IDLE} {} (org.eclipse.jetty.io.WriteFlusher)

[2016-12-10 05:04:17,594] DEBUG Ignored idle endpoint SelectChannelEndPoint@5e731e91{/10.0.0.28:35172<->8083,Open,in,out,-,-,30000/30000,HttpConnection}{io=0,kio=0,kro=1} (org.eclipse.jetty.io.AbstractEndPoint)

[2016-12-10 05:04:17,594] DEBUG Uncaught exception in REST call: (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper)

org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Request timed out

at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:265)

at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:273)

at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.listConnectors(ConnectorsResource.java:83)

at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)