Request to leader to reconfigure connector tasks failed

[Adarsh Lal]

Hi,

I am trying to start Kafka connect in distributed mode in a 3 node Kafka cluster.

I am starting 1 worker each on 3 nodes in the cluster. 

After starting the Kafka connect in distributed mode, one worker is always showing this error in the log

[2016-10-11 02:13:17,966] DEBUG onClose SelectChannelEndPoint@59f2d5c4{/127.0.0.1:43106<->8083,CLOSED,ISHUT,OSHUT,-,-,5/30000,HttpConnection}{io=0,kio=-1,kro=-1} (org.eclipse.jetty.io.AbstractEndPoint:108)

[2016-10-11 02:13:17,966] ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestServer:241)

java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at java.net.Socket.connect(Socket.java:538)

at sun.net.NetworkClient.doConnect(NetworkClient.java:180)

at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)

at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)

at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:667)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)

at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)

at org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:222)

at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.run(DistributedHerder.java:878)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

[2016-10-11 02:13:17,969] ERROR Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder:881)

org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error trying to forward REST request: Connection refused

at org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:242)

at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.run(DistributedHerder.java:878)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at java.net.Socket.connect(Socket.java:538)

at sun.net.NetworkClient.doConnect(NetworkClient.java:180)

at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)

at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)

at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:667)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)

at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)

at org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:222)

... 6 more

But I am getting the success response when I fire the REST calls to all the three workers. 

I was able to fire the POST, PUT and GET message on the same worker where I see this error.

What is the reason for this error? If there was any problem with the worker's connection how am I able to get response to the REST calls in that worker.

Thanks in advance.

[Shikhar Bhushan]

Hi Adarsh,

Perhaps it is a firewall issue - can you test that all the other workers' REST endpoints are reachable from each worker?

Best,

Shikhar

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ed18c03f-8a0a-4b04-bc4f-cd191428b4b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adarsh Lal]

Hi Shikhar,

Thank you for the reply.

I checked the connections. Ping was working between all three machines. All of these are digital ocean droplets(Ubuntu).

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

[Ewen Cheslack-Postava]

Another possibility is that the hostname/port being advertised aren't routable by other instances. This is somewhat common on cloud providers, and especially happens if they provide private/public IPs. Maybe try explicitly setting the rest.advertised.host.name and/or rest.advertised.port settings for the workers?

With sufficiently detailed logging level, you should also see the hostname/port of the leader included in the logs, which might also help track down the issue.

-Ewen

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ed18c03f-8a0a-4b04-bc4f-cd191428b4b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/12bb6fbf-d741-4287-b970-8cf3bd413fc4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen