Re: Issues while running kafka connect in distributed mode

Ewen Cheslack-Postava

unread,

Jan 27, 2017, 1:11:33 PM1/27/17

to Confluent Platform

For the unsupported media type issue, try removing everything from the -H parameter other than the Content-Type. My guess would be that one of them is causing that issue.

Other than that, if you get a timeout or other error, looking at the server log will provide more useful information. Without some info from that log it'll be hard to determine what the problem is (unless it simply turns out to simply be the address you are using, but since you're getting a response in at least one case that address seems to be usable).

-Ewen

On Thu, Jan 26, 2017 at 5:41 AM, Nishant Verma <nishant....@gmail.com> wrote:

We are testing kafka connect in distributed mode to pull topic records from kafka to HDFS. We have two boxes. One in which kafka and zookeeper daemons are running. We have kept one instance of kafka connect in this box. We have another box where HDFS namenode is present. We have kept another instance of kafka connect here.

We started kafka,zookeeper and kafka connect in first box. We started kafka connect in second box as well. Now as per confluent documentation, we have to start the HDFS connector(or any other connector for that matter) using REST API. So, after starting kafka connect in these two boxes, we tried starting connector through REST API. We tried below command:-

curl -X POST -H "HTTP/1.1 Host: ip-10-16-34-57.ec2.internal:9092 Content-Type: application/json Accept: application/json" --data '{"name": "hdfs-sink", "config": {"connector.class":"io.confluent.connect.hdfs.HdfsSinkConnector", "format.class":"com.qubole.streamx.SourceFormat", "tasks.max":"1", "hdfs.url":"hdfs://ip-10-16-37-124:9000", "topics":"Prd_IN_TripAnalysis,Prd_IN_Alerts,Prd_IN_GeneralEvents", "partitioner.class":"io.confluent.connect.hdfs.partitioner.DailyPartitioner", "locale":"", "timezone":"Asia/Calcutta" }}' http://ip-10-16-34-57.ec2.internal:8083/connectors

As soon as we press enter here, we get below response:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 415 </title>
</head>
<body>
<h2>HTTP ERROR: 415</h2>
<p>Problem accessing /connectors. Reason:
<pre> Unsupported Media Type</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

(tried with localhost,ip-10-16-37-124,10.16.34.57,10.16.37.124 also). Have got "curl: (7) Failed connect to 10.16.37.124:8083; Connection refused" and "Error 500: Request Timed out" in those cases.

The connect-distributed.properties file at etc/kafka/ is below in both the kafka connect nodes. We have created the said three topics as well (connect-offsets,connect-configs,connect-status)

bootstrap.servers=ip-10-16-34-57.ec2.internal:9092
group.id=connect-cluster
key.converter=com.qubole.streamx.ByteArrayConverter
value.converter=com.qubole.streamx.ByteArrayConverter
enable.auto.commit=true
auto.commit.interval.ms=1000
offset.flush.interval.ms=1000
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
rest.port=8083
config.storage.topic=connect-configs
status.storage.topic=connect-status
offset.flush.interval.ms=10000

ip-10-16-34-57.ec2.internal = server where kafka,zookeeper and one instance of kafka connect is running
ip-10-16-37-124 = server where hdfs namenode and second instance of kafka connect is runinng

What is the issue here? Are we missing something to start kafka connect in distributed mode to work with HDFS connectors.
Looking forward to hearing from you.

kafka connect in standalone mode is working fine.

Nishant

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c612f5da-4f86-4c2e-ae77-9e926d458538%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

konst...@confluent.io

unread,

Jan 27, 2017, 3:05:32 PM1/27/17

to Confluent Platform

Ewen is right, it should work only with Content-type in the headers.

But anytime you need to add multiple headers with curl (and make your example above work), you need to add each header as a separate argument preceded by -H

My guess is that, your example, if you run it as:

curl -X POST -H "Content-Type: application/json" - H "Accept: application/json" [the remainder of your curl command]

will succeed.

A good way to debug curl commands you want to submit to the rest endpoint is to also use Postman while you write them. It often helps to get it right sooner.

Cheers,

Konstantine

To post to this group, send email to confluent...@googlegroups.com.

Nishant Verma

unread,

Jan 29, 2017, 10:39:57 PM1/29/17

to Confluent Platform

Thanks for the update Ewen and Konstantine.

The unsupported media type issue is not coming anymore. There was some problem with kafka connect rest service at that time.

Currently, below issue is coming:

curl -X POST -H "Content-Type: application/json" --data '{"name": "hdfs-st.connect.hdfs.HdfsSinkConnector", "format.class": "com.qubole.streamx.SourceFormat", "tasks.max": "1", "hdfs.url": "hdfs://10.16.37.124:9000", "topics": "Prd_IN_TripAnalysis,Prd_IN_Alerts,Prd_IN_GeneralEvents", "partitioner.class": "io.confluent.connect.hdfs.partitioner.DailyPartitioner", "locale": "", "timezone": "Asia/Calcutta" }}' http://localhost:8083/connectors

As we press enter here, we get below response after some time:

{"error_code":500,"message":"Request timed out

Also, when we give "curl http://localhost:8083/connector-plugins", we get below o/p:

[{"class":"com.qubole.streamx.s3.S3SinkConnector"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector"},{"class":"io.confluent.connect.hdfs.HdfsSinkConnector"},{"class":"io.confluent.connect.jdbc.JdbcSourceConnector"},{"class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector"},{"class":"io.confluent.connect.hdfs.tools.SchemaSourceConnector"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector"},{"class":"io.confluent.connect.jdbc.JdbcSinkConnector"}]

But when we try, " curl http://localhost:8083/connectors", we get :-

{"error_code":500,"message":"Request timed out"} after some time.

Please node we use below steps to run:-

a- start kafka-rest services in both the boxes.

b- start ConnectDistributed services in both the boxes.

c- run the curl command and see the said error.

What could be the issue here now?

Thanks

Nishant Verma

To post to this group, send email to confluent...@googlegroups.com.

Nishant Verma

unread,

Jan 29, 2017, 10:43:03 PM1/29/17

to Confluent Platform

Duplicate thread started by me at below link. Excuse me for this.

https://groups.google.com/forum/#!topic/confluent-platform/e9TKC3UtOfk

On Friday, January 27, 2017 at 11:41:33 PM UTC+5:30, Ewen Cheslack-Postava wrote:

To post to this group, send email to confluent...@googlegroups.com.

Nishant Verma

unread,

Jan 30, 2017, 4:05:39 AM1/30/17

to Confluent Platform

We found from ConnectorsResource.class from org.apache.kafka.connect.runtime.rest.resources package that the listConnectors, createConnectors methods require a boolean value in url for forward. Hence we also tried with - curl http://localhost:8083?forward=true.

But the request timed out response is still coming.

(Tried replacing localhost with IPs as well but no change in error or response)

On Friday, January 27, 2017 at 11:41:33 PM UTC+5:30, Ewen Cheslack-Postava wrote:

To post to this group, send email to confluent...@googlegroups.com.

Abdurrahman Adebiyi

unread,

Jun 17, 2017, 6:35:14 AM6/17/17

to Confluent Platform

@Banjo did you finally have this resolved as I have similar issues and I have try on a CP-OSS-3.2.1 and Apache kafka 10.2.1. I run a 3N - Kafka Broker.

I will appreciate your help.

To post to this group, send email to confluent...@googlegroups.com.

Abdurrahman Adebiyi

unread,

Jun 17, 2017, 7:07:47 AM6/17/17

to Confluent Platform

In my kafka-connect runtime I see the following warnings

WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.

WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.

WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.

WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

Could this be the reason /connectors API ain't working?

Abdurrahman Adebiyi

unread,

Jun 18, 2017, 1:24:41 AM6/18/17

to Confluent Platform

Well ... finally got it work ... apparently I needed to ensure I had a broker with id "0". Once I did that i was able to get pass it and even spawned a connect cluster.

One more thing the WARNINGS didn't matter . It worked.Please snippets of my logs below . Hope it helps someone else.

[2017-06-18 06:12:35,231] INFO Started o.e.j.s.ServletContextHandler@1cbf6e72{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)

[2017-06-18 06:12:35,232] INFO Finished reading KafkaBasedLog for topic connect-configs (org.apache.kafka.connect.util.KafkaBasedLog:146)

[2017-06-18 06:12:35,232] INFO Started KafkaBasedLog for topic connect-configs (org.apache.kafka.connect.util.KafkaBasedLog:148)

[2017-06-18 06:12:35,232] INFO Started KafkaConfigBackingStore (org.apache.kafka.connect.storage.KafkaConfigBackingStore:248)

[2017-06-18 06:12:35,233] INFO Herder started (org.apache.kafka.connect.runtime.distributed.DistributedHerder:195)

[2017-06-18 06:12:35,238] INFO Started ServerConnector@2b35894d{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266)

[2017-06-18 06:12:35,238] INFO Started @1136ms (org.eclipse.jetty.server.Server:379)

[2017-06-18 06:12:35,238] INFO REST server listening at http://10.77.8.72:8083/, advertising URL http://10.77.8.72:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:150)

[2017-06-18 06:12:35,238] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:56)

[2017-06-18 06:12:35,260] INFO Discovered coordinator insight-01a:9092 (id: 2147483647 rack: null) for group amghouse-connect-cluster. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:586)

[2017-06-18 06:12:35,263] INFO (Re-)joining group amghouse-connect-cluster (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:420)

[2017-06-18 06:12:35,283] INFO Successfully joined group amghouse-connect-cluster with generation 3 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:388)

[2017-06-18 06:12:35,284] INFO Joined group and got assignment: Assignment{error=0, leader='connect-1-c8b5aed3-937e-4f0b-aaf2-4b929659f8d8', leaderUrl='http://10.77.8.72:8083/', offset=-1, connectorIds=[], taskIds=[]} (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1151)

[2017-06-18 06:12:35,284] INFO Starting connectors and tasks using config offset -1 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:814)

[2017-06-18 06:12:35,284] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:824)

[2017-06-18 06:12:36,734] INFO Reflections took 1438 ms to scan 79 urls, producing 3636 keys and 18128 values (org.reflections.Reflections:229)

Pavan Gavvala

unread,

Jul 11, 2017, 10:26:12 PM7/11/17

to Confluent Platform

Hi,

I am getting the same error:

[root@pcdh2 ~]# curl localhost:8083/connectors
{"error_code":500,"message":"Request timed out"}[root@pcdh2 ~]#

While the below command works fine.

[root@pcdh2 ~]# curl localhost:8083/connector-plugins
[{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector"},{"class":"io.confluent.connect.hdfs.tools.SchemaSourceConnector"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector"},{"class":"io.confluent.connect.jdbc.JdbcSourceConnector"},{"class":"io.confluent.connect.hdfs.HdfsSinkConnector"}]

Could someone help me in debugging this ?

On Friday, January 27, 2017 at 11:41:33 PM UTC+5:30, Ewen Cheslack-Postava wrote:

To post to this group, send email to confluent...@googlegroups.com.

Pavan Gavvala

unread,

Jul 11, 2017, 10:33:49 PM7/11/17

to Confluent Platform

I am running kafka in distributed mode. These commands work fine in standalone mode but not in distributed mode.

Pavan Gavvala

unread,

Jul 11, 2017, 11:05:29 PM7/11/17

to Confluent Platform

The error I see in DEBUG mode:

[2017-07-12 08:25:26,114] DEBUG Uncaught exception in REST call: (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:35)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Request timed out
        at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:265)
        at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:273)
        at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.listConnectors(ConnectorsResource.java:83)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)

On Friday, January 27, 2017 at 11:41:33 PM UTC+5:30, Ewen Cheslack-Postava wrote:

To post to this group, send email to confluent...@googlegroups.com.

Pavan Gavvala

unread,

Jul 12, 2017, 11:16:47 AM7/12/17

to Confluent Platform

Could someone help me on how to proceed to resolve this issue ?

On Friday, January 27, 2017 at 11:41:33 PM UTC+5:30, Ewen Cheslack-Postava wrote:

To post to this group, send email to confluent...@googlegroups.com.

Blazej Checinski

unread,

Sep 18, 2017, 8:03:21 AM9/18/17

to Confluent Platform

Have the same issue.
Any luck?

Birender Singh

unread,

Oct 31, 2017, 7:25:47 AM10/31/17

to Confluent Platform

Hi,
We are also facing the same issue. Getting the error as {"error_code":500,"message":"Request timed out"}.
As you mentioned for having the broker with id '0'. In our use case we have broker 0,1,2.

The curl command is
curl -X POST -H "Content-Type:application/json" -H "Accept:application/json" --data '{"name":"test-source-mssql-jdbc", "config":{"connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max":"1", "connection.url":"jdbc:sqlserver://IP:1433;databaseName=xx;user=aa;password=aa@123", "table.whitelist":"", "mode":"bulk", "poll.interval.ms":"10000000", "topic.prefix":"D-"}}' http://ip:8083/connectors?forward=true.

Any other configuration needs to change..?

Regards
Birender

Hien Long

unread,

Jul 11, 2018, 5:27:24 PM7/11/18

to Confluent Platform

I have the same issue too. I'm using Confluent 4.0.0. It keeps giving me 500 Timed out error while running curl localhost:8083/connectors.

Is there any known solution that can fixes this problem?

Thanks

ajay kumar

unread,

Jul 20, 2018, 8:22:31 AM7/20/18

to Confluent Platform

when i try to post to the connectors, i get the below error message .GET request works fine

<html>

<head>

<title>Error 500 </title>

</head>

<body>

<h2>HTTP ERROR: 500</h2>

<p>Problem accessing /connectors. Reason:

<pre> Request failed.</pre>

Phu Nguyen

unread,

Dec 25, 2018, 5:45:54 AM12/25/18

to Confluent Platform

Error "Request timed out" comes from many reasons, one of which is my custom plugin (connector, transform, converter,...) contains some jars dependency which belong to connect (such as connect-api, connect-json, connect-transform,...), when connect startup, it'll load them and may be caused a problem "freeze" state. You may exclude it when build a uber jar.

Reply all

Reply to author

Forward