Kafka REST Proxy dies few seconds into start

[Stuart Wong]

Hi,

My KafkA REST Proxy (CP 2.0.0) in AWS keeps dieing seconds into starting up per the log extract below:

2016-01-15 17:32:07,396 INFO  (io.confluent.kafkarest.KafkaRestConfig:135) KafkaRestConfig values:

simpleconsumer.pool.timeout.ms = 2000

metric.reporters = []

debug = true

producer.threads = 5

shutdown.graceful.ms = 2000

consumer.iterator.timeout.ms = 1

response.mediatype.preferred = [application/vnd.kafka.v1+json, application/vnd.kafka+json, application/json]

consumer.request.timeout.ms = 5000

response.mediatype.default = application/vnd.kafka.v1+json

consumer.instance.timeout.ms = 300000

schema.registry.url = http://schema-registry.datahub.internal:8081

metrics.jmx.prefix = kafka.rest

consumer.iterator.backoff.ms = 50

access.control.allow.origin =

zookeeper.connect = zk.datahub.internal:2181/kafka/prd

port = 8082

request.logger.name = io.confluent.rest-utils.requests

metrics.sample.window.ms = 30000

id = kafka-rest-2

host.name = kafka-rest-2.datahub.internal

simpleconsumer.pool.size.max = 25

consumer.request.max.bytes = 67108864

metrics.num.samples = 2

consumer.threads = 1

2016-01-15 17:32:07,638 DEBUG (org.I0Itec.zkclient.ZkConnection:66) Creating new ZookKeeper instance to connect to zk.datahub.internal:2181/kafka/prd.

2016-01-15 17:32:07,641 INFO  (org.I0Itec.zkclient.ZkEventThread:64) Starting ZkClient event thread.

2016-01-15 17:32:07,646 INFO  (org.apache.zookeeper.ZooKeeper:98) Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT

....

2016-01-15 17:32:07,652 INFO  (org.apache.zookeeper.ZooKeeper:433) Initiating client connection, connectString=zk.datahub.internal:2181/kafka/prd sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@25b485ba

2016-01-15 17:32:07,670 DEBUG (org.apache.zookeeper.ClientCnxn:99) zookeeper.disableAutoWatchReset is false

2016-01-15 17:32:07,680 DEBUG (org.I0Itec.zkclient.ZkClient:1118) Awaiting connection to Zookeeper server

2016-01-15 17:32:07,680 INFO  (org.apache.zookeeper.ClientCnxn:933) Opening socket connection to server /10.50.4.145:2181

2016-01-15 17:32:07,680 DEBUG (org.I0Itec.zkclient.ZkClient:832) Waiting for keeper state SyncConnected

2016-01-15 17:32:07,686 INFO  (org.apache.zookeeper.client.ZooKeeperSaslClient:125) Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.

2016-01-15 17:32:07,745 INFO  (org.apache.zookeeper.ClientCnxn:846) Socket connection established to ip-10-50-4-145.ec2.internal/10.50.4.145:2181, initiating session

2016-01-15 17:32:07,747 DEBUG (org.apache.zookeeper.ClientCnxn:886) Session establishment request sent on ip-10-50-4-145.ec2.internal/10.50.4.145:2181

2016-01-15 17:32:07,757 INFO  (org.apache.zookeeper.ClientCnxn:1175) Session establishment complete on server ip-10-50-4-145.ec2.internal/10.50.4.145:2181, sessionid = 0x5523bc02b4e0008, negotiated timeout = 30000

2016-01-15 17:32:07,758 DEBUG (org.I0Itec.zkclient.ZkClient:543) Received event: WatchedEvent state:SyncConnected type:None path:null

2016-01-15 17:32:07,758 INFO  (org.I0Itec.zkclient.ZkClient:641) zookeeper state changed (SyncConnected)

2016-01-15 17:32:07,759 DEBUG (org.I0Itec.zkclient.ZkClient:587) Leaving process event

2016-01-15 17:32:07,759 DEBUG (org.I0Itec.zkclient.ZkClient:842) State is SyncConnected

2016-01-15 17:32:07,895 DEBUG (org.apache.zookeeper.ClientCnxn:812) Reading reply sessionid:0x5523bc02b4e0008, packet:: clientPath:null serverPath:null finished:false header:: 1,8  replyHeader:: 1,4294967525,0  request:: '/kafka/prd/brokers/ids,F  response:: v{}

2016-01-15 17:32:07,919 INFO  (org.apache.kafka.clients.producer.ProducerConfig:165) ProducerConfig values:

compression.type = none

metric.reporters = []

metadata.max.age.ms = 300000

metadata.fetch.timeout.ms = 60000

reconnect.backoff.ms = 50

sasl.kerberos.ticket.renew.window.factor = 0.8

bootstrap.servers = []

retry.backoff.ms = 100

sasl.kerberos.kinit.cmd = /usr/bin/kinit

buffer.memory = 33554432

timeout.ms = 30000

key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer

sasl.kerberos.service.name = null

sasl.kerberos.ticket.renew.jitter = 0.05

ssl.keystore.type = JKS

ssl.trustmanager.algorithm = PKIX

block.on.buffer.full = false

ssl.key.password = null

max.block.ms = 60000

sasl.kerberos.min.time.before.relogin = 60000

connections.max.idle.ms = 540000

ssl.truststore.password = null

max.in.flight.requests.per.connection = 5

metrics.num.samples = 2

client.id =

ssl.endpoint.identification.algorithm = null

ssl.protocol = TLS

request.timeout.ms = 30000

ssl.provider = null

ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

acks = 1

batch.size = 16384

ssl.keystore.location = null

receive.buffer.bytes = 32768

ssl.cipher.suites = null

ssl.truststore.type = JKS

security.protocol = PLAINTEXT

retries = 0

max.request.size = 1048576

value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer

ssl.truststore.location = null

ssl.keystore.password = null

ssl.keymanager.algorithm = SunX509

metrics.sample.window.ms = 30000

partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner

send.buffer.bytes = 131072

linger.ms = 0

2016-01-15 17:32:07,928 DEBUG (org.apache.kafka.common.metrics.Metrics:201) Added sensor with name bufferpool-wait-time

2016-01-15 17:32:07,932 DEBUG (org.apache.kafka.common.metrics.Metrics:201) Added sensor with name buffer-exhausted-records

2016-01-15 17:32:07,933 INFO  (org.apache.kafka.clients.producer.KafkaProducer:613) Closing the Kafka producer with timeoutMillis = 0 ms.

2016-01-15 17:32:07,935 DEBUG (org.apache.kafka.clients.producer.KafkaProducer:654) The Kafka producer has closed.

2016-01-15 17:32:07,935 ERROR (io.confluent.kafkarest.KafkaRestMain:44) Server died unexpectedly: org.apache.kafka.common.KafkaException: Failed to construct kafka producer

I'm not seeing why it's getting a "timeoutMillis = 0 ms" much less where to fix this. Any ideas, thoughts?

Thanks.

[sneha george]

I am running into this issue too, how was it resolved?

[Stuart Wong]

Hi,

I do apologize for not posting the resolution, and for not remembering currently. It was most likely something silly which I was doing/missing in the configs so I'd check that and ensure everything is as it should be (definitely security groups as well). I'll see if I can dig around some more as to the issue and resolution.

- Regards,