StreamThread: Error while fetching metadata

2,642 views
Skip to first unread message

Tianxiang Xiong

unread,
Mar 1, 2017, 3:05:44 PM3/1/17
to Confluent Platform
Using Kafka Streams 0.10.1.0, I'm seeing an awful lot of the following when starting up our Kafka Streams app when running tests w/ a local instance of ZooKeeper/Kafka/Schema Registry:

21:59:00.815 [StreamThread-1] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.816 [StreamThread-7] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=INVALID_TOPIC_EXCEPTION, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.817 [StreamThread-9] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=INVALID_TOPIC_EXCEPTION, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.821 [StreamThread-5] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=INVALID_TOPIC_EXCEPTION, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.823 [StreamThread-6] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.827 [StreamThread-4] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.828 [StreamThread-3] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.840 [StreamThread-2] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.890 [StreamThread-11] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.893 [StreamThread-12] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.898 [StreamThread-10] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.904 [StreamThread-13] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:00.912 [StreamThread-8] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}

This continues for a long time, and the stream thread indices increase:

21:59:28.098 [main] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {ledger-client-error-1=LEADER_NOT_AVAILABLE}
21:59:28.103 [StreamThread-56] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 10 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.118 [StreamThread-48] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 10 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.126 [StreamThread-58] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 10 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.131 [StreamThread-54] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 10 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.136 [StreamThread-60] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 10 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.149 [StreamThread-51] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.170 [StreamThread-59] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 10 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.178 [StreamThread-46] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.179 [StreamThread-47] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.180 [StreamThread-53] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
21:59:28.182 [StreamThread-57] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 10 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.189 [StreamThread-49] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE, ledger-request-1=LEADER_NOT_AVAILABLE}
21:59:28.222 [StreamThread-55] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
21:59:28.236 [StreamThread-52] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}

Eventually, however, the tests pass.

It seems this is due to the producer trying to figure out who's the leader of a given partition and failing. However, I've set the `listeners` broker config to `"PLAINTEXT://localhost:9092"`, where `localhost:9092` is the `bootstrap.servers` config for the producer/broker.

Notes:

- `num.stream.threads` is set to 15; each topic for the app has 15 partitions.

A few questions:

- What is happening w/ the increasing stream thread indices? Are stream threads dying and being restarted?
- Why does this happen for a time, but the tests eventually pass, seemingly indicating that the problem resolves itself?
  - Unfortunately, this seems to happen much longer when running tests on CircleCI, often (though not always) resulting in a timeout.

Tianxiang Xiong

unread,
Mar 1, 2017, 3:08:19 PM3/1/17
to Confluent Platform
On CircleCI, I also see errors like:

19:50:19.210 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] WARN  o.a.zookeeper.server.NIOServerCnxn - caught end of stream exception
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x15a8b6c52a60014, likely client has closed socket
 at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
 at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
 at java.lang.Thread.run(Thread.java:745)


Which I don't see when testing locally.

Matthias J. Sax

unread,
Mar 1, 2017, 5:15:52 PM3/1/17
to confluent...@googlegroups.com
The second one is a known issue:
https://issues.apache.org/jira/browse/KAFKA-4369

Overall, it's only WARN messages. Your app should still work.

Right now I am not sure why the thread count goes up. But StreamThreads
are not automatically recreated when they die. It must be something
different...

I agree that it's annoying. You should upgrade to Kafka 0.10.2 to get
rid of it -- CP 3.2 will be released soon.

Btw: for 0.10.2, you can also only upgrade your Streams app without
upgrading your brokers (even if I would recommend to upgrade brokers, too).


-Matthias
> withcorrelation id 12:{ledger-client-error-1=LEADER_NOT_AVAILABLE}
> withcorrelation id
> 12:{ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
> 21:59:28.182[StreamThread-57]WARN
> o.apache.kafka.clients.NetworkClient-Errorwhilefetching metadata
> withcorrelation id
> 10:{entry-added-1=LEADER_NOT_AVAILABLE,ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE,ledger-request-1=LEADER_NOT_AVAILABLE}
> 21:59:28.189[StreamThread-49]WARN
> o.apache.kafka.clients.NetworkClient-Errorwhilefetching metadata
> withcorrelation id
> 12:{entry-added-1=LEADER_NOT_AVAILABLE,ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE,ledger-request-1=LEADER_NOT_AVAILABLE}
> 21:59:28.222[StreamThread-55]WARN
> o.apache.kafka.clients.NetworkClient-Errorwhilefetching metadata
> withcorrelation id
> 12:{ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
> 21:59:28.236[StreamThread-52]WARN
> o.apache.kafka.clients.NetworkClient-Errorwhilefetching metadata
> withcorrelation id
> 12:{ledger-0.1.0-SNAPSHOT-65cbead8-9586-40b1-bc2c-ee8cfa2d7150-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
> |
>
> Eventually, however, the tests pass.
>
> It seems this is due to
> <http://stackoverflow.com/questions/35788697/leader-not-available-kafka-in-console-producer>
> the producer trying to figure out who's the leader of a given
> partition and failing. However, I've set the `listeners` broker
> config to `"PLAINTEXT://localhost:9092"`, where `localhost:9092` is
> the `bootstrap.servers` config for the producer/broker.
>
> Notes:
>
> - `num.stream.threads` is set to 15; each topic for the app has 15
> partitions.
>
> A few questions:
>
> - What is happening w/ the increasing stream thread indices? Are
> stream threads dying and being restarted?
> - Why does this happen for a time, but the tests eventually pass,
> seemingly indicating that the problem resolves itself?
> - Unfortunately, this seems to happen much longer when running
> tests on CircleCI, often (though not always) resulting in a timeout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/f3da5cc1-11c4-4e2f-9c23-1cee4e475802%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/f3da5cc1-11c4-4e2f-9c23-1cee4e475802%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

signature.asc

Tianxiang Xiong

unread,
Mar 1, 2017, 5:41:29 PM3/1/17
to Confluent Platform
What exactly is a message like this trying to tell us?

21:59:00.817 [StreamThread-9] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {entry-added-1=LEADER_NOT_AVAILABLE, ledger-0.1.0-SNAPSHOT-d99a92e5-42bd-4f77-bd8c-10f71f59ed7e-account-balance-updated-1-repartition=INVALID_TOPIC_EXCEPTION, ledger-request-1=LEADER_NOT_AVAILABLE}

- Topic doesn't exist (yet)?
- Can't connect to ZK?

The frustrating thing is:

- This wasn't happening w/ Kafka 0.10.0.1
- The app seems to work locally despite these warnings, but not in CircleCI, where we see a bunch of messages like these that we don't see locally:

org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x15a8b6c7f020014, likely client has closed socket
 at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)

 at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
 at java.lang.Thread.run(Thread.java:745)
19:55:35.411 [SyncThread:0] ERROR o.a.zookeeper.server.NIOServerCnxn - Unexpected Exception:
java.nio.channels.CancelledKeyException: null
 at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
 at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
 at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
 at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
 at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
 at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:169)

Tianxiang Xiong

unread,
Mar 1, 2017, 6:25:52 PM3/1/17
to Confluent Platform
Some more messages from local testing:

15:20:34.247 [main] WARN  i.c.k.s.storage.KafkaStore - Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic.
15:20:34.943 [StreamThread-2] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-fbbf1312-5e4c-4258-910a-d5a73262e47f-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:20:34.944 [StreamThread-1] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-fbbf1312-5e4c-4258-910a-d5a73262e47f-account-balance-updated-1-repartition=INVALID_TOPIC_EXCEPTION}
15:20:34.947 [StreamThread-3] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-fbbf1312-5e4c-4258-910a-d5a73262e47f-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:20:38.986 [main] WARN  i.c.k.s.storage.KafkaStore - Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic.
15:20:39.832 [main] WARN  o.a.k.c.consumer.ConsumerConfig - The configuration 'zookeeper.connect' was supplied but isn't a known config.
15:20:39.935 [StreamThread-4] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-6a292ffd-2b84-49a9-a5bc-d8035a932072-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:20:39.936 [StreamThread-5] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-6a292ffd-2b84-49a9-a5bc-d8035a932072-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:20:39.937 [StreamThread-6] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-6a292ffd-2b84-49a9-a5bc-d8035a932072-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:20:49.583 [main] WARN  i.c.k.s.storage.KafkaStore - Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic.
15:20:50.066 [main] WARN  o.a.k.c.consumer.ConsumerConfig - The configuration 'zookeeper.connect' was supplied but isn't a known config.
15:20:50.163 [StreamThread-7] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-26f11f14-744d-4e1d-b3dc-3a110c4ab47d-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:20:50.164 [StreamThread-8] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-26f11f14-744d-4e1d-b3dc-3a110c4ab47d-account-balance-updated-1-repartition=INVALID_TOPIC_EXCEPTION}
15:20:50.165 [StreamThread-9] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-26f11f14-744d-4e1d-b3dc-3a110c4ab47d-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
:ledger.specs/ledger-request-value
15:20:56.187 [main] WARN  i.c.k.s.storage.KafkaStore - Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic.
15:20:56.662 [main] WARN  o.a.k.c.consumer.ConsumerConfig - The configuration 'zookeeper.connect' was supplied but isn't a known config.
15:20:56.764 [StreamThread-11] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-c46faea2-badd-4855-ad58-ddaed4093ebb-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:20:56.765 [StreamThread-12] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-c46faea2-badd-4855-ad58-ddaed4093ebb-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:20:56.765 [StreamThread-10] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-c46faea2-badd-4855-ad58-ddaed4093ebb-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:21:03.949 [main] WARN  i.c.k.s.storage.KafkaStore - Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic.
15:21:04.412 [main] WARN  o.a.k.c.consumer.ConsumerConfig - The configuration 'zookeeper.connect' was supplied but isn't a known config.
15:21:04.515 [StreamThread-13] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-590a5057-b4c7-4ffd-bcf7-eb3f169cd0ab-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:21:04.515 [StreamThread-14] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-590a5057-b4c7-4ffd-bcf7-eb3f169cd0ab-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
15:21:04.515 [StreamThread-15] WARN  o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {ledger-0.1.0-SNAPSHOT-590a5057-b4c7-4ffd-bcf7-eb3f169cd0ab-account-balance-updated-1-repartition=LEADER_NOT_AVAILABLE}
In: [:entries] val: () fails spec: :ledger.specs.fields/entry at: [:entries] predicate: :ledger.specs.fields/entry,  Insufficient input


lein test ledger.topology-test
15:21:11.848 [main-SendThread(localhost:2181)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x15a8c2d4e91000c for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
 at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
{:result true, :num-tests 100, :seed 1488410471644, :test-var "balance-deltas-test"}
15:21:11.966 [main-SendThread(localhost:2181)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x15a8c2d4e91000b for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
 at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

The stream thread indices increasing is just strange. Any idea what could be causing it? It looks like something goes wrong and the app gets restarted?

Tianxiang Xiong

unread,
Mar 1, 2017, 8:23:56 PM3/1/17
to Confluent Platform
*Slaps forehead* I think I've figured it out.

We have fixtures for integration tests that spin up and tear down Kafka Streams topologies. If we use 15 threads, we make 15 client connections to ZK per test. By default, ZK 3.4.6 has a `maxClientCnxns` of 60. Since the ZK client is not being closed properly, and we didn't set `maxClientCnxns` to "0" (thereby removing the limit), and we just happened to have 5 integration tests, we were running into a situation in which we're over the connection limit, thereby causing the tests to time out in CI.

Michael Noll

unread,
Mar 2, 2017, 6:04:05 AM3/2/17
to confluent...@googlegroups.com
Glad to hear you figured it out!

FYI: In Kafka 0.10.2 the ZK dependency of Kafka Streams was removed.  That means applications that use the Streams API no longer talk to ZK -- at all, which is even better. ;-)


--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ff76b342-8104-4119-afce-2de5702fcc11%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Michael G. Noll
Product Manager | Confluent
Follow us: Twitter | Blog

Tianxiang Xiong

unread,
Mar 3, 2017, 4:05:44 AM3/3/17
to Confluent Platform
I may have spoken too soon. The ZK connection problems still occur w/ a smaller # of threads (5 instead of 15) and even just running 1 dummy test.

I've created a support ticket, 957, w/ attached logs. We can take it from there.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages