Control Center is not starting up

2,553 views
Skip to first unread message

singh.sh...@gmail.com

unread,
Sep 27, 2017, 5:20:49 PM9/27/17
to Confluent Platform
Hello All,

suddenly, Control Center is not starting up ..

The error it shows -

[2017-09-27 21:12:26,400] INFO unable to get command store (io.confluent.command.CommandStore:86)
org.apache.kafka.streams.errors.InvalidStateStoreException: the state store, commander, may have migrated to another instance.
 at org.apache.kafka.streams.state.internals.StreamThreadStateStoreProvider.stores(StreamThreadStateStoreProvider.java:42)
 at org.apache.kafka.streams.state.internals.QueryableStoreProvider.getStore(QueryableStoreProvider.java:55)
 at org.apache.kafka.streams.KafkaStreams.store(KafkaStreams.java:709)
 at io.confluent.command.CommandStore.getStore(CommandStore.java:141)
 at io.confluent.command.CommandStore.start(CommandStore.java:84)
 at io.confluent.controlcenter.ControlCenter.main(ControlCenter.java:117)
[2017-09-27 21:12:27,400] INFO unable to get command store (io.confluent.command.CommandStore:86)
org.apache.kafka.streams.errors.InvalidStateStoreException: the state store, commander, may have migrated to another instance.
 at org.apache.kafka.streams.state.internals.StreamThreadStateStoreProvider.stores(StreamThreadStateStoreProvider.java:42)
 at org.apache.kafka.streams.state.internals.QueryableStoreProvider.getStore(QueryableStoreProvider.java:55)
 at org.apache.kafka.streams.KafkaStreams.store(KafkaStreams.java:709)
 at io.confluent.command.CommandStore.getStore(CommandStore.java:141)
 at io.confluent.command.CommandStore.start(CommandStore.java:84)
 at io.confluent.controlcenter.ControlCenter.main(ControlCenter.java:117) 

[2017-09-27 18:47:25,338] WARN Could not create task 1_0 due to task [1_0] Failed to lock the state directory for task 1_0. Will retry. (org.apache.kafka.streams.processor.internals.StreamThread:1217)
[2017-09-27 18:47:25,338] DEBUG stream-thread [StreamThread-2] creating new task 9_8 (org.apache.kafka.streams.processor.internals.StreamThread:1240)
 

This was woking earlier, but it restarted happening after i restarted the brokers. 
Any ideas why has caused this ?
What needs to be done to fix this ?



singh.sh...@gmail.com

unread,
Sep 27, 2017, 7:21:51 PM9/27/17
to Confluent Platform

attaching the log file 
any ideas on this .. i'm stuck for now :( 
ControlCenterNotStartingUp.txt

sham singh

unread,
Sep 27, 2017, 7:57:36 PM9/27/17
to confluent...@googlegroups.com
fyi. this is Confluent 3.2.2

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/msGWEEQjkro/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/8175c113-889c-454b-b153-68e09999cceb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Roger Hoover

unread,
Sep 27, 2017, 8:14:22 PM9/27/17
to confluent...@googlegroups.com
That's a race-condition in Kafka Streams that I think has been fixed in later versions.  You can either

1. Keep trying with 3.2.2 and it should start up after a few tries
2. Upgrade to 3.3.0

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/8175c113-889c-454b-b153-68e09999cceb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAFcRHREH1Yfto29CX_wc8nTf%3DPHFA4ADpnHwy6%3DJJP1tGx8UOA%40mail.gmail.com.

singh.sh...@gmail.com

unread,
Sep 28, 2017, 3:16:02 PM9/28/17
to Confluent Platform
Roger - thanks for the response on this ..

I've faced this issue earlier, but on restart the control center, it has got resolved ..
However, this time it is not starting up after multiple data resets & services restarts..

I've tried multiple things --

1) changing the - confluent.controlcenter.id

control-center.properties file has configuration ->
confluent.controlcenter.streams.consumer.auto.offset.reset=latest

2) also, i've cleared the zookeeper (by loggin into zookeeper & deleting directories), 
changed the data directory of the control-center, brokers & zookeeper - and restarted the services multiple times.

3) I've tried doing a fresh re-start - deleting the data in the control-center, brokers & zookeeper - and restarted services multiple times.
(amounts to the same as in pt 2)

4) Reset the ControlCenter/RocksDB using script -> bin/control-center-reset <property file> 

Pls. note - i can afford to do the above on this dev cluster :) 

However, Control center seems have cached the KafkaStreams state store (possibly ?),
and deleting the data & restarting services is not resetting it either.

How do i refresh the state store (to get around this problem of ) ? 

P.S. - i was testing the functionality of Control center managing multiple clusters when i did a restart for a different problem
- Control Center was CACHING/showing the clusterId which i'd deleted (in zookeeper)

This is when i started having this persistent problem.

(either it says 
- there is a lock issue 
[2017-09-27 18:47:25,338] WARN Could not create task 1_0 due to task [1_0] Failed to lock the state directory for task 1_0. Will retry. (org.apache.kafka.streams.processor.internals.StreamThread:1217)
[2017-09-27 18:47:25,338] DEBUG stream-thread [StreamThread-2] creating new task 9_8 (org.apache.kafka.streams.processor.internals.StreamThread:1240)
OR 
gives InvalidStateStoreException
- org.apache.kafka.streams.errors.InvalidStateStoreException: the state store, commander, may have migrated to another instance.
at org.apache.kafka.streams.state.internals.StreamThreadStateStoreProvider.stores(StreamThreadStateStoreProvider.java:42)
 at org.apache.kafka.streams.state.internals.QueryableStoreProvider.getStore(QueryableStoreProvider.java:55)
 at org.apache.kafka.streams.KafkaStreams.store(KafkaStreams.java:709)
 at io.confluent.command.CommandStore.getStore(CommandStore.java:141)
 at io.confluent.command.CommandStore.start(CommandStore.java:84)
 at io.confluent.controlcenter.ControlCenter.main(ControlCenter.java:117)
)

Appreciate help on this !
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

singh.sh...@gmail.com

unread,
Sep 28, 2017, 3:20:12 PM9/28/17
to Confluent Platform

pls note. - i'm not really using KafkaStreams(API) now .. though i'd run a couple of sample programs earlier.

singh.sh...@gmail.com

unread,
Sep 29, 2017, 6:00:05 PM9/29/17
to Confluent Platform

update - i was finally able to reset & restart the cluster & control center .. i changed the ports & the data dir etc
However, the questions wrt Control Center not starting up still remain.  

Gwen Shapira

unread,
Sep 29, 2017, 6:20:04 PM9/29/17
to Confluent Platform
Maybe upgrade to 3.3.0? We did not encounter the race condition on that version. You can upgrade Control Center to 3.3.0 and keep the brokers on 3.2.x version.

Gwen

To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4a99c488-18a8-4b92-99ef-c285de2364fd%40googlegroups.com.

Gwen Shapira

unread,
Sep 29, 2017, 6:21:43 PM9/29/17
to Confluent Platform
Regarding the "control center was caching" - Control center does not get data from ZK at all (!) and will continue showing any cluster it saw in the last 3 years (since it has data for those).

I usually rename clusters (In C3, under "cluster management") to indicate that they are no longer used.

Gwen

Nishanth S

unread,
Dec 19, 2017, 1:39:19 PM12/19/17
to Confluent Platform
All,
I am running into this error with   confluent 4.0 . We were at 3.3 and then upgraded  brokers as well CC  to 4.0 . It  has been working ever since  but failing on restart .Appreciate your help. 
java.lang.IllegalArgumentException: Illegal value provided for SubCode.
        at org.rocksdb.Status$SubCode.getSubCode(Status.java:109)
        at org.rocksdb.Status.<init>(Status.java:30)
        at org.rocksdb.RocksDB.open(Native Method)
        at org.rocksdb.RocksDB.open(RocksDB.java:231)
        at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:197)
        at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:174)
        at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:185)
        at org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:59)
        at org.apache.kafka.streams.state.internals.InnerMeteredKeyValueStore.init(InnerMeteredKeyValueStore.java:160)
        at org.apache.kafka.streams.state.internals.MeteredKeyValueBytesStore.init(MeteredKeyValueBytesStore.java:104)
        at org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:224)
        at org.apache.kafka.streams.processor.internals.StreamTask.initialize(StreamTask.java:163)
        at org.apache.kafka.streams.processor.internals.AssignedTasks.initializeNewTasks(AssignedTasks.java:122)
        at org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:260)
        at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:813)
        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
[2017-12-19 18:38:24,736] INFO [main] unable to get command store (io.confluent.command.CommandStore)
[2017-12-19 18:38:25,737] INFO [main] unable to get command store (io.confluent.command.CommandStore)
[2017-12-19 18:38:26,738] INFO [main] unable to get command store (io.confluent.command.CommandStore)
[2017-12-19 18:38:27,738] INFO [main] unable to get command store (io.confluent.command.CommandStore)
[2017-12-19 18:38:28,739] INFO [main] unable to get command store (io.confluent.command.CommandStore)
[2017-12-19 18:38:29,740] INFO [main] unable to get command store (io.confluent.command.CommandStore)
[2017-12-19 18:38:30,742] INFO [main] unable to get command store (io.confluent.command.CommandStore)
[2017-12-19 18:38:31,743] INFO [main] unable to get command store (io.confluent.command.CommandStore)
[2017-12-19 18:38:32,743] INFO [main] unable to get command store (io.confluent.command.CommandStore)

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

dan

unread,
Dec 20, 2017, 12:09:44 PM12/20/17
to confluent...@googlegroups.com
Nishanth,

can you check to see if your disk is full?

thanks
dan

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/3488a359-be2b-4b28-9973-c01252247063%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages