WARN || UNKNOWN_TOPIC_OR_PARTITION
1,602 views
Skip to first unread message
Anup Tiwari
Feb 22, 2021, 7:07:52 AM2/22/21
to ksqldb-users
Hi Team,
I had a table "TEST_MC_UDAF_TABLE" whose query(CTAS_TEST_MC_UDAF_TABLE_4565) i terminated and then dropped table with delete topic from one of node(Node1) of KsqlDB cluster(3 node).
But even after multiple hours of termination, i can see below warning in other node of ksqlDB(Node2) :-
WARN [Producer clientId=_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1-producer] Error while fetching metadata with correlation id 11387 : {_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1073)
Note : I am using ksqlDB from confluent-6.0.1 package.
Please find below logs from different servers :-
1. From node(Node1) where i executed terminate / drop-delete command :-
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:23,494] INFO Processed unsuccessfully: KsqlRequest{ksql='DROP TABLE TEST_MC_UDAF_TABLE DELETE TOPIC ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[-1]}, reason: Cannot drop TEST_MC_UDAF_TABLE.
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: The following queries read from this source: [].
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: The following queries write into this source: [CTAS_TEST_MC_UDAF_TABLE_4565].
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: You need to terminate them before dropping TEST_MC_UDAF_TABLE.
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: Statement: DROP TABLE TEST_MC_UDAF_TABLE; (io.confluent.ksql.rest.server.resources.KsqlResource:267)
Feb 21 20:30:33 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:33,270] INFO Received: KsqlRequest{ksql='terminate CTAS_TEST_MC_UDAF_TABLE_4565 ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[-1]} (io.confluent.ksql.rest.server.resources.KsqlResource:223)
Feb 21 20:30:33 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:33,270] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.KafkaStreams:285)
Feb 21 20:30:33 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:33,271] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] Informed to shut down (org.apache.kafka.streams.processor.internals.StreamThread:896)
Feb 21 20:30:33 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:33,271] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread:220)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,141] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] Shutting down (org.apache.kafka.streams.processor.internals.StreamThread:910)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,141] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_1] Suspended running (org.apache.kafka.streams.processor.internals.StreamTask:273)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,141] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_4] Suspended running (org.apache.kafka.streams.processor.internals.StreamTask:273)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,141] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_7] Suspended running (org.apache.kafka.streams.processor.internals.StreamTask:273)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,164] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_7] Closing record collector clean (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:255)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,164] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_7] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask:501)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,164] ERROR KafkaStreams is not running. State is PENDING_SHUTDOWN. (io.confluent.ksql.util.QueryMetadata:182)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,177] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_4] Closing record collector clean (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:255)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,177] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_4] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask:501)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,191] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_1] Closing record collector clean (org.apache.kafka.streams.processor.internals.RecordCollec
torImpl:255)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,191] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_1] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask:501)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,191] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_0] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,211] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_0] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,211] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_2] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,341] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_2] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,341] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_3] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,371] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_3] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,371] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_5] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,390] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_5] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,390] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_6] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,419] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_6] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,419] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_8] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,900] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_8] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,900] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_9] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,918] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_9] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,924] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] State transition from PENDING_SHUTDOWN to DEAD (org.apache.kafka.streams.processor.internals.StreamThread:220)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,925] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] Shutdown complete (org.apache.kafka.streams.processor.internals.StreamThread:935)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,927] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] State transition from PENDING_SHUTDOWN to NOT_RUNNING (org.apache.kafka.streams.KafkaStreams:285)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,928] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Streams client stopped completely (org.apache.kafka.streams.KafkaStreams:994)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,139] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Already in the pending shutdown state, wait to complete shutdown (org.apache.kafka.streams.KafkaStreams:942)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,139] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Streams client stopped completely (org.apache.kafka.streams.KafkaStreams:994)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,143] INFO Executing statement: terminate CTAS_TEST_MC_UDAF_TABLE_4565 ; (io.confluent.ksql.rest.server.computation.CommandRunner:231)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,143] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Already in the pending shutdown state, wait to complete shutdown (org.apache.kafka.streams.KafkaStreams:942)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,143] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Streams client stopped completely (org.apache.kafka.streams.KafkaStreams:994)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,160] INFO Executed statement: terminate CTAS_TEST_MC_UDAF_TABLE_4565 ; (io.confluent.ksql.rest.server.computation.CommandRunner:238)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,161] INFO Processed successfully: KsqlRequest{ksql='terminate CTAS_TEST_MC_UDAF_TABLE_4565 ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[-1]} (io.confluent.ksql.rest.server.resources.KsqlResource:261)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,184] INFO Received: KsqlRequest{ksql='list streams; list tables; list queries;', configOverrides={}, requestProperties={request.ksql.internal.request=true}, commandSequenceNumber=Optional.empty} (io.confluent.ksql.rest.server.resources.KsqlResource:223)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,185] INFO Processed successfully: KsqlRequest{ksql='list streams; list tables; list queries;', configOverrides={}, requestProperties={request.ksql.internal.request=true}, commandSequenceNumber=Optional.empty} (io.confluent.ksql.rest.server.resources.KsqlResource:261)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,200] INFO Received: KsqlRequest{ksql='DROP TABLE TEST_MC_UDAF_TABLE DELETE TOPIC ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[4572]} (io.confluent.ksql.rest.server.resources.KsqlResource:223)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,328] INFO Executed statement: DROP TABLE TEST_MC_UDAF_TABLE; (io.confluent.ksql.rest.server.computation.CommandRunner:238)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,329] INFO Processed successfully: KsqlRequest{ksql='DROP TABLE TEST_MC_UDAF_TABLE DELETE TOPIC ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[4572]} (io.confluent.ksql.rest.server.resources.KsqlResource:261)
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: The following queries read from this source: [].
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: The following queries write into this source: [CTAS_TEST_MC_UDAF_TABLE_4565].
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: You need to terminate them before dropping TEST_MC_UDAF_TABLE.
Feb 21 20:30:23 ip-10-51-*-* ksql-server-start: Statement: DROP TABLE TEST_MC_UDAF_TABLE; (io.confluent.ksql.rest.server.resources.KsqlResource:267)
Feb 21 20:30:33 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:33,270] INFO Received: KsqlRequest{ksql='terminate CTAS_TEST_MC_UDAF_TABLE_4565 ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[-1]} (io.confluent.ksql.rest.server.resources.KsqlResource:223)
Feb 21 20:30:33 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:33,270] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.KafkaStreams:285)
Feb 21 20:30:33 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:33,271] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] Informed to shut down (org.apache.kafka.streams.processor.internals.StreamThread:896)
Feb 21 20:30:33 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:33,271] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread:220)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,141] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] Shutting down (org.apache.kafka.streams.processor.internals.StreamThread:910)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,141] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_1] Suspended running (org.apache.kafka.streams.processor.internals.StreamTask:273)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,141] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_4] Suspended running (org.apache.kafka.streams.processor.internals.StreamTask:273)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,141] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_7] Suspended running (org.apache.kafka.streams.processor.internals.StreamTask:273)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,164] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_7] Closing record collector clean (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:255)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,164] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_7] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask:501)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,164] ERROR KafkaStreams is not running. State is PENDING_SHUTDOWN. (io.confluent.ksql.util.QueryMetadata:182)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,177] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_4] Closing record collector clean (org.apache.kafka.streams.processor.internals.RecordCollectorImpl:255)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,177] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_4] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask:501)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,191] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_1] Closing record collector clean (org.apache.kafka.streams.processor.internals.RecordCollec
torImpl:255)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,191] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] task [0_1] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask:501)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,191] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_0] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,211] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_0] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,211] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_2] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,341] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_2] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,341] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_3] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,371] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_3] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,371] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_5] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,390] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_5] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,390] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_6] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,419] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_6] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,419] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_8] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,900] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_8] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,900] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_9] Suspended running (org.apache.kafka.streams.processor.internals.StandbyTask:129)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,918] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] standby-task [0_9] Closed clean (org.apache.kafka.streams.processor.internals.StandbyTask:223)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,924] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] State transition from PENDING_SHUTDOWN to DEAD (org.apache.kafka.streams.processor.internals.StreamThread:220)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,925] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe-StreamThread-1] Shutdown complete (org.apache.kafka.streams.processor.internals.StreamThread:935)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,927] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] State transition from PENDING_SHUTDOWN to NOT_RUNNING (org.apache.kafka.streams.KafkaStreams:285)
Feb 21 20:30:34 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:34,928] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Streams client stopped completely (org.apache.kafka.streams.KafkaStreams:994)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,139] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Already in the pending shutdown state, wait to complete shutdown (org.apache.kafka.streams.KafkaStreams:942)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,139] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Streams client stopped completely (org.apache.kafka.streams.KafkaStreams:994)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,143] INFO Executing statement: terminate CTAS_TEST_MC_UDAF_TABLE_4565 ; (io.confluent.ksql.rest.server.computation.CommandRunner:231)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,143] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Already in the pending shutdown state, wait to complete shutdown (org.apache.kafka.streams.KafkaStreams:942)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,143] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-fa9f3639-3c09-4be7-92bf-afc5b2aa57fe] Streams client stopped completely (org.apache.kafka.streams.KafkaStreams:994)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,160] INFO Executed statement: terminate CTAS_TEST_MC_UDAF_TABLE_4565 ; (io.confluent.ksql.rest.server.computation.CommandRunner:238)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,161] INFO Processed successfully: KsqlRequest{ksql='terminate CTAS_TEST_MC_UDAF_TABLE_4565 ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[-1]} (io.confluent.ksql.rest.server.resources.KsqlResource:261)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,184] INFO Received: KsqlRequest{ksql='list streams; list tables; list queries;', configOverrides={}, requestProperties={request.ksql.internal.request=true}, commandSequenceNumber=Optional.empty} (io.confluent.ksql.rest.server.resources.KsqlResource:223)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,185] INFO Processed successfully: KsqlRequest{ksql='list streams; list tables; list queries;', configOverrides={}, requestProperties={request.ksql.internal.request=true}, commandSequenceNumber=Optional.empty} (io.confluent.ksql.rest.server.resources.KsqlResource:261)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,200] INFO Received: KsqlRequest{ksql='DROP TABLE TEST_MC_UDAF_TABLE DELETE TOPIC ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[4572]} (io.confluent.ksql.rest.server.resources.KsqlResource:223)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,328] INFO Executed statement: DROP TABLE TEST_MC_UDAF_TABLE; (io.confluent.ksql.rest.server.computation.CommandRunner:238)
Feb 21 20:30:37 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:37,329] INFO Processed successfully: KsqlRequest{ksql='DROP TABLE TEST_MC_UDAF_TABLE DELETE TOPIC ;', configOverrides={}, requestProperties={}, commandSequenceNumber=Optional[4572]} (io.confluent.ksql.rest.server.resources.KsqlResource:261)
2. Logs from Other KsqlDB node(Node2) AT THE TIME OF TERMINATE statement( This node is still giving warnings after several hours) :-
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,143] INFO Executing statement: terminate CTAS_TEST_MC_UDAF_TABLE_4565 ; (io.confluent.ksql.rest.server.computation.CommandRunner:231)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,143] INFO stream-client [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.KafkaStreams:285)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,144] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1] Informed to shut down (org.apache.kafka.streams.processor.internals.StreamThread:896)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,144] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread:220)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,163] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1] Shutting down (org.apache.kafka.streams.processor.internals.StreamThread:910)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,163] INFO stream-thread [_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1] task [0_2] Suspended running (org.apache.kafka.streams.processor.internals.StreamTask:273)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,171] WARN [Producer clientId=_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1-producer] Got error produce response with correlation id 11385 on topic-partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2, retrying (2147483646 attempts left). Error: UNKNOWN_TOPIC_OR_PARTITION (org.apache.kafka.clients.producer.internals.Sender:595)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,171] WARN [Producer clientId=_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1-producer] Got error produce response with correlation id 11385 on topic-partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2, retrying (2147483646 attempts left). Error: UNKNOWN_TOPIC_OR_PARTITION (org.apache.kafka.clients.producer.internals.Sender:595)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,171] WARN [Producer clientId=_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1-producer] Received unknown topic or partition error in produce request on partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2. The topic-partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender:624)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,171] WARN [Producer clientId=_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1-producer] Received unknown topic or partition error in produce request on partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2. The topic-partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender:624)
Feb 21 20:30:35 ip-10-51-*-* ksql-server-start: [2021-02-21 20:30:35,171] WARN [Producer clientId=_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1-producer] Error while fetching metadata with correlation id 11387 : {_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1073)
Feb 21 19:30:09 ip-10-51-*-* ksql-server-start: [2021-02-21 19:30:09,918] WARN [Producer clientId=_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-523fadb2-1253-4f11-9aab-7a26b60204a9-StreamThread-1-producer] delivery.timeout.ms should be equal to or larger than linger.ms + request.timeout.ms. Setting it to 300100. (org.apache.kafka.clients.producer.KafkaProducer:505)
3. Logs from One of Kafka Broker nodes AT THE TIME OF TERMINATE statement :-
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,159] INFO [GroupCoordinator 2]: Removed 0 offsets associated with deleted partitions: _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-1, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-3, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-4, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-5, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-6, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-7, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-8, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-0, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-9. (kafka.coordinator.group.GroupCoordinator)
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,170] ERROR [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error for partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2 at offset 229611 (kafka.server.ReplicaFetcherThread)
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,170] ERROR [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error for partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-8 at offset 321257 (kafka.server.ReplicaFetcherThread)
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,170] ERROR [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error for partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-5 at offset 326272 (kafka.server.ReplicaFetcherThread)
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,176] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions Set(_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-6, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-5, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-4, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-3, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-1, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-0, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-9, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-8, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-7) (kafka.server.ReplicaFetcherManager)
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,170] ERROR [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error for partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2 at offset 229611 (kafka.server.ReplicaFetcherThread)
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,170] ERROR [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error for partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-8 at offset 321257 (kafka.server.ReplicaFetcherThread)
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,170] ERROR [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error for partition _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-5 at offset 326272 (kafka.server.ReplicaFetcherThread)
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
Feb 21 20:30:35 ip-10-*-*-* kafka-server-start.sh: [2021-02-21 20:30:35,176] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions Set(_confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-6, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-5, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-4, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-3, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-2, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-1, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-0, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-9, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-8, _confluent-ksql-test_query_CTAS_TEST_MC_UDAF_TABLE_4565-Aggregate-Aggregate-Materialize-changelog-7) (kafka.server.ReplicaFetcherManager)
Regards,
Anup Tiwari
Anup Tiwari
Anup Tiwari
Feb 24, 2021, 11:57:48 PM2/24/21
to Guozhang Wang, ksqldb-users
Hi Guozhang,
I can't see much progress on those links. So by when can we expect a solution on this issue?
Also do we have any temp fix apart from restarting other servers ? Actually I haven't observed this issue on confluent 5.5 ksqlDB and recently we upgraded to confluent 6.0.1 ksqlDB after which this has become a very frequent issue.
Also can you tell me the impact of this on pull / push queries ?
Regards,
Anup Tiwari
Anup Tiwari
On Tue, Feb 23, 2021 at 6:36 AM Guozhang Wang <guoz...@confluent.io> wrote:
Hello Anup,From the other client logs (2) it seems the terminating process did not complete on time and got stuck. We've seen this issue in old versions:I think this is a known bug even in the latest version that when deleting the source topic / sink topic or internal changelog topic too quickly on the triggering node, other nodes that are getting the termination command a bit late could be taking much longer time to terminate. We are currently working on a fix on this issue.Guozhang
--
You received this message because you are subscribed to the Google Groups "ksqldb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ksql-users+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/ksql-users/CAH8KkAq24wr3a40gQ7B_wQH3sHL0Xdjrm3g7a8FzigQDJAgEFQ%40mail.gmail.com.
--
Reply all
Reply to author
Forward
0 new messages