Default retention is set to "forever" but kafka topics went missing after attempting to delete one

[daniel....@parkhub.com]

Hello,

I'm running Confluent Kafka 4.1.0 on Kubernetes and it has worked without many problems. My default configuration for topic retention is set to "forever"(big number) and we set topics that don't need to stick around to shorter retention times manually. 

Yesterday, there was a topic that god some bad data in it and instead of working around it in code, I wanted to simply delete the topic. I did so using the kafka-topics tool and then I got impatient. After checking the list of topics(no more than 10 minutes passed) I noticed the topic was still there so I hopped on Zookeeper, open the shell, and delete the /brokers/topics/<topic> directory along with the /admin/delete_topics/<topic> directory and called it a day. This was with the kafka cluster still up, which was probably a mistake on my part.

After I did this, I went back to the kafka bash shell and listed topics and the topic I wanted to be gone was gone. I figured it did what I wanted it too. 

Unfortunately, after I started a new application that consumes topics that I didn't delete, a brand new consumer group with offset.reset of "beginning" it didn't process any of the topics that I know had data to be reprocessed. Every single time I did before I deleted the topic, those other topics would get reprocessed. This time, nothing. After fumbling around thinking it was a client issue, I finally started checking my brokers. Although those topics are listed, they don't seem to have any data. I created a kafka-console-consumer on the topics my app was consuming and nothing. However, there were other topics whose data was there and processed through the consumer.

I'm trying to figure out how I can verify for sure that my data for those topics is gone. I know I didn't delete all those topics and those logs weren't new so the replication should of been done. They are still listed when I do a "--list" command. 

Is there any way to check to see if the data is truly gone or could there be something else going on?

[Sabri Boukari]

Hi, 

I never though that kafka topic can be retained more than 7 days..

Likely you can check if topic has gone when you go to zookeeper ( prefer to check on leader).

and then : ls /admin

in that way you can verify if there's a topic being deleted or stail.

Good luck,

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/b4dc24c2-6e7e-49f2-b508-ddfa9ebbd2e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Boukari Sabri

Cloud DevOps Specialist

Failure is a Great Teacher

www.boukari-sabri.org

www.finyx.fr

www.fean.fr