Graceful deletion of kafka topic

[Prasad]

Hello,

I am looking to list steps to gracefully delete kafka topics. This is one part to address another issue that I am working on to fix. 

For my testing I need to execute initial snapshot multiple times on same tables. So to avoid duplication of data, I am deleting existing kafka topics. Automatic topic creation is set to true. So next time when connector is deployed, it creates topics again. 

steps followed

1. LIST TOPICS:  kubectl exec --kubeconfig advftr.yaml -n debezium-poc kafka-0 -- /kafka/bin/kafka-topics.sh --list --bootstrap-server kafka-0.kafka.debezium-poc.svc.cluster.local:9092
2. DELETE TOPICS: kubectl exec --kubeconfig advftr.yaml -n debezium-poc kafka-0 -- /kafka/bin/kafka-topics.sh --bootstrap-server kafka-0.kafka.debezium-poc.svc.cluster.local:9092 --delete --topic 'TABLE1,TABLE2,TABLE3' 

Here topic = table name

But when connectors are deployed second time, I notice below errors in kafka server.logs

ERROR [kafka-scheduler-6:Logging@76] - Error while deleting dir for TABLE1-0 in dir /kafka/data/1
java.nio.file.FileSystemException: /kafka/data/1/TABLE1-0.a85d5c4ad62a48d09d22ddfeddfdd164-delete/.nfs00000000146b56a200001328: Device or resource busy

ERROR [kafka-scheduler-6:Logging@76] - Exception while deleting Log(dir=/kafka/data/1/TABLE1-0.a85d5c4ad62a48d09d22ddfeddfdd164-delete, topicId=HpFiTfEgRIewwWu8BQiSEQ, topic=TABLE1, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) in dir /kafka/data/1.
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting dir for TABLE1-0 in dir /kafka/data/1
Caused by: java.nio.file.FileSystemException: /kafka/data/1/TABLE1-0.a85d5c4ad62a48d09d22ddfeddfdd164-delete/.nfs00000000146b56a200001328: Device or resource busy

One suggestion that I find on another forum is to set log.retention.ms to a very low value, say 1000 = 1 sec. Then delete topics. Wait for a while and then revert log.retention.ms to original value. Also plan to set log.retention.check.interval.ms. 

One more suggestion is to recreate topic after deletion. I have automatic topic creation set to true. So is manual topic recreation required? 

Appreciate your help.

Thanks,

-Prasad  

[Prasad]

Note: DB is oracle. Debezium is on latest stable version 2.3. 

Detailed error logs. Somehow I am unable to attach. Hence pasting here. 

2023-07-19 19:10:46,944 - ERROR [kafka-scheduler-6:Logging@76] - Error while deleting dir for TABLE1-0 in dir /kafka/data/1

java.nio.file.FileSystemException: /kafka/data/1/TABLE1-0.a85d5c4ad62a48d09d22ddfeddfdd164-delete/.nfs00000000146b56a200001328: Device or resource busy

at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249)
at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
at java.base/java.nio.file.Files.delete(Files.java:1142)
at org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:878)
at org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:867)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
at org.apache.kafka.common.utils.Utils.delete(Utils.java:867)
at kafka.log.LocalLog.$anonfun$deleteEmptyDir$2(LocalLog.scala:238)
at kafka.log.LocalLog.deleteEmptyDir(LocalLog.scala:786)
at kafka.log.UnifiedLog.$anonfun$delete$2(UnifiedLog.scala:1604)
at kafka.log.UnifiedLog.delete(UnifiedLog.scala:1772)
at kafka.log.LogManager.deleteLogs(LogManager.scala:1031)
at kafka.log.LogManager.$anonfun$deleteLogs$6(LogManager.scala:1045)
at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:116)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-07-19 19:10:47,065 - ERROR [kafka-scheduler-6:Logging@76] - Exception while deleting Log(dir=/kafka/data/1/TABLE1-0.a85d5c4ad62a48d09d22ddfeddfdd164-delete, topicId=HpFiTfEgRIewwWu8BQiSEQ, topic=TABLE1, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) in dir /kafka/data/1.

org.apache.kafka.common.errors.KafkaStorageException: Error while deleting dir for TABLE1-0 in dir /kafka/data/1
Caused by: java.nio.file.FileSystemException: /kafka/data/1/TABLE1-0.a85d5c4ad62a48d09d22ddfeddfdd164-delete/.nfs00000000146b56a200001328: Device or resource busy

at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249)
at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
at java.base/java.nio.file.Files.delete(Files.java:1142)
at org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:878)
at org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:867)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
at org.apache.kafka.common.utils.Utils.delete(Utils.java:867)
at kafka.log.LocalLog.$anonfun$deleteEmptyDir$2(LocalLog.scala:238)
at kafka.log.LocalLog.deleteEmptyDir(LocalLog.scala:786)
at kafka.log.UnifiedLog.$anonfun$delete$2(UnifiedLog.scala:1604)
at kafka.log.UnifiedLog.delete(UnifiedLog.scala:1772)
at kafka.log.LogManager.deleteLogs(LogManager.scala:1031)
at kafka.log.LogManager.$anonfun$deleteLogs$6(LogManager.scala:1045)
at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:116)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-07-19 19:10:47,065 - WARN  [LogDirFailureHandler:Logging@70] - [ReplicaManager broker=1] Stopping serving replicas in dir /kafka/data/1

[Chris Cranford]

Hi,

This looks like you've hit KAFKA-13553 [1].  I must admit this is the first time I've seen this error myself, but I suppose since you're already manually deleting the topics, it would seem reasonable to manually re-create them afterward if that somehow addresses the problem, no?

Thanks,
Chris

[1]: https://issues.apache.org/jira/browse/KAFKA-13353

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/26087107-bc64-4e09-a6ba-3f4d830306c6n%40googlegroups.com.