Kafka Exception on Log cleanup

[Ramandeep Sandhu]

Hello,

We are using Kafka on Windows platform. We have observed that whenever the system tries to clean up the logs as per the retention policy - it fails with an exception that requires Kafka to be restarted after deleting the logs manually.

Here is exception stack-trace: 

java.nio.file.FileSystemException: C:\kafka\__consumer_offsets-22\00000000000000000000.log.cleaned -> C:\kafka\__consumer_offsets-22\00000000000000000000.log.swap: The process cannot access the file because it is being used by another process.

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
        at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697)
        at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212)
        at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415)
        at kafka.log.Log.replaceSegments(Log.scala:1644)
        at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:535)
        at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:462)
        at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:461)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at kafka.log.Cleaner.doClean(LogCleaner.scala:461)
        at kafka.log.Cleaner.clean(LogCleaner.scala:438)
        at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:305)
        at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:291)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
        Suppressed: java.nio.file.FileSystemException: C:\kafka\__consumer_offsets-22\00000000000000000000.log.cleaned -> C:\kafka\__consumer_offsets-22\00000000000000000000.log.swap: The process cannot access the file because it is being used by another process.

                at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
                at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
                at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
                at java.nio.file.Files.move(Files.java:1395)
                at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
                ... 12 more

Anyone else facing this issue? Is there a workaround for this ?

Thanks in advance.

[14196...@qq.com]

I had exactly the same issue

在 2018年9月7日星期五 UTC+8下午12:18:31，Ramandeep Sandhu写道：

[Ramón Jansen]

Hi,

I was using Kafka 2.0 on windows SO, and after two weeks I moved to Linux SO, that's because Kafka doesn't fit well in other SO that its not Linux(I highly advice you to move to any Linux SO). 

I got that issue on Linux and killed the Kafka service that i had underground because the console where I was running it, died. Review if you have any underground process of Kafka. Anyway i had problems in Windows, because the directory where I had the logs, doesn't allow Kafka process to delete files(for log compaction, for example) and after X amount of data, I had to clean all the data(zookeeper also) to start again the server.

If you don't worry about your local information, try that, clean all your Kafka data.

Hope it helps.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c5cd4023-799e-4a1d-b9dc-34bdfacd0b1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[14196...@qq.com]

A PR with some fixes is: https://github.com/apache/kafka/pull/3283 which I see some fixes about log bugs in windows,but somehow it is not released.

Thanks for your tips,it's really helpful. 

在 2018年10月9日星期二 UTC+8下午3:34:14，Ramón Jansen写道：

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.