Cannot restart Kafka on Windows, "found a corrupted index file" error

[gvdm]

Hi all,

I'm running Kafka 1.0.0 on a Windows environment. The first time I start the Kafka broker everything works fine and the data is written.

Then if I try to restart the Kafka broker the data gets corrupted and I get a bunch of errors like this:

[2018-02-06 11:13:05,180] INFO Loading logs. (kafka.log.LogManager)
[2018-02-06 11:13:05,227] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (D:\kafka\logs\kafka-streams-win-KSTREAM-AGGREGATE-STATE-STORE-0000000040-changelog-0\00000000000000000000.index) has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting D:\kafka\logs\kafka-streams-win-KSTREAM-AGGREGATE-STATE-STORE-0000000040-changelog-0\00000000000000000000.timeindex, D:\kafka\logs\kafka-streams-win-KSTREAM-AGGREGATE-STATE-STORE-0000000040-changelog-0\00000000000000000000.index, and D:\kafka\logs\kafka-streams-win-KSTREAM-AGGREGATE-STATE-STORE-0000000040-changelog-0\00000000000000000000.txnindex and rebuilding index... (kafka.log.Log)
[2018-02-06 11:13:05,227] ERROR Error while loading log dir D:\kafka\logs (kafka.log.LogManager)
java.nio.file.FileSystemException: D:\kafka\logs\kafka-streams-win-KSTREAM-AGGREGATE-STATE-STORE-0000000040-changelog-0\00000000000000000000.timeindex: The process cannot access the file because it is being used by another process.
 at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
 at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
 at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
 at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
 at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
 at java.nio.file.Files.deleteIfExists(Files.java:1165)
 at kafka.log.Log$$anonfun$loadSegmentFiles$3.apply(Log.scala:335)
 at kafka.log.Log$$anonfun$loadSegmentFiles$3.apply(Log.scala:297)
 at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
 at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
 at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
 at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
 at kafka.log.Log.loadSegmentFiles(Log.scala:297)
 at kafka.log.Log.loadSegments(Log.scala:406)
 at kafka.log.Log.<init>(Log.scala:203)
 at kafka.log.Log$.apply(Log.scala:1734)
 at kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:221)
 at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$8$$anonfun$apply$16$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:292)
 at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

In this old bug report https://issues.apache.org/jira/browse/KAFKA-1554, the proposed workaround is to delete the `.index` files from the kafka's log folder. After this, at the restart Kafka rebuilds the index files and the old data is not lost.

[2018-02-06 12:13:24,178] ERROR Could not find offset index file corresponding to log file D:\kafka\logs\__consumer_offsets-13\00000000000000000000.log, rebuilding index... (kafka.log.Log)
[2018-02-06 12:13:24,193] INFO Recovering unflushed segment 0 in log __consumer_offsets-13. (kafka.log.Log)
[2018-02-06 12:13:24,209] INFO Loading producer state from offset 0 for partition __consumer_offsets-13 with message format version 2 (kafka.log.Log)
[2018-02-06 12:13:24,209] INFO Completed load of log __consumer_offsets-13 with 1 log segments, log start offset 0 and log end offset 0 in 31 ms (kafka.log.Log)

This seems a very common problem of Kafka on Windows (see https://www.mail-archive.com/d...@kafka.apache.org/msg82704.html, https://issues.apache.org/jira/browse/KAFKA-6200, https://groups.google.com/forum/#!searchin/confluent-platform/Found$20a$20corrupted$20index$20file|sort:date/confluent-platform/ouzw59gW7gA/H-Ud3hnqAwAJ and https://stackoverflow.com/questions/47577812/kafka-fails-on-start-due-to-topic-not-being-loaded?noredirect=1&lq=1 for example).

On the other hand I've only experienced this issue in Windows environments, as I have one another running Kafka instance on CentOS and I had no such a problem.

I know that Kafka on Windows is not supported officially, but do you know any news regarding this problem? May this issue be caused by a misconfiguration or something not depending directly by Kafka (e.g. OS permissions)?

Thank you

Giulio

[gvdm]

Up