1 Host - 2 Workers - HDFS Connector - Task is killed when appending to WAL

[Houman Babai]

I'm trying hdfs connector in distributed mode.  When I have one worker on one machine all is good.  

I'm trying out 2 workers on one machine just to test it out.

When I add the second worker (with a different rest.port) the first worker throws an exception:

[2016-01-18 21:23:03,458] ERROR Task is being killed and will not recover until manually restarted: (org.apache.kafka.connect.runtime.WorkerSinkTask:303)
java.lang.ArrayIndexOutOfBoundsException: 278
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:76)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
at io.confluent.connect.hdfs.wal.WALFile$Writer.append(WALFile.java:289)
at io.confluent.connect.hdfs.wal.WALFile$Writer.append(WALFile.java:270)
at io.confluent.connect.hdfs.wal.FSWAL.append(FSWAL.java:60)
at io.confluent.connect.hdfs.TopicPartitionWriter.beginAppend(TopicPartitionWriter.java:540)
at io.confluent.connect.hdfs.TopicPartitionWriter.appendToWAL(TopicPartitionWriter.java:531)
at io.confluent.connect.hdfs.TopicPartitionWriter.write(TopicPartitionWriter.java:280)
at io.confluent.connect.hdfs.DataWriter.write(DataWriter.java:234)
at io.confluent.connect.hdfs.HdfsSinkTask.put(HdfsSinkTask.java:90)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:280)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:176)
at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
[2016-01-18 21:23:03,459] ERROR Thread WorkerSinkTask-hdfs-sink-connector-0 exiting with uncaught exception:  (org.apache.kafka.connect.util.ShutdownableThread:84)
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:304)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:176)
at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
Exception in thread "WorkerSinkTask-hdfs-sink-connector-0" org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:304)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:176)
at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)

I'm using the same property file for both workers, except for the second worker I use a rest.port=8084 (both are based on the packaged property file in folder: etc/schema-registry).  Please see attached.

Thanks in advance!