[Kafka Connect] Exception : Offset Storage Writer is already flushing

[Sreejith S]

I am getting below exception from one connector. Connect is running in distributed mode.

[2016-08-26 10:27:13,351] ERROR Invalid call to OffsetStorageWriter flush() while already flushing, the framework should not allow this (org.apache.kafka.connect.storage.OffsetStorageWriter:110)

[2016-08-26 10:27:13,352] ERROR Unhandled exception when committing WorkerSourceTask{id=jmx-source-0}:  (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:115)

org.apache.kafka.connect.errors.ConnectException: OffsetStorageWriter is already flushing

        at org.apache.kafka.connect.storage.OffsetStorageWriter.beginFlush(OffsetStorageWriter.java:112)

        at org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:279)

        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter.commit(SourceTaskOffsetCommitter.java:107)

        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter.access$000(SourceTaskOffsetCommitter.java:44)

        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter$1.run(SourceTaskOffsetCommitter.java:73)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

What would be the problem here ? Any pointers please 

Thanks

Sreejith

[Shikhar Bhushan]

Hi Sreejith,

This looks like a bug as the error message indicates. Do you see this error consistently? Can you share a more complete log, perhaps also with DEBUG level? If you don't want to share it publicly please feel free to email me directly.

Thanks,

Shikhar

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/11e96ec6-004d-44d6-a0f8-ec06df2b4ae0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sreejith S]

Hi Shikhar,

Thank you.

When i tried it with standalone mode , it works , there is no such exception.

I have a question here. Is it require to run connect-distributed in all kafka brokers ? Probably all connector JAR's and proeprties in all kafka brokers ?

Regards,
Srijith

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

[Sreejith S]

Hi Shikhar,

Adding to below note,

I tried by keeping the JAR and properties in all kafka brokers. But Still the exception is there.

And i see similar issues in kafka-connect-jdbc as well , https://github.com/confluentinc/kafka-connect-jdbc/issues/54

Thanks & Regards,

Srijith

[Shikhar Bhushan]

Hi Sreejith,

The periodic commit failures start with an initial NullPointerException when committing offsets. That issue was fixed in Kafka/Connect 0.10.0.1, which will be included in CP 3.0.1. (https://issues.apache.org/jira/browse/KAFKA-3500). I updated the JDBC connector issue you linked to as well, thanks for digging that up.

Best,

Shikhar

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/11e96ec6-004d-44d6-a0f8-ec06df2b4ae0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c5c4206a-e62c-4f9e-8c52-5ed1459023cc%40googlegroups.com.

[Sreejith S]

Thank you Shikhar,  will update my kafka version then.

Regards,
Sreejith

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/11e96ec6-004d-44d6-a0f8-ec06df2b4ae0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c5c4206a-e62c-4f9e-8c52-5ed1459023cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/i-tT-JnbbGY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAPXNJrV4hMDvqF1PXvXGZgRryQGEFmmKdNui0XrFxJgHWL6aYQ%40mail.gmail.com.