Error in Kafka Streams using kafka-node - negative timestamp

[Martin Tyler]

I am trying to add messages to a topic using kafka-node - https://github.com/SOHU-Co/kafka-node

This seems pretty easy, and using the console consumer the message comes through as expected.

However, in my Kafka Streams application I get an exception receiving the message before it gets to my code.

Exception in thread "StreamThread-7" org.apache.kafka.streams.errors.StreamsException: Extracted timestamp value is negative, which is not allowed.

        at org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:90)

        at org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:117)

        at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:138)

        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:304)

        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:218)

This check for negative timestamps appears to have been added in May 2016 - https://github.com/apache/kafka/commit/53fd22a76613b309b7941a5b0c64f17523b39202

Is kafka-node out of date and sending incompatible messages, or is there something I might be doing wrong?

[Andrew Otto]

kafka-node might be out of date.  I’d suggest trying

  https://github.com/Blizzard/node-rdkafka

or perhaps

  https://github.com/apparatus/Kafkaesque

although I haven’t tried this one.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/49e9ea06-371d-4d6d-b355-e6a4587dae9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthias J. Sax]

In Kafka Streams, all records must have a valid timestamp. If you
process data from a topic that was written prior to 0.10, those messages
do not have an embedded message timestamp (which is the default
timestamp used by Kafka Streams).

To fix this issue, you need to provide a different timestamp extractor, via

streamsConfig(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG,
"package.and.name.of.extractor");

If you messaged have a useful timestamp in the data themselves, you
should provide a custom extractor. If your data does not have a useful
timestamp, you can fall back to WallclockTimestampExtractor (however,
getting different time semantic with this)

See http://docs.confluent.io/3.0.1/streams/concepts.html#time for
further details.


-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/49e9ea06-371d-4d6d-b355-e6a4587dae9e%40googlegroups.com

> <https://groups.google.com/d/msgid/confluent-platform/49e9ea06-371d-4d6d-b355-e6a4587dae9e%40googlegroups.com?utm_medium=email&utm_source=footer>.