Monitor Interceptor report Invalid timestamp

[万浩然]

Hello

     I found this exception when the  "consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"    was configured . Below is the segment of the log:

      

     ERROR Monitoring Interceptor failed to record message metrics for topic: upload_flow, partition: 4 (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor:146)

java.lang.IllegalArgumentException: Invalid timestamp -1

        at io.confluent.monitoring.clients.interceptor.TimeBuckets.get(TimeBuckets.java:65)

        at io.confluent.monitoring.clients.interceptor.MonitoringTimeBuckets.record(MonitoringTimeBuckets.java:61)

        at io.confluent.monitoring.clients.interceptor.MonitoringInterceptor.recordMessageMetric(MonitoringInterceptor.java:139)

        at io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor.onConsume(MonitoringConsumerInterceptor.java:48)

        at org.apache.kafka.clients.consumer.internals.ConsumerInterceptors.onConsume(ConsumerInterceptors.java:57)

        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:952)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:305)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:222)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:170)

        at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142)

        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)

        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

       In fact I used schema server before transfer records into kafka, Below is the schema for topic upload_flow :

{

    "fields": [

        {

            "name": "id",

            "type": "string"

        },

        {

            "name": "timestamp",

            "type": "long"

        },

        {

            "name": "peer_id",

            "type": "string"

        },

        {

            "name": "upload",

            "type": "long"

        },

        {

            "default": null,

            "name": "duration",

            "type": [

                "null",

                "long"

            ]

        },

        {

            "name": "input_time",

            "type": "long"

        },

        {

            "name": "output_time",

            "type": "long"

        },

        {

            "default": null,

            "name": "play_type",

            "type": [

                "null",

                "string"

            ]

        },

        {

            "name": "public_ip",

            "type": "string"

        }

    ],

    "name": "upload_flow",

    "type": "record"

}

    And This is a record in kafka:

    {"id":"10.103.0.4:770052:987742508","timestamp":1472182628479,"peer_id":"0001000714724E5191DC23AD8065FCD8","upload":0,"duration":{"long":60},"input_time":1472182628479,"output_time":1472182628614,"play_type":null,"public_ip":"*.*.*.*"}

     Because the type of timestamp is limited with long  I cannot image the the Monitor report this java.lang.IllegalArgumentException . 

     Anyone can give some tips?

Regards

[万浩然]

Pardon me, I forget mention this exception is reported by Confluent HDFS connector's log.

[elizabet...@stitchfix.com]

I encountered the same error and am also wondering what to do about it.

[vikas...@zefr.com]

Facing the same problem. I am running Kafka Connect using the cp-docker images.

Tried unsuccessfully to search the source for "io.confluent.monitoring.clients.interceptor" on the net. Any help would be appreciated, thanks!

[Ewen Cheslack-Postava]

What version are you using and what version of clients are producing the data? It looks like you're getting a Kafka record timestamp of -1 (which is different than reporting a timestamp field in your value).

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4b92ad66-30b1-49d6-b3da-22b98cc2f2f9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Elizabeth Bennett]

For me the issue was happening because the source connector I had written was not populating the timestamp field in the source records it was creating and publishing. It's not enough to have a payload with a field called "timestamp", you have to create SourceRecord objects using the constructor that includes the timestamp parameter.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4b92ad66-30b1-49d6-b3da-22b98cc2f2f9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/Si3JVG1Q51o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAE1jLMNRXvMSKeNb8STKzijqzEViapJYWB-6guBwaaC0LM_RZQ%40mail.gmail.com.