Checksum error on a partition

[Marek Dolgos]

Has anyone seen this error, or know how to resolve this?

I checked the past posts and it does not seem to be that common, but I'm not sure of the fix.

The error:

topic=raw_events partition=0 leaderId=10881778 server= service= beginOffset=204569175 offset=204569176 server= checksum=2986914431 time=1397179213328

java.lang.Exception: Invalid message checksum 649153691. Expected 2986914431

at com.linkedin.camus.etl.kafka.mapred.EtlRecordReader.nextKeyValue(EtlRecordReader.java:247)

at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)

...

Caused by: org.apache.hadoop.fs.ChecksumException: Invalid message checksum 649153691. Expected 2986914431

... 12 more

Any info appreciated.

-Marek

[Gaurav Gupta]

This is the file you need to look at :

https://raw.githubusercontent.com/linkedin/camus/master/camus-etl-kafka/src/main/java/com/linkedin/camus/etl/kafka/mapred/EtlRecordReader.java

---

 

byte[] bytes = getBytes(msgValue);

                    byte[] keyBytes = getBytes(msgKey);

                    // check the checksum of message.

                    // If message has partition key, need to construct it with Key for checkSum to match

                    Message messageWithKey = new Message(bytes,keyBytes);

                    Message messageWithoutKey = new Message(bytes);

                    long checksum = key.getChecksum();

                    if (checksum != messageWithKey.checksum() && checksum != messageWithoutKey.checksum()) {

                    throw new ChecksumException("Invalid message checksum : MessageWithKey : "

                              + messageWithKey.checksum() + " MessageWithoutKey checksum : " 

                     + messageWithoutKey.checksum() 

                     + ". Expected " + key.getChecksum(),

                     key.getOffset());

                    }

—

The code will handle the case when the partitionId is encoded along with the message when you are sending it to Kafka.

Not sure if you are passing on any more details.

In either case you need to look at how you are encoding your messages.

Gaurav

--
You received this message because you are subscribed to the Google Groups "Camus - Kafka ETL for Hadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camus_etl+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.