time partition result with 1 second deviation

21 views
Skip to first unread message

Miao Liu

unread,
Jul 10, 2015, 4:00:21 AM7/10/15
to camu...@googlegroups.com
I have a customize MessageDecoder.

Message format: 2 fields. The first field is timestamp. As following:
yyMMddHHmmss<table>aaaaaaa

the code like this:

public class StringMessageDecoder extends MessageDecoder<Message, String> {
       ....................

        try {
            String[] tokens = payloadString.split("\t");
            SimpleDateFormat sdf = new SimpleDateFormat("yyMMddHHmmss");
            Date dt = sdf.parse(tokens[0]);
            timestamp = dt.getTime();
        } catch (ParseException e) {
            log.error("Unable to parse time", e);
            timestamp = System.currentTimeMillis();
        }

        return new CamusWrapper<String>(payloadString, timestamp);
}


Then, run the camus for houly partition. 
for example, The result is:

all of the records with time(150709040000) were pulled to '2015/07/09/03' partition instead of '2015/07/09/04' partition.
some records with time(150709040001) were pulled to '2015/07/09/03' partition instead of '2015/07/09/04', and others were correctly pulled to '2015/07/09/04' partition.
all of the records with time(150709050000) were pulled to '2015/07/09/04' partition instead of '2015/07/09/05' partition.
others records with time(04:00:02~04:59:59) were correctly pulled to '2015/07/09/04' partition.

there seems to be 1 second deviation...If anyone know why?

Thanks.



Reply all
Reply to author
Forward
0 new messages