Realtime Node: "The end instant must be greater or equal to the start" error

[Alexander Makarenko]

Hello, guys!

Using Druid 0.7.0.

I rarely got these errors in Realtime nodes logs:

2015-03-23T06:15:00,020 ERROR [earthwise0-2015-03-08T04:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[d018]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.lang.IllegalArgumentException, exceptionMessage=The end instant must be greater or equal to the start, interval=2015-03-08T04:00:00.000Z/2015-03-08T05:00:00.000Z}
java.lang.IllegalArgumentException: The end instant must be greater or equal to the start
        at org.joda.time.base.AbstractInterval.checkInterval(AbstractInterval.java:63) ~[joda-time-2.6.jar:2.6]
        at org.joda.time.base.BaseInterval.<init>(BaseInterval.java:94) ~[joda-time-2.6.jar:2.6]
        at org.joda.time.Interval.<init>(Interval.java:122) ~[joda-time-2.6.jar:2.6]
        at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:466) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.merge(IndexMerger.java:319) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:206) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:199) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:438) [druid-server-0.7.0.jar:0.7.0]
        at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:40) [druid-common-0.7.0.jar:0.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]

 

2015-03-23T06:15:00,021 ERROR [d018-2015-03-08T03:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[d018]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.lang.IllegalArgumentException, exceptionMessage=The end instant must be greater or equal to the start, interval=2015-03-08T03:00:00.000Z/2015-03-08T04:00:00.000Z}
java.lang.IllegalArgumentException: The end instant must be greater or equal to the start
        at org.joda.time.base.AbstractInterval.checkInterval(AbstractInterval.java:63) ~[joda-time-2.6.jar:2.6]
        at org.joda.time.base.BaseInterval.<init>(BaseInterval.java:94) ~[joda-time-2.6.jar:2.6]
        at org.joda.time.Interval.<init>(Interval.java:122) ~[joda-time-2.6.jar:2.6]
        at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:466) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.merge(IndexMerger.java:319) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:206) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:199) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:438) [druid-server-0.7.0.jar:0.7.0]
        at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:40) [druid-common-0.7.0.jar:0.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]

 

2015-03-23T06:15:02,586 ERROR [d018-2015-03-08T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[d018]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.lang.IllegalArgumentException, exceptionMessage=The end instant must be greater or equal to the start, interval=2015-03-08T06:00:00.000Z/2015-03-08T07:00:00.000Z}
java.lang.IllegalArgumentException: The end instant must be greater or equal to the start
        at org.joda.time.base.AbstractInterval.checkInterval(AbstractInterval.java:63) ~[joda-time-2.6.jar:2.6]
        at org.joda.time.base.BaseInterval.<init>(BaseInterval.java:94) ~[joda-time-2.6.jar:2.6]
        at org.joda.time.Interval.<init>(Interval.java:122) ~[joda-time-2.6.jar:2.6]
        at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:466) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.merge(IndexMerger.java:319) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:206) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:199) ~[druid-processing-0.7.0.jar:0.7.0]
        at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:438) [druid-server-0.7.0.jar:0.7.0]
        at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:40) [druid-common-0.7.0.jar:0.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]

But segments mentioned in errors are still queryable.

At this moment Realtime node stopped serving these segments (after I added rules for Historical Nodes to load).

Just wondering what may be a cause of such errors?

[Fangjin Yang]

Hi Alexander,

If you look at the interval for 2015-03-08T06:00:00.000Z/2015-03-08T07:00:00.000Z, are there empty directories there? Do you have any exceptions while intermediate persists are occurring?