Timestamp Out Of Scope 에러에 관한 문의

[이정환]

안녕하세요.

시스템 운영중 다음과 같은 에러가 발생하여 문의를 드리고자 합니다.

아래 Query를 실행하면 Worker에서 [Timestamp Out Of Scope] 에러가 발생합니다.

Query의 WHERE 부분의 기간을 1일정도로 짧게 하면 에러가 발생하지 않습니다.

또한 ORDER BY time DESC 부분을 없애고 실행하여도 에러가 발생하지 않습니다.

일정기간 넘게 ORDER BY를 할 경우 발생하는 것 같습니다만 혹 어느 쪽을 살펴 봐야 할지 조언 가능하시면 부탁 드리겠습니다.

----

사용 버젼

Java : Jdk1.8.0_102

Hadoop : 2.7.2

Tajo : 0.11.3

----

Query

SELECT *
  FROM log.log_001
WHERE time >= TO_TIMESTAMP('2016-08-03 09:00:00', 'YYYY-MM-DD HH24:MI:SS')
  AND time < TO_TIMESTAMP('2016-08-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS')
  AND ymd >= '20160802'
  AND ymd < '20160811'
ORDER BY time DESC

(time 칼럼은 timestamp형 이며 ymd로 partition 되어 있습니다.)

---

TajoMaster log

2016-08-10 09:49:09,866 INFO org.apache.tajo.master.QueryInProgress: Try to connect to QueryMaster:hts-0147/192.168.101.157:28093
2016-08-10 09:49:09,871 INFO org.apache.tajo.master.QueryInProgress: Initializing QueryInProgress for QueryID=q_1470788823573_0018
2016-08-10 09:49:09,871 INFO org.apache.tajo.master.scheduler.SimpleScheduler: Running Queries: 1
2016-08-10 09:49:09,871 INFO org.apache.tajo.master.QueryInProgress: Call executeQuery to :hts-0147:28093,q_1470788823573_0018
2016-08-10 09:49:10,678 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.0, queryMaster=hts-0147
2016-08-10 09:49:12,685 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.0, queryMaster=hts-0147
2016-08-10 09:49:14,696 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.0255, queryMaster=hts-0147
2016-08-10 09:49:16,712 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.056, queryMaster=hts-0147
2016-08-10 09:49:18,711 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.119, queryMaster=hts-0147
2016-08-10 09:49:20,717 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.206, queryMaster=hts-0147
2016-08-10 09:49:22,723 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.2605, queryMaster=hts-0147
2016-08-10 09:49:24,723 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.34, queryMaster=hts-0147
2016-08-10 09:49:26,725 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.409, queryMaster=hts-0147
2016-08-10 09:49:28,726 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_RUNNING,progress=0.4995, queryMaster=hts-0147
2016-08-10 09:49:28,859 INFO org.apache.tajo.master.QueryInProgress: Received QueryMaster heartbeat:q_1470788823573_0018,state=QUERY_ERROR,progress=0.5, queryMaster=hts-0147
2016-08-10 09:49:28,859 ERROR org.apache.tajo.master.QueryInProgress: q_1470788823573_0018 is stopped because null
2016-08-10 09:49:28,859 INFO org.apache.tajo.master.QueryManager: Stop QueryInProgress:q_1470788823573_0018
2016-08-10 09:49:28,859 INFO org.apache.tajo.master.QueryInProgress: =========================================================
2016-08-10 09:49:28,859 INFO org.apache.tajo.master.QueryInProgress: Stop query:q_1470788823573_0018

----

TajoWorker log

2016-08-10 09:49:28,845 ERROR org.apache.tajo.querymaster.Stage: Stage (eb_1470788823573_0018_000002) ERROR:
java.lang.RuntimeException: Timestamp Out Of Scope
        at org.apache.tajo.util.datetime.DateTimeUtil.toJulianTimeMeta(DateTimeUtil.java:489)
        at org.apache.tajo.util.datetime.TimeMeta.convertToLocalTime(TimeMeta.java:146)
        at org.apache.tajo.util.datetime.DateTimeUtil.toUserTimezone(DateTimeUtil.java:2109)
        at org.apache.tajo.datum.TimestampDatum.asChars(TimestampDatum.java:134)
        at org.apache.tajo.datum.TimestampDatum.asChars(TimestampDatum.java:153)
        at org.apache.tajo.datum.TimestampDatum.toString(TimestampDatum.java:123)
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at org.apache.tajo.storage.VTuple.toDisplayString(VTuple.java:261)
        at org.apache.tajo.storage.VTuple.toString(VTuple.java:232)
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at org.apache.tajo.storage.TupleRange.toString(TupleRange.java:66)
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at org.apache.tajo.engine.planner.UniformRangePartition.partition(UniformRangePartition.java:155)
        at org.apache.tajo.querymaster.Repartitioner.scheduleRangeShuffledFetches(Repartitioner.java:689)
        at org.apache.tajo.querymaster.Repartitioner.scheduleFragmentsForNonLeafTasks(Repartitioner.java:612)
        at org.apache.tajo.querymaster.Stage$InitAndRequestContainer.schedule(Stage.java:1098)
        at org.apache.tajo.querymaster.Stage$InitAndRequestContainer.access$900(Stage.java:842)
        at org.apache.tajo.querymaster.Stage$InitAndRequestContainer$1.run(Stage.java:869)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
--------