tajo worker 추가 문의

[권기영]

안녕하세요 타조 0.10.0 버전 사용중인 유저 입니다.

worker 한대를 추가 하려고 하는데 아래와 같이 수행하니 배치 작업 중 에러가 발생해서요

 ㄴ web ui 상으로는 정상 처럼 보이나 배치 수행시 insert 구문에서 오류가 발생함

worker 추가 절차가 잘못된건지 확인 좀 부탁 드리겠습니다. 

* 수행환경

  - 하둡 2.6.0

  - hbase-0.98.13-hadoop2

  - tajo-0.10.0

1. 수행 중인 타조 전체 정지

2. /conf/worker 에 worker 노드 추가

3. 각 노드에 worker 배포

4. tajo 전체 시작

에러로그

======================================================================= =========================================================================

2016-12-07 15:40:16,727 ERROR org.apache.tajo.querymaster.TaskAttempt: ta_1481092630696_0004_000002_000000_00 FROM node4 >> org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2

2016-12-07 15:40:16,727 INFO org.apache.tajo.querymaster.Task: ====================================================================================

2016-12-07 15:40:16,727 INFO org.apache.tajo.querymaster.Task: >>> Task Failed: ta_1481092630696_0004_000002_000000_00, retry:true, attempts:1 <<<

2016-12-07 15:40:16,727 INFO org.apache.tajo.querymaster.Task: ====================================================================================

2016-12-07 15:40:16,879 ERROR org.apache.tajo.querymaster.TaskAttempt: ta_1481092630696_0004_000002_000000_01 FROM node4 >> org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2

2016-12-07 15:40:16,879 INFO org.apache.tajo.querymaster.Task: ====================================================================================

2016-12-07 15:40:16,879 INFO org.apache.tajo.querymaster.Task: >>> Task Failed: ta_1481092630696_0004_000002_000000_01, retry:true, attempts:2 <<<

2016-12-07 15:40:16,879 INFO org.apache.tajo.querymaster.Task: ====================================================================================

2016-12-07 15:40:16,979 ERROR org.apache.tajo.querymaster.TaskAttempt: ta_1481092630696_0004_000002_000000_02 FROM node4 >> org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2

2016-12-07 15:40:16,979 INFO org.apache.tajo.querymaster.Task: ====================================================================================

2016-12-07 15:40:16,979 INFO org.apache.tajo.querymaster.Task: >>> Task Failed: ta_1481092630696_0004_000002_000000_02, retry:false, attempts:3 <<<

2016-12-07 15:40:16,979 INFO org.apache.tajo.querymaster.Task: ====================================================================================

2016-12-07 15:40:16,980 INFO org.apache.tajo.querymaster.DefaultTaskScheduler: TaskScheduler schedulingThread stopped

2016-12-07 15:40:16,980 INFO org.apache.tajo.querymaster.DefaultTaskScheduler: Task Scheduler stopped

2016-12-07 15:40:16,980 INFO org.apache.tajo.querymaster.Stage: Stage completed - eb_1481092630696_0004_000002 (total=1, success=0, killed=0)

2016-12-07 15:40:16,981 INFO org.apache.tajo.querymaster.QueryMaster: cleanup executionBlocks: eb_1481092630696_0004_000001

2016-12-07 15:40:16,983 INFO org.apache.tajo.querymaster.Query: Processing q_1481092630696_0004 of type STAGE_COMPLETED

2016-12-07 15:40:16,984 INFO org.apache.tajo.querymaster.Query: Processing q_1481092630696_0004 of type QUERY_COMPLETED

2016-12-07 15:40:16,984 INFO org.apache.tajo.querymaster.Query: q_1481092630696_0004 Query Transitioned from QUERY_RUNNING to QUERY_FAILED

2016-12-07 15:40:16,985 INFO org.apache.tajo.querymaster.QueryMasterTask: Query completion notified from q_1481092630696_0004

2016-12-07 15:40:16,985 INFO org.apache.tajo.querymaster.QueryMasterTask: Query final state: QUERY_FAILED

2016-12-07 15:40:16,987 ERROR org.apache.tajo.querymaster.Stage: Can't handle this event at current state, eventType:SQ_SHUFFLE_REPORT, oldState:FAILED, nextState:FAILED

org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: SQ_SHUFFLE_REPORT at FAILED

        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)

        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)

        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)

        at org.apache.tajo.querymaster.Stage.handle(Stage.java:743)

        at org.apache.tajo.querymaster.QueryMasterTask$StageEventDispatcher.handle(QueryMasterTask.java:226)

        at org.apache.tajo.querymaster.QueryMasterTask$StageEventDispatcher.handle(QueryMasterTask.java:220)

        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)

        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)

        at java.lang.Thread.run(Thread.java:745)

2016-12-07 15:40:16,988 INFO org.apache.tajo.querymaster.QueryMaster: cleanup executionBlocks: eb_1481092630696_0004_000001

2016-12-07 15:40:16,991 INFO org.apache.tajo.querymaster.Query: Processing q_1481092630696_0004 of type STAGE_COMPLETED

2016-12-07 15:40:16,991 ERROR org.apache.tajo.querymaster.Query: Can't handle this event at current state, type:EventType: STAGE_COMPLETED, oldState:QUERY_FAILED, nextState:QUERY_FAILED

org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: STAGE_COMPLETED at QUERY_FAILED

        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)

        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)

        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)

        at org.apache.tajo.querymaster.Query.handle(Query.java:717)

        at org.apache.tajo.querymaster.Query.handle(Query.java:61)

        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)

        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)

        at java.lang.Thread.run(Thread.java:745)

2016-12-07 15:40:16,991 INFO org.apache.tajo.querymaster.QueryMasterTask: Stopping QueryMasterTask:q_1481092630696_0004

2016-12-07 15:40:16,992 INFO org.apache.tajo.querymaster.Query: Processing q_1481092630696_0004 of type INTERNAL_ERROR

2016-12-07 15:40:16,993 INFO org.apache.tajo.querymaster.Query: q_1481092630696_0004 Query Transitioned from QUERY_FAILED to QUERY_ERROR

2016-12-07 15:40:16,994 INFO org.apache.tajo.querymaster.QueryMasterTask: Query completion notified from q_1481092630696_0004

2016-12-07 15:40:16,994 INFO org.apache.tajo.querymaster.QueryMasterTask: Query final state: QUERY_ERROR