Trouble with Tez and cascading-hive

20 views
Skip to first unread message

Russ

unread,
Sep 30, 2016, 10:42:01 AM9/30/16
to cascading-user
Hello,

I am using cascading-hive:2.1.0 with cascading 3.1.1 on HDP 2.4.2. I have a complex flow that reads from several Hive/HCat tables using HcatTap and HCatScheme. Everything is working great on MR1, but now I am trying to switch to TEZ for better performance. Unfortunately the HCatTap does not seem to be working right on TEZ. I am getting a vertex failure with the following cause:

Caused by: java.lang.NullPointerException: inputJobInfo is null, setInput has not yet been called to save job into conf supplied
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:209)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.getPartitionColumns(HCatInputFormat.java:155)
at cascading.scheme.hcatalog.HCatScheme.sourcePrepare(HCatScheme.java:115)
...

I can see that the HCatTap does indeed call HCatInputFormat.setInput during sourceConfInit, and like I said everything was working fine on MapReduce.

Does anyone have any idea of what could be wrong here? Is there some way that HCatTap.sourceConfInit might not be getting called on TEZ?

Thanks
-Russ
Reply all
Reply to author
Forward
0 new messages