java.lang.RuntimeException: hbase-default.xml file seems to be for and old version of HBase (null)

[shyam....@gmail.com]

Hey Guys,

I am using 3.4.0 to build hydrator pipeline. My question is how to use Hbase sink component. I am connecting oracle database and getting records. Now I would like to put these records in Hbase table. I created table and gave correct Zookeeper information. But I am seeing runtime exception. Here is the log -

016-05-03 09:22:32,768 - ERROR [MapReduceRunner-phase-1:c.c.c.i.a.r.b.MapReduceRuntimeService@299] - Exception when submitting MapReduce Job: job=phase-1,=namespaceId=default, applicationId=NovusBulkLoad, program=phase-1, runid=8f493f72-1110-11e6-ae87-0800277b27cc

co.cask.tephra.TransactionFailureException: Failed to execute method beforeSubmit inside a transaction

at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:177) ~[co.cask.cdap.cdap-data-fabric-3.4.0.jar:na]

at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.beforeSubmit(MapReduceRuntimeService.java:466) ~[co.cask.cdap.cdap-app-fabric-3.4.0.jar:na]

at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:204) ~[co.cask.cdap.cdap-app-fabric-3.4.0.jar:na]

at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na]

at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:386) [co.cask.cdap.cdap-app-fabric-3.4.0.jar:na]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be for and old version of HBase (null), this version is 0.98.6.1-hadoop2

at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:73) ~[na:na]

at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105) ~[na:na]

at co.cask.hydrator.plugin.sink.HBaseSink.prepareRun(HBaseSink.java:75) ~[1462267351269-0/:na]

at co.cask.hydrator.plugin.sink.HBaseSink.prepareRun(HBaseSink.java:57) ~[1462267351269-0/:na]

at co.cask.cdap.etl.batch.LoggedBatchConfigurable$1.call(LoggedBatchConfigurable.java:44) ~[cdap-etl-batch-3.4.0.jar:na]

at co.cask.cdap.etl.batch.LoggedBatchConfigurable$1.call(LoggedBatchConfigurable.java:41) ~[cdap-etl-batch-3.4.0.jar:na]

at co.cask.cdap.etl.log.LogContext.run(LogContext.java:59) ~[cdap-etl-core-3.4.0.jar:na]

at co.cask.cdap.etl.batch.LoggedBatchConfigurable.prepareRun(LoggedBatchConfigurable.java:41) ~[cdap-etl-batch-3.4.0.jar:na]

at co.cask.cdap.etl.batch.mapreduce.ETLMapReduce.beforeSubmit(ETLMapReduce.java:189) ~[cdap-etl-batch-3.4.0.jar:na]

at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2.call(MapReduceRuntimeService.java:471) ~

One more question, if cluster is enabled with kerberoes authentication, how does this Hbase sink behave, where should we specify the keytab file?

Thank you,

Regards,

Shyam

[Ali Anwar]

Hi Shyam.

When you are getting that exception, is it in SDK or an installed distributed CDAP? If it is a distributed CDAP, can you give the environment details such as the hadoop version?
I will look further into how HBase sink works with kerberos auth enabled.

Regards,

Ali Anwar

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/2a402b0d-107f-4027-a5f3-0df1be702dd5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ali Anwar]

Hey Shyam.

Regarding HBase Sink with kerberos authentication: the CDAP program being run (likely a MapReduce job) will have delegation tokens from CDAP master, and so it will work fine. I tested this to verify.
I was also able to have a pipeline that writes from Stream to HBase sink on a CDAP cluster, installed using 3.4.0 packages.

Could you share more information so I can reproduce this issue?
Useful items would be:
1) Share pipeline information, by exporting the pipeline configuration into JSON. In order to get export functionality, you will have to clone  it.
2) Hadoop version and how its installed (rpm/deb packages, Cloudera Manager, etc).

Regards,

Ali Anwar

[vin...@cask.co]

Hi Shyam,

For  java.lang.RuntimeException, we have fixed hydrator plugins. Please follow https://github.com/caskdata/hydrator-plugins/pull/251 for more information on the fix. 

To use the fixed version, there are 2 ways:

1.) Build the system artifacts from https://github.com/caskdata/hydrator-plugins/tree/release/1.3, delete the old hbase-plugins artifacts and load the new one.

2.) We are planning on a minor release for cdap, within next week, which will have latest artifacts.

Please let me know if you have any other questions.

Regards,

Vinisha