Trying to batch index with SNAPSHOT using CDH4.5

[Ian Soboroff]

Hi, folks.  I'm trying to get the MapReduceIndexerTool to run over a pile of data I have in HBase.  I asked on the CDH community forum also, but in case the right answer is "Use a later version of hbase-indexer", I am also asking here.

Using the version of the indexer in CDH4.5, I was getting an odd misconfiggish error message:

2014-02-19 15:46:28,954 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1392054788802_0008_000001
2014-02-19 15:46:28,996 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
    at org.apache.hadoop.mapreduce.util.ConfigUtil.addDeprecatedKeys(ConfigUtil.java:52)
    at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:41)
    at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:120)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1088)
2014-02-19 15:46:29,008 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler.

So, in the spirit of adventure I cloned SNAPSHOT from github and tried running that.  I get a different weird error:

014-02-19 15:35:18,092 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1392054788802_0007_000001
2014-02-19 15:35:20,545 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: dfs.datanode.data.dir;  Ignoring.
2014-02-19 15:35:20,590 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: jobSubmitDir=file:/hadoop1/yarn/local/usercache/soboroff/appcache/application_1392054788802_0007/container_1392054788802_0007_01_000001/jobSubmitDir jobTokenFile=file:/hadoop1/yarn/local/usercache/soboroff/appcache/application_1392054788802_0007/container_1392054788802_0007_01_000001/jobSubmitDir/appTokens
2014-02-19 15:35:20,590 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: AM Retries: 1 attempt num: 1 is last retry: true
2014-02-19 15:35:20,642 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2014-02-19 15:35:20,643 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2014-02-19 15:35:20,671 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    org/apache/hadoop/mapreduce/TypeConverter.fromYarn(Lorg/apache/hadoop/mapreduce/v2/api/records/JobReport;Ljava/lang/String;)Lorg/apache/hadoop/mapreduce/JobStatus; @87: invokevirtual
  Reason:
    Type 'org/apache/hadoop/mapred/JobStatus' (current frame, stack[0]) is not assignable to 'org/apache/hadoop/mapreduce/JobStatus'
  Current Frame:
    bci: @87
    flags: { }
    locals: { 'org/apache/hadoop/mapreduce/v2/api/records/JobReport', 'java/lang/String', 'org/apache/hadoop/mapred/JobPriority', 'org/apache/hadoop/mapred/JobStatus' }
    stack: { 'org/apache/hadoop/mapred/JobStatus', 'java/lang/String' }
  Bytecode:
    0000000: b200 964d bb00 9759 2ab9 0098 0100 b800
    0000010: 282a b900 9901 002a b900 9a01 002a b900
    0000020: 9b01 002a b900 9c01 002a b900 9d01 00b8
    0000030: 009e 2c2a b900 9f01 002a b900 a001 002a
    0000040: b900 a101 002b 2ab9 00a2 0100 b700 a34e
    0000050: 2d2a b900 a401 00b6 00a5 2db0         

    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:346)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init(MRAppMaster.java:245)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1145)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1142)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1097)
2014-02-19 15:35:20,684 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler.

Again, looks like violent disagreement about library versions someplace. Anyone have a recommendation on (a) which hbase-indexer I should be using, and/or (b) how do I get everyone agreeing about what jars they should be using?

TIA,
ian

[Ian Soboroff]

I can confirm that the CDH4.5 jar for hbase-indexer-mr-1.3-search-1.1.0-job has mismatched hadoop jars in it with respect to the rest of a CDH4.5 setup.  Filing an issue with them...