Scoobi with CHD4

[piotr...@gmail.com]

Hi Guys,

I have been recently using scoobi (0.6.2-cdh4) to run some jobs on a remote CHD4 (cdh4.2.0).

So my scoobi is on a machine that only has hadoop-client installed with the hadoop configuration at /etc/hadoop/conf and it submits jobs to the remote cluster defined in this configuration.

I have run into a couple of issues with that setup:

1.       useconfdir  option has an undocumented feature of first trying to determine the HADOOP_HOME from the location of hadoop script and only if it does not exists, checks the env variable. This does not work with the out of the box CHD4 installation (as hadoop is /usr/bin/hadoop and the config is in /etc/hadoop/conf).  Would you mind reversing the order so that env variable goes first? Or alternatively add an option to pass the conf location.

2.       I was using upload jars option and (that the dependencies were uploaded to distributed cache) and I got the java.lang.ClassNotFoundException: com.nicta.scoobi.impl.exec. MscrMapper. I think the problem is that the jars are added to the classpath as full URIs including  ‘hdfs://hostname:323434/’ while they should be local paths. It seems that the code that does it is a bit dodgy as it adds jars with DistributedCache.addFileToClassPath and full URIs (some of them twice) and then sets mapred. classpath property in configuration using local paths. Actually addFileToClassPath uses Path but at least in my setup FileSystem returns them as URIs with schema and hostname. I have managed to fix that by setting mapred.job.classpath.files to whatever was set in mapred. classpath and it works. I am not sure however where in mapred. classpath comes from and why it’s need (that is if DistributedCache.addFileToClassPath were used with local paths)?

3.       Another problem was that the dependencies include some (but not all) hadoop jars which are different in scoobi dependencies and in my runtime (different version of CHD4). I was able to solve this issue by filtering out the hadoop jars from the job classpath and I am wondering if that should not be done always (that is treat hadoop-* as provided and not uploading them and not including into job classpath)

I am happy to fix those issues (provided they are issues) and summit a pull request but I am wondering what is a correct process for that and also if I can still (and how) get them to the 0.6.2 version of scoobi.

Cheers,

-          Piotr

[Eric Torreborre]

Hi Piotr,

You can fork the repository, create a pull request on the 0.6.3 branch I just created. Then submit your pull request and I can publish a 0.6.3 version from there.

About point n.3 I remember that in the past we were filtering out hadoop jars and eventually we left it because,... I don't remember precisely but one execution mode was not working without it: from sbt/outside sbt, locally/on the cluster or a combination. Your use case might also be seen as "experimental" because we can't certify much that Scoobi is working with any other CDH version that specified in its own sbt file. So in that case overriding the dependencies is the right find to do. But I'm open to suggestions.

Thanks,

Eric.