parsing hadoop args

[Elliot Chow]

Hi,

I am using the 0.7.0-cdh3-SNAPSHOT.  I am trying to set the hadoop queue and it seems to be failing now (I have been doing it this way for a while now and it had been working).  The command I am using is

hadoop jar ~/jar/dp-assembly-1.0-SNAPSHOT.jar com.ebay.ss.dp.Template -Dmapred.job.queue.name=hdmi-set -Dmapred.reduce.tasks.speculative.execution=true -Dscoobi.mapreduce.reducers.max=20000 -- scoobi all.times

The error I get is:

cause:org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.IllegalArgumentException: There is no queue named default
        at org.apache.hadoop.mapred.QueueManager.getQueueACL(QueueManager.java:394)
        at org.apache.hadoop.mapred.JobTracker.getQueueAdmins(JobTracker.java:4454)
        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399)

I see the following debug info:

[DEBUG] ScoobiApp - parsing the hadoop arguments com.ebay.ss.dp.Template, -Dmapred.job.queue.name=hdmi-set, -Dmapred.reduce.tasks.speculative.execution=true, -Dscoobi.mapreduce.reducers.max=20000, --, scoobi, all.times
[DEBUG] ScoobiApp - setting the non-hadoop arguments com.ebay.ss.dp.Template, -D, mapred.job.queue.name=hdmi-set, -D, mapred.reduce.tasks.speculative.execution=true, -D, scoobi.mapreduce.reducers.max=20000, scoobi, all.times



Am I doing something wrong, or has something changed?

[Eric Torreborre]

Hi Elliot,

I've been trying to reproduce your issue and insert a few debug statements here and there to see where your setting might get lost but I couldn't find anything.

Can you please do the same on your end and, for example, print out the value of the mapred.job.queue.name parameter just before creating your DList? And then right after persisting it?

Can you also check if this is the same thing for other properties? Passed from the command-line or directly set inside the application?

Thanks,

Eric.

[Elliot Chow]

I think I found the issue.  If sbt assembly sets a main class, then it takes everything I have passed on the command line, including the class name, as arguments.  Since the class name is not a hadoop arg, everything gets passed as a remaining arg.

The work around I have found is to explicitly tell sbt assembly not to use a main class by adding the following to your build.sbt:

mainClass in assembly := None

My guess (not confirmed) is that if you have only 1 ScoobiApp in your jar, sbt assembly takes that as the main class - if you have multiple ScoobiApp's, it cannot guess properly so it does not set the main class (this is probably why I have never seen this before).

[Eric Torreborre]

I see, that's good to know, thanks.

E.