Problems running a scalding job from java

[Hagai Attias]

Hello,

what is the proper way to run a scalding job from Java?

I have my scalding job defined as

class MyJob(args: Args) extends Job(args) {...}

From Java I try:

Hdfs hdfs = new Hdfs(strict, jobconf);

Args myArgs = buildMyJobArgs();

Args args = Mode$.MODULE$.putMode(hdfs, myArgs);

Job job = new MyJob(args);

Flow f = job.buildFlow();

f.complete();

When running on the hadoop cluster I get 

java.lang.IllegalStateException: path not initialized

at cascading.tap.hadoop.Hfs.getPath(Hfs.java:371)

at cascading.tap.hadoop.Hfs.getIdentifier(Hfs.java:362)

at cascading.tap.Tap.equals(Tap.java:711)

at java.util.HashMap.containsValue(HashMap.java:634)

at java.util.HashMap$Values.contains(HashMap.java:916)

at cascading.flow.planner.FlowPlanner.verifySourceNotSinks(FlowPlanner.java:217)

at cascading.flow.planner.FlowPlanner.verifyAllTaps(FlowPlanner.java:170)

at cascading.flow.hadoop.planner.HadoopPlanner.buildFlow(HadoopPlanner.java:222)

 [wrapped] cascading.flow.planner.PlannerException: could not build flow from assembly: [path not initialized]

at cascading.flow.planner.FlowPlanner.handleExceptionDuringPlanning(FlowPlanner.java:576)

at cascading.flow.hadoop.planner.HadoopPlanner.buildFlow(HadoopPlanner.java:265)

at cascading.flow.hadoop.planner.HadoopPlanner.buildFlow(HadoopPlanner.java:80)

at cascading.flow.FlowConnector.connect(FlowConnector.java:459)

at com.twitter.scalding.Job.buildFlow(Job.scala:226)

at com.akamai.csi.jobs.commons.BasicInitialCsiScaldingJob.executeJob(BasicInitialCsiScaldingJob.java:28)

at com.akamai.csi.jobs.commons.BasicCsiHadoopJob.execute(BasicCsiHadoopJob.java:359)

at com.akamai.csi.jobs.commons.BasicInitialCsiHadoopJob.process(BasicInitialCsiHadoopJob.java:42)

at com.akamai.csi.jobs.commons.BasicCsiHadoopJob.run(BasicCsiHadoopJob.java:172)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at com.akamai.csi.jobs.commons.BasicCsiHadoopJob.main(BasicCsiHadoopJob.java:143)

at com.akamai.csi.jobs.commons.BasicCsiHadoopJob.main(BasicCsiHadoopJob.java:138)

at com.akamai.csi.jobs.needles.attack.AttackNeedleTestJob.main(AttackNeedleTestJob.java:48)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495)

at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

 

[Hagai Attias]

direction anyone?

[Ian O'Connell]

Well i guess why would you need to/want to is probably the first thing to look at. You can just use the scalding tool runner to launch something for most scenarios. After that the easiest way to launch most scala apps is build yourself an easy to call runner in scala and then call that from java. In general calling scala from java can be awkward. 

In this case it does look like your job is probably just broken though. But hard to know from the error.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/0bdad994-de26-4832-bf17-3865291ad858%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.