cascading, lazy initialization and karmasphere

[anahap]

I am encountering a strange problem with cascading and karmasphere
studio.

When I deploy to a local one node cluster the temporary job jar
created bei karmasphere get deleted before
the flow is able to copy the jar to its local filesystem. This doesnt
happen when i create a custom mapreduce job with
cascading.flow.MapReduceFlow and add it to a cascade. It only happens
with "regular" cascading flows.

I checked that the jar existed when the deployment started and it
works well when using custom mapreduce jobs.
however the jar is deleted before cascading has used it causing the
below error.

I tried it with cascading 1.1.2 and 1.2-wip with the same results,
using hadoop 0.20.2. The error message I got was:

cascading.flow.FlowException: unhandled exception
at cascading.flow.Flow.complete(Flow.java:802)
at cascading.cascade.Cascade$CascadeJob.call(Cascade.java:706)
at cascading.cascade.Cascade$CascadeJob.call(Cascade.java:654)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.io.FileNotFoundException: File /private/var/folders/7V/
7V2iKIiHHtaFmJAWfz2zbE TI/-Tmp-/karmasphere-
assembler-4398225942085160682.dir/hadoop-job-6253249942248680449.jar
does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:
361)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:
245)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:
1189)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:
1165)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:
1137)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:
657)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:
761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:
730)
at cascading.flow.FlowStepJob.blockOnJob(FlowStepJob.java:162)
at cascading.flow.FlowStepJob.start(FlowStepJob.java:138)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:127)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:39)
... 5 more

I would appreciate any help, because this is driving me nuts.

TIA
Andy

[anahap]

Some more info:

The error already happens with a simple flow like:


public static void main(String[] args) throws Exception {
Tap source = new Hfs( new TextLine(), "/tmp/testpationing/
syslog" );
Tap sink = new Hfs(new TextLine(), "/tmp/syslog_out");
Pipe assembly = new Pipe( "test" );
Properties properties = new Properties();
FlowConnector.setApplicationJarClass(properties,
Eightylegsproccessing.class);
FlowConnector flowConnector = new FlowConnector( properties );
Flow flow = flowConnector.connect( "test", source, sink,
assembly );
flow.complete();

}

[Chris K Wensel]

i've forwarded your email to the Karmasphere folk, as this is their issue if I understand correctly.

ckw

> --
> You received this message because you are subscribed to the Google Groups "cascading-user" group.
> To post to this group, send email to cascadi...@googlegroups.com.
> To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
>

--
Chris K Wensel
ch...@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support, and licensing for Cascading

[anahap]

Sorry for this post, and update to the newest version of karmasphere
solved the problem!
Thanks for your support chris!

On Oct 22, 10:44 pm, Chris K Wensel <ch...@wensel.net> wrote:
> i've forwarded your email to the Karmasphere folk, as this is their issue if I understand correctly.
>
> ckw
>
> On Oct 22, 2010, at 1:41 PM, anahap wrote:
>
>
>
>
>
> > Some more info:
>
> > The error already happens with a simple flow like:
>
> >    public static void main(String[] args) throws Exception  {
> >        Tap source = new Hfs( new TextLine(), "/tmp/testpationing/
> > syslog" );
> >        Tap sink = new Hfs(new TextLine(), "/tmp/syslog_out");
> >        Pipe assembly = new Pipe( "test" );
> >        Properties properties = new Properties();
> >        FlowConnector.setApplicationJarClass(properties,
> > Eightylegsproccessing.class);
> >        FlowConnector flowConnector = new FlowConnector( properties );
> >        Flow flow = flowConnector.connect( "test", source, sink,
> > assembly );
> >        flow.complete();
>
> >    }
>
> > --
> > You received this message because you are subscribed to the Google Groups "cascading-user" group.
> > To post to this group, send email to cascadi...@googlegroups.com.
> > To unsubscribe from this group, send email to cascading-use...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/cascading-user?hl=en.
>
> --
> Chris K Wensel
> ch...@concurrentinc.comhttp://www.concurrentinc.com

[Chris K Wensel]

No worries. glad it sorted itself out.

ckw

> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.