error building cascading: The <fileset> type doesn't support the "erroronmissingdir" attribute

[R Daneel]

I've installed the latest cascading from github (https://github.com/cwensel/cascading.git) but when

I try to build it I get a weird error (from any ant command: retrieve, compile, jar, etc.):

$ ant retrieve

Buildfile: build.xml

BUILD FAILED

/homes/mpotts/dev/analysis/cascalog/cascading/build.xml:96: The <fileset> type doesn't support the "erroronmissingdir" attribute.

Total time: 0 seconds

Any ideas?

This is on RHEL 5.1.

Thanks!

Mike

[Chris K Wensel]

You might upgrade your ant version.

or just download a prebuilt version from the community site, or grab wip builds from the company site.

http://www.cascading.org/downloads.html
http://www.concurrentinc.com/downloads/

ckw

> --
> You received this message because you are subscribed to the Google Groups "cascading-user" group.
> To post to this group, send email to cascadi...@googlegroups.com.
> To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

--
Chris K Wensel
ch...@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support for Cascading

[R Daneel]

Thanks!  I installed cascading-1.2.3-hadoop-0.19.2+ and git cloned cascading.samples,

and was able to build the wordcount example:

  cd cascading.samples/wordcount

  ant -Dcascading.home=../../cascading-1.2.3 jar

but then when I try to run it I get an error caused by trying to set permissions

on a nonexistent path (/user/mpotts), which would have been valid in HDFS,

but not in the local filesystem:

hadoop jar ./build/wordcount.jar data/url+page.200.txt output local

11/05/07 01:15:40 INFO flow.MultiMapReducePlanner: using application jar: /homes/mpotts/dev/analysis/cascalog/cascading.samples/wordcount/./build/wordcount.jar

11/05/07 01:15:40 INFO flow.MultiMapReducePlanner: using application jar: /homes/mpotts/dev/analysis/cascalog/cascading.samples/wordcount/./build/wordcount.jar

11/05/07 01:15:40 INFO flow.MultiMapReducePlanner: using application jar: /homes/mpotts/dev/analysis/cascalog/cascading.samples/wordcount/./build/wordcount.jar

11/05/07 01:15:40 INFO flow.MultiMapReducePlanner: using application jar: /homes/mpotts/dev/analysis/cascalog/cascading.samples/wordcount/./build/wordcount.jar

11/05/07 01:15:41 INFO cascade.Cascade: Concurrent, Inc - Cascading 1.2.3 [hadoop-0.19.2+]

11/05/07 01:15:41 INFO cascade.Cascade: [import pages+url pipe+...] starting

11/05/07 01:15:41 INFO cascade.Cascade: [import pages+url pipe+...]  parallel execution is enabled: true

11/05/07 01:15:41 INFO cascade.Cascade: [import pages+url pipe+...]  starting flows: 4

11/05/07 01:15:41 INFO cascade.Cascade: [import pages+url pipe+...]  allocating threads: 4

11/05/07 01:15:41 INFO cascade.Cascade: [import pages+url pipe+...] starting flow: import pages

11/05/07 01:15:41 INFO flow.Flow: [import pages] atleast one sink does not exist

11/05/07 01:15:41 INFO flow.Flow: [import pages] starting

11/05/07 01:15:41 INFO flow.Flow: [import pages]  source: Lfs["TextLine[['offset', 'line']->[ALL]]"]["data/url+page.200.txt"]"]

11/05/07 01:15:41 INFO flow.Flow: [import pages]  sink: Hfs["SequenceFile[['url', 'page']]"]["output/pages/"]"]

11/05/07 01:15:41 INFO tap.Hfs: forcing job to local mode, via source: Lfs["TextLine[['offset', 'line']->[ALL]]"]["data/url+page.200.txt"]"]

11/05/07 01:15:41 INFO flow.Flow: [import pages]  parallel execution is enabled: true

11/05/07 01:15:41 INFO flow.Flow: [import pages]  starting jobs: 1

11/05/07 01:15:41 INFO flow.Flow: [import pages]  allocating threads: 1

11/05/07 01:15:41 INFO flow.FlowStep: [import pages] starting step: (1/1) Hfs["SequenceFile[['url', 'page']]"]["output/pages/"]"]

11/05/07 01:15:41 WARN flow.Flow: stopping jobs

11/05/07 01:15:41 INFO flow.FlowStep: [import pages] stopping: (1/1) Hfs["SequenceFile[['url', 'page']]"]["output/pages/"]"]

11/05/07 01:15:41 WARN flow.Flow: stopped jobs

11/05/07 01:15:41 WARN flow.Flow: shutting down job executor

11/05/07 01:15:41 WARN flow.Flow: shutdown complete

11/05/07 01:15:41 WARN cascade.Cascade: [import pages+url pipe+...] flow failed: import pages

cascading.flow.FlowException: unhandled exception

at cascading.flow.Flow.complete(Flow.java:821)

at cascading.cascade.Cascade$CascadeJob.call(Cascade.java:705)

at cascading.cascade.Cascade$CascadeJob.call(Cascade.java:653)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

at java.util.concurrent.FutureTask.run(FutureTask.java:138)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:619)

Caused by: java.io.IOException: Failed to set permissions of path: file:/user/mpotts-1189815238/.staging to 0700

at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525)

at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318)

at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)

at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:798)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)

at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)

at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)

at cascading.flow.FlowStepJob.blockOnJob(FlowStepJob.java:164)

at cascading.flow.FlowStepJob.start(FlowStepJob.java:140)

at cascading.flow.FlowStepJob.call(FlowStepJob.java:129)

at cascading.flow.FlowStepJob.call(FlowStepJob.java:39)

... 5 more

is this something trivial (hopefully!)?

Mike

[R Daneel]

sorry: meant to say the nonexistent path is /user/mpotts-1189815238 (looks like a one-off).

[Chris K Wensel]

probably has much to do with having permissions enabled on your cluster. might turn them off.

On May 6, 2011, at 6:22 PM, R Daneel wrote:

> sorry: meant to say the nonexistent path is /user/mpotts-1189815238 (looks like a one-off).
>
>

[R Daneel]

I'm afraid I can't make any changes to the cluster; might there be any configuration parameters I can supply?

I tried: -Dhadoop.tmp.dir=/tmp (which had solved an earlier cascalog problem), but it made no difference.  I

know I can get cascading working on this cluster if I run the cascalog-demo example; just not yet using

the cascalog.samples wordcount example!

[Chris K Wensel]

you need to take this up on the cascalog list, i've no idea how to supply properties to the FlowConnector.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

[R Daneel]

Actually at this point I'm just trying to get the cascading sample(s) working.  Then I want

to debug a new Scheme I've written; I do have an intention of using that Scheme from

cascalog as well, but here I'm all about cascading itself.

[Chris K Wensel]

Then try passing the the property to the FlowConnector constructor.

if you cannot change the hadoop temp directory, use this method

http://www.cascading.org/1.2/javadoc/cascading/tap/Hfs.html#setTemporaryDirectory(java.util.Map,%20java.lang.String)

If that doesn't work, you have no choice but to get permissions on your cluster.

ckw

[nathanmarz]

You can pass properties to the flow connector by wrapping your queries
in the with-job-conf macro. e.g.:

(with-job-conf {"io.sort.mb" 20}
(?<- ...))

On May 9, 1:55 pm, Chris K Wensel <ch...@wensel.net> wrote:
> Then try passing the the property to the FlowConnector constructor.
>
> if you cannot change the hadoop temp directory, use this method
>

> http://www.cascading.org/1.2/javadoc/cascading/tap/Hfs.html#setTempor...)

>
> If that doesn't work, you have no choice but to get permissions on your cluster.
>
> ckw
>
> On May 9, 2011, at 1:38 PM, R Daneel wrote:
>
> > Actually at this point I'm just trying to get the cascading sample(s) working.  Then I want
> > to debug a new Scheme I've written; I do have an intention of using that Scheme from
> > cascalog as well, but here I'm all about cascading itself.
>
> > --
> > You received this message because you are subscribed to the Google Groups "cascading-user" group.
> > To post to this group, send email to cascadi...@googlegroups.com.
> > To unsubscribe from this group, send email to cascading-use...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/cascading-user?hl=en.
>
> --
> Chris K Wensel
> ch...@concurrentinc.comhttp://www.concurrentinc.com

[Ken Krugler]

Hi there,

I see this:

11/05/07 01:15:41 INFO tap.Hfs: forcing job to local mode, via source: Lfs["TextLine[['offset', 'line']->[ALL]]"]["data/url+page.200.txt"]"]

and this:

Caused by: java.io.IOException: Failed to set permissions of path: file:/user/mpotts-1189815238/.staging to 0700

You're running in local mode, so I don't think this has anything to do with your HDFS permissions or cluster configuration.

A quick search on Google:

getStagingDir "Failed to set permissions of path"

indicates this can happen if you're running on Windows, using cygwin, and that wasn't installed properly.

-- Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g