demo fails: not stripping -Dmapred.queue.job.name via Tool interface?
R Daneel
run the
cascalog-demo example on our hadoop (0.20.104) cluster because I have
to specify the queue (-Dmapred.job.queue.name), and it seems that
demo's
main() is not stripping that argument from the command-line
(presumably
does not implement Tool/ToolRunner?).
Here's what I get:
hadoop jar cascalog-demo-1.0.0-SNAPSHOT-standalone.jar
cascalog_demo.demo -Dmapred.job.queue.name=unfunded /tmp/follows /tmp/
action /tmp/results
Exception in thread "main" java.lang.IllegalArgumentException: Wrong
number of args (4) passed to: demo$-main
at clojure.lang.AFn.throwArity(AFn.java:439)
at clojure.lang.AFn.invoke(AFn.java:51)
at clojure.lang.AFn.applyToHelper(AFn.java:174)
at clojure.lang.AFn.applyTo(AFn.java:151)
at cascalog_demo.demo.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Is there an obvious way to fix this? I have to be able to pass the -D
argument; I just need to make sure that hadoop strips it (and applies
it) before passing the remaining three args to demo's main().
Any help gratefully appreciated; can't wait to be able to play around
with
cascalog in earnest :)
Thanks,
Mike
nathanmarz
The demo doesn't implement ToolRunner, so it won't respond to any of
those command line options. Alternatively, you can just set your queue
name within the demo code itself. Just change the -main function to
look like this:
(defn -main [follows-dir action-dir output-dir]
(with-job-conf {"mapred.job.queue.name" "unfunded"}
(compute-news-feed (hfs-textline output-dir) follows-dir action-
dir)))
Hope that helps,
Nathan
Michael Potts
Thanks for the speedy response! That does the trick! Unfortunately the setup tasks
are failing, apparently trying to write to "tmp". Here's the build:
gwbl6003 cascalog-demo [0] lein uberjar
Cleaning up.
Copying 4 files to /homes/mpotts/cascalog-demo/lib
Overriding previous definition of reference to cascalog-demo
Overriding previous definition of reference to dependency.fileset
Copying 23 files to /homes/mpotts/cascalog-demo/lib/dev
Overriding previous definition of reference to cascalog-demo
Overriding previous definition of reference to dependency.fileset
Copying 4 files to /homes/mpotts/cascalog-demo/lib
Overriding previous definition of reference to cascalog-demo
Overriding previous definition of reference to dependency.fileset
Copying 23 files to /homes/mpotts/cascalog-demo/lib/dev
Created /homes/mpotts/cascalog-demo/cascalog-demo-1.0.0-SNAPSHOT.jar
Including cascalog-demo-1.0.0-SNAPSHOT.jar
Including clojure-contrib-1.2.0.jar
Including clojure-1.2.0.jar
Including cascalog-1.3.0-20100821.141701-1.jar
Including cascading1.1-1.1.3-20100308.145053-4.jar
Created /homes/mpotts/cascalog-demo/cascalog-demo-1.0.0-SNAPSHOT-standalone.jar
Here's the run:
gwbl6003 cascalog-demo [0] hadoop jar cascalog-demo-1.0.0-SNAPSHOT-standalone.jar cascalog_demo.demo /user/mpotts/follows /user/mpotts/action /user/mpotts/cascalog_demo
10/10/19 03:15:34 INFO util.Util: resolving application jar from found main method on: cascalog_demo.demo
10/10/19 03:15:34 INFO flow.MultiMapReducePlanner: using application jar: /homes/mpotts/cascalog-demo/cascalog-demo-1.0.0-SNAPSHOT-standalone.jar
10/10/19 03:15:34 INFO cascade.Cascade: Concurrent, Inc - Cascading 1.1.rc3-wip-dev [hadoop-0.19.2+]
10/10/19 03:15:34 INFO flow.Flow: [] starting
10/10/19 03:15:34 INFO flow.Flow: [] source: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/follows"]"]
10/10/19 03:15:34 INFO flow.Flow: [] source: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/action"]"]
10/10/19 03:15:34 INFO flow.Flow: [] sink: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
10/10/19 03:15:35 INFO flow.Flow: [] parallel execution is enabled: true
10/10/19 03:15:35 INFO flow.Flow: [] starting jobs: 4
10/10/19 03:15:35 INFO flow.Flow: [] allocating threads: 4
10/10/19 03:15:35 INFO flow.FlowStep: [] starting step: (1/4) ...eFile[['!__gen1__gen4', '!__gen2__gen5']]"][de9808e2-a583-46f0-a770-f/975/]
10/10/19 03:16:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
10/10/19 03:17:12 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1406918 for mpotts
10/10/19 03:17:12 INFO security.TokenCache: Got dt for hdfs://nitroblue-nn1.blue.ygrid.yahoo.com/user/mpotts/.staging/job_201007120904_1129946;uri=98.137.109.218:8020;t.service=98.137.109.218:8020
10/10/19 03:17:12 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
10/10/19 03:17:12 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library
10/10/19 03:17:12 INFO mapred.FileInputFormat: Total input paths to process : 1
10/10/19 03:24:30 WARN flow.FlowStep: [] completion events count: 7
10/10/19 03:24:30 WARN flow.FlowStep: [] event = Task Id : attempt_201007120904_1129946_m_000003_0, Status : FAILED
10/10/19 03:24:30 WARN flow.FlowStep: [] event = Task Id : attempt_201007120904_1129946_r_000001_0, Status : FAILED
10/10/19 03:24:30 WARN flow.FlowStep: [] event = Task Id : attempt_201007120904_1129946_m_000003_1, Status : FAILED
10/10/19 03:24:30 WARN flow.FlowStep: [] event = Task Id : attempt_201007120904_1129946_m_000003_2, Status : FAILED
10/10/19 03:24:30 WARN flow.FlowStep: [] event = Task Id : attempt_201007120904_1129946_r_000001_1, Status : FAILED
10/10/19 03:24:30 WARN flow.FlowStep: [] event = Task Id : attempt_201007120904_1129946_m_000003_3, Status : TIPFAILED
10/10/19 03:24:30 WARN flow.FlowStep: [] event = Task Id : attempt_201007120904_1129946_m_000002_0, Status : SUCCEEDED
10/10/19 03:24:51 WARN flow.FlowStep: [] abandoning step: (3/4) ...?person2__gen22', '?followers__gen27']]"][5e42ee56-21f8-4014-b8f5-9/99866/], predecessor failed: (1/4) ...eFile[['!__gen1__gen4', '!__gen2__gen5']]"][de9808e2-a583-46f0-a770-f/975/]
10/10/19 03:24:51 INFO flow.FlowStep: [] stopping: (3/4) ...?person2__gen22', '?followers__gen27']]"][5e42ee56-21f8-4014-b8f5-9/99866/]
10/10/19 03:25:05 WARN flow.FlowStep: [] abandoning step: (2/4) ...', '?person__gen21', '?action__gen24']]"][71f932ca-5a4a-4339-a3ab-6/82634/], predecessor failed: (1/4) ...eFile[['!__gen1__gen4', '!__gen2__gen5']]"][de9808e2-a583-46f0-a770-f/975/]
10/10/19 03:25:05 INFO flow.FlowStep: [] stopping: (2/4) ...', '?person__gen21', '?action__gen24']]"][71f932ca-5a4a-4339-a3ab-6/82634/]
10/10/19 03:25:05 WARN flow.FlowStep: [] abandoning step: (2/4) ...', '?person__gen21', '?action__gen24']]"][71f932ca-5a4a-4339-a3ab-6/82634/], predecessor failed: (3/4) ...?person2__gen22', '?followers__gen27']]"][5e42ee56-21f8-4014-b8f5-9/99866/]
10/10/19 03:25:05 INFO flow.FlowStep: [] stopping: (2/4) ...', '?person__gen21', '?action__gen24']]"][71f932ca-5a4a-4339-a3ab-6/82634/]
10/10/19 03:25:05 WARN flow.FlowStep: [] abandoning step: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"], predecessor failed: (2/4) ...', '?person__gen21', '?action__gen24']]"][71f932ca-5a4a-4339-a3ab-6/82634/]
10/10/19 03:25:05 INFO flow.FlowStep: [] stopping: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
10/10/19 03:25:05 WARN flow.Flow: stopping jobs
10/10/19 03:25:05 INFO flow.FlowStep: [] stopping: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
10/10/19 03:25:05 INFO flow.FlowStep: [] stopping: (2/4) ...', '?person__gen21', '?action__gen24']]"][71f932ca-5a4a-4339-a3ab-6/82634/]
10/10/19 03:25:05 INFO flow.FlowStep: [] stopping: (3/4) ...?person2__gen22', '?followers__gen27']]"][5e42ee56-21f8-4014-b8f5-9/99866/]
10/10/19 03:25:05 INFO flow.FlowStep: [] stopping: (1/4) ...eFile[['!__gen1__gen4', '!__gen2__gen5']]"][de9808e2-a583-46f0-a770-f/975/]
10/10/19 03:25:18 WARN flow.Flow: stopped jobs
10/10/19 03:25:18 WARN flow.Flow: shutting down job executor
10/10/19 03:25:18 WARN flow.Flow: shutdown complete
Exception in thread "main" cascading.flow.FlowException: step failed: (1/4) ...eFile[['!__gen1__gen4', '!__gen2__gen5']]"][de9808e2-a583-46f0-a770-f/975/]
at cascading.flow.FlowStepJob.blockOnJob(FlowStepJob.java:173)
at cascading.flow.FlowStepJob.start(FlowStepJob.java:138)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:127)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:39)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
and here is a setup task log:
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=mpotts, access=WRITE, inode="tmp":dfsload:hdfs:rwxr-xr-x at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:994) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:316) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1170) at org.apache.hadoop.mapred.FileOutputCommitter.setupJob(FileOutputCommitter.java:52) at org.apache.hadoop.mapred.OutputCommitter.setupJob(OutputCommitter.java:148) at org.apache.hadoop.mapred.Task.runJobSetupTask(Task.java:979) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Permission denied: user=mpotts, access=WRITE, inode="tmp":dfsload:hdfs:rwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:199) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:180) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:128) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4694) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4668) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:1876) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:1845) at org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:659) at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1301) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1297) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1295) at org.apache.hadoop.ipc.Client.call(Client.java:951) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223) at $Proxy6.mkdirs(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy6.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:992) ... 11 more
Any idea what's trying to write to "tmp" and why? I'm sure it's another simple
fix but I have no idea what! This is very tantalizing :)
Thanks again!
Mike
nathanmarz
data in temporary directories on HDFS. The error you're seeing appears
to be because your process doesn't have permission to write to the
temporary directory location. You can try changing "hadoop.tmp.dir"
property of JobConf to somewhere where your process has permissions
(or turn permissions off altogether if you're organization isn't using
them).
Hope that helps,
Nathan
> at java.lang.Thread.run(Thread.java:619)
>
> and here is a setup task log:
>
Michael Potts
Thanks! It's odd though: hadoop.tmp.dir is supposed to be a path in the
local fs (ie. not hdfs), according to the documentation. I tried changing
it to /homes/mpotts/tmp (which exists and is writable by me locally, via
nfs) and got the same exception but complaining about "homes":
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=mpotts, access=WRITE, inode="homes":rdare:hdfs:rwxr-xr-x at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
I'm confused now: do you think cascading is trying (and failing) to write
to hadoop.tmp.dir in hdfs, or in my local file system?
Mike
Michael Potts
Hi again! Looks like it's trying to write to the local file system: when I set
hadoop.tmp.dir to "/tmp" I get a successul MR job. However I then get an unhandled
cascading.flow.FlowException... I don't suppose there's an easy answer to this
one too?!
Mike
gwbl6003 cascalog-demo [0] !hadoop
hadoop jar cascalog-demo-1.0.0-SNAPSHOT-standalone.jar cascalog_demo.demo /user/mpotts/follows /user/mpotts/action /user/mpotts/cascalog_demo
10/10/19 17:44:12 INFO util.Util: resolving application jar from found main method on: cascalog_demo.demo
10/10/19 17:44:12 INFO flow.MultiMapReducePlanner: using application jar: /homes/mpotts/cascalog-demo/cascalog-demo-1.0.0-SNAPSHOT-standalone.jar
10/10/19 17:44:12 INFO cascade.Cascade: Concurrent, Inc - Cascading 1.1.rc3-wip-dev [hadoop-0.19.2+]
10/10/19 17:44:12 INFO flow.Flow: [] starting
10/10/19 17:44:12 INFO flow.Flow: [] source: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/follows"]"]
10/10/19 17:44:12 INFO flow.Flow: [] source: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/action"]"]
10/10/19 17:44:12 INFO flow.Flow: [] sink: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
10/10/19 17:44:13 INFO flow.Flow: [] parallel execution is enabled: true
10/10/19 17:44:13 INFO flow.Flow: [] starting jobs: 4
10/10/19 17:44:13 INFO flow.Flow: [] allocating threads: 4
10/10/19 17:44:13 INFO flow.FlowStep: [] starting step: (1/4) ...File[['!__gen1__gen4', '!__gen2__gen5']]"][3fe83aef-a30c-4a6d-b400-5/1117/]
10/10/19 17:44:43 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
10/10/19 17:44:55 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1416037 for mpotts
10/10/19 17:44:55 INFO security.TokenCache: Got dt for hdfs://nitroblue-nn1.blue.ygrid.yahoo.com/user/mpotts/.staging/job_201010190419_6874;uri=98.137.109.218:8020;t.service=98.137.109.218:8020
10/10/19 17:44:55 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
10/10/19 17:44:55 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library
10/10/19 17:44:55 INFO mapred.FileInputFormat: Total input paths to process : 1
10/10/19 17:47:04 INFO flow.FlowStep: [] starting step: (3/4) ...?person2__gen22', '?followers__gen27']]"][28f958c0-32fc-4578-8e4a-9/73038/]
10/10/19 17:47:08 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
10/10/19 17:47:11 INFO mapred.FileInputFormat: Total input paths to process : 2
10/10/19 17:47:16 INFO mapred.JobClient: Cleaning up the staging area hdfs://nitroblue-nn1.blue.ygrid.yahoo.com/user/mpotts/.staging/job_201010190419_6905
10/10/19 17:47:16 WARN flow.FlowStep: [] abandoning step: (2/4) ...', '?person__gen21', '?action__gen24']]"][c4c4ded4-b9d9-4141-ace7-5/47956/], predecessor failed: (3/4) ...?person2__gen22', '?followers__gen27']]"][28f958c0-32fc-4578-8e4a-9/73038/]
10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (2/4) ...', '?person__gen21', '?action__gen24']]"][c4c4ded4-b9d9-4141-ace7-5/47956/]
10/10/19 17:47:16 WARN flow.FlowStep: [] abandoning step: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"], predecessor failed: (2/4) ...', '?person__gen21', '?action__gen24']]"][c4c4ded4-b9d9-4141-ace7-5/47956/]
10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
10/10/19 17:47:16 WARN flow.Flow: stopping jobs
10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (2/4) ...', '?person__gen21', '?action__gen24']]"][c4c4ded4-b9d9-4141-ace7-5/47956/]
10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (3/4) ...?person2__gen22', '?followers__gen27']]"][28f958c0-32fc-4578-8e4a-9/73038/]
10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (1/4) ...File[['!__gen1__gen4', '!__gen2__gen5']]"][3fe83aef-a30c-4a6d-b400-5/1117/]
10/10/19 17:47:19 WARN flow.Flow: stopped jobs
10/10/19 17:47:19 WARN flow.Flow: shutting down job executor
10/10/19 17:47:19 WARN flow.Flow: shutdown complete
Exception in thread "main" cascading.flow.FlowException: unhandled exception
at cascading.flow.Flow.complete(Flow.java:699)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:90)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:265)
at cascalog.api$_QMARK__.doInvoke(api.clj:47)
at clojure.lang.RestFn.invoke(RestFn.java:422)
at cascalog_demo.demo$compute_news_feed.invoke(demo.clj:33)
at cascalog_demo.demo$_main.invoke(demo.clj:46)
at clojure.lang.AFn.applyToHelper(AFn.java:169)
at clojure.lang.AFn.applyTo(AFn.java:151)
at cascalog_demo.demo.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.ipc.RemoteException: token is expired or doesn't exist
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3673)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1301)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1297)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1295)
Caused by: org.apache.hadoop.ipc.RemoteException: token is expired or doesn't exist
at org.apache.hadoop.ipc.Client.call(Client.java:951)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223)
at $Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:364)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:208)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:175)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1430)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1448)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
at org.apache.hadoop.mapred.JobInProgress$1.run(JobInProgress.java:347)
at org.apache.hadoop.mapred.JobInProgress$1.run(JobInProgress.java:345)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:345)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3671)
... 10 more
at org.apache.hadoop.ipc.Client.call(Client.java:951)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223)
at org.apache.hadoop.mapred.$Proxy6.submitJob(Unknown Source)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:818)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:752)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:752)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:726)
at cascading.flow.FlowStepJob.blockOnJob(FlowStepJob.java:162)
at cascading.flow.FlowStepJob.start(FlowStepJob.java:138)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:127)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:39)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
gwbl6003 cascalog-demo [0]
gwbl6003 cascalog-demo [0]
gwbl6003 cascalog-demo [0]
On Oct 19, 2010, at 12:07 AM, nathanmarz wrote:
nathanmarz
exception before. If you jump into the #cascading room on freenode
Chris Wensel and I can help you figure out what's going on.
-Nathan
On Oct 19, 10:52 am, Michael Potts <maspo...@gmail.com> wrote:
> Hi again! Looks like it's trying to write to the local file system: when I set
> hadoop.tmp.dir to "/tmp" I get a successul MR job. However I then get an unhandled
> cascading.flow.FlowException... I don't suppose there's an easy answer to this
> one too?!
>
> Mike
>
> gwbl6003 cascalog-demo [0] !hadoop
> hadoop jar cascalog-demo-1.0.0-SNAPSHOT-standalone.jar cascalog_demo.demo /user/mpotts/follows /user/mpotts/action /user/mpotts/cascalog_demo
> 10/10/19 17:44:12 INFO util.Util: resolving application jar from found main method on: cascalog_demo.demo
> 10/10/19 17:44:12 INFO flow.MultiMapReducePlanner: using application jar: /homes/mpotts/cascalog-demo/cascalog-demo-1.0.0-SNAPSHOT-standalone.jar
> 10/10/19 17:44:12 INFO cascade.Cascade: Concurrent, Inc - Cascading 1.1.rc3-wip-dev [hadoop-0.19.2+]
> 10/10/19 17:44:12 INFO flow.Flow: [] starting
> 10/10/19 17:44:12 INFO flow.Flow: [] source: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/follows"]"]
> 10/10/19 17:44:12 INFO flow.Flow: [] source: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/action"]"]
> 10/10/19 17:44:12 INFO flow.Flow: [] sink: Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
> 10/10/19 17:44:13 INFO flow.Flow: [] parallel execution is enabled: true
> 10/10/19 17:44:13 INFO flow.Flow: [] starting jobs: 4
> 10/10/19 17:44:13 INFO flow.Flow: [] allocating threads: 4
> 10/10/19 17:44:13 INFO flow.FlowStep: [] starting step: (1/4) ...File[['!__gen1__gen4', '!__gen2__gen5']]"][3fe83aef-a30c-4a6d-b400-5/1117/]
> 10/10/19 17:44:43 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> 10/10/19 17:44:55 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1416037 for mpotts
> 10/10/19 17:44:55 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library
> 10/10/19 17:44:55 INFO mapred.FileInputFormat: Total input paths to process : 1
> 10/10/19 17:47:04 INFO flow.FlowStep: [] starting step: (3/4) ...?person2__gen22', '?followers__gen27']]"][28f958c0-32fc-4578-8e4a-9/73038/]
> 10/10/19 17:47:08 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> 10/10/19 17:47:11 INFO mapred.FileInputFormat: Total input paths to process : 2
> 10/10/19 17:47:16 INFO mapred.JobClient: Cleaning up the staging area hdfs://nitroblue-nn1.blue.ygrid.yahoo.com/user/mpotts/.staging/job_20101019 0419_6905
> 10/10/19 17:47:16 WARN flow.FlowStep: [] abandoning step: (2/4) ...', '?person__gen21', '?action__gen24']]"][c4c4ded4-b9d9-4141-ace7-5/47956/], predecessor failed: (3/4) ...?person2__gen22', '?followers__gen27']]"][28f958c0-32fc-4578-8e4a-9/73038/]
> 10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (2/4) ...', '?person__gen21', '?action__gen24']]"][c4c4ded4-b9d9-4141-ace7-5/47956/]
> 10/10/19 17:47:16 WARN flow.FlowStep: [] abandoning step: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"], predecessor failed: (2/4) ...', '?person__gen21', '?action__gen24']]"][c4c4ded4-b9d9-4141-ace7-5/47956/]
> 10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
> 10/10/19 17:47:16 WARN flow.Flow: stopping jobs
> 10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (4/4) Hfs["TextLine[['line']->[ALL]]"]["/user/mpotts/cascalog_demo"]"]
> 10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (2/4) ...', '?person__gen21', '?action__gen24']]"][c4c4ded4-b9d9-4141-ace7-5/47956/]
> 10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (3/4) ...?person2__gen22', '?followers__gen27']]"][28f958c0-32fc-4578-8e4a-9/73038/]
> 10/10/19 17:47:16 INFO flow.FlowStep: [] stopping: (1/4) ...File[['!__gen1__gen4', '!__gen2__gen5']]"][3fe83aef-a30c-4a6d-b400-5/1117/]
> 10/10/19 17:47:19 WARN flow.Flow: stopped jobs
> 10/10/19 17:47:19 WARN flow.Flow: shutting down job executor
> 10/10/19 17:47:19 WARN flow.Flow: shutdown complete
> Exception in thread "main" cascading.flow.FlowException: unhandled exception
> at cascading.flow.Flow.complete(Flow.java:699)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:265)
> at cascalog.api$_QMARK__.doInvoke(api.clj:47)
> at clojure.lang.RestFn.invoke(RestFn.java:422)
> at cascalog_demo.demo$compute_news_feed.invoke(demo.clj:33)
> at cascalog_demo.demo$_main.invoke(demo.clj:46)
> at clojure.lang.AFn.applyToHelper(AFn.java:169)
> at clojure.lang.AFn.applyTo(AFn.java:151)
> at cascalog_demo.demo.main(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.ipc.RemoteException: token is expired or doesn't exist
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3673)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1301)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1297)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:1062)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1295)
> at org.apache.hadoop.ipc.Client.call(Client.java:951)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223)
> at $Proxy5.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:364)
> at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:208)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:175)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1448)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
> at org.apache.hadoop.mapred.JobInProgress$1.run(JobInProgress.java:347)
> at org.apache.hadoop.mapred.JobInProgress$1.run(JobInProgress.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3671)
> ... 10 more
>
> at org.apache.hadoop.ipc.Client.call(Client.java:951)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223)
> at org.apache.hadoop.mapred.$Proxy6.submitJob(Unknown Source)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:818)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:752)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:726)
> at cascading.flow.FlowStepJob.blockOnJob(FlowStepJob.java:162)
> at cascading.flow.FlowStepJob.start(FlowStepJob.java:138)
> at cascading.flow.FlowStepJob.call(FlowStepJob.java:127)
> at cascading.flow.FlowStepJob.call(FlowStepJob.java:39)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 908)
> at java.lang.Thread.run(Thread.java:619)
> gwbl6003 cascalog-demo [0]
> gwbl6003 cascalog-demo [0]
>
> On Oct 19, 2010, at 12:07 AM, nathanmarz wrote:
>
>
>
> > Cascading (which Cascalog uses for job execution) stores intermediate
> > data in temporary directories on HDFS. The error you're seeing appears
> > to be because your process doesn't have permission to write to the
> > temporary directory location. You can try changing "hadoop.tmp.dir"
> > property of JobConf to somewhere where your process has permissions
> > (or turn permissions off altogether if you're organization isn't using
> > them).
>
> > Hope that helps,
> > Nathan
>
> > On Oct 18, 10:13 pm, Michael Potts <maspo...@gmail.com> wrote:
> >> Thanks for the speedy response! That does the trick! Unfortunately the setup tasks
> >> are failing, apparently trying to write to "tmp". Here's the build:
>
> >> gwbl6003 cascalog-demo [0] lein uberjar
> >> Cleaning up.
> >> Copying 4 files to /homes/mpotts/cascalog-demo/lib
> >> Overriding previous definition of reference to cascalog-demo
> >> Overriding previous
>
>
> read more »
nathanmarz
the new Hadoop security features enabled. For those of you who are
curious, the solution is to set
"mapreduce.job.complete.cancel.delegation.tokens" in the JobConf to
false. More information can be found at the following page:
http://yahoo.github.com/hadoop-common/releases/0.20.104.2/mapred_tutorial.html#Job+Submission+and+Monitoring
>
> read more »