s3distcp not working for me.

[Anson Abraham]

I'm running CDH4.5.  I was trying to use distcp to s3n but am getting issues since I upgraded to 4.5.

I'm trying to get s3distcp up and running, but am getting issues.  I downloaded it, and am running this command:

hadoop jar /usr/lib/hadoop/lib/s3distcp.jar  --src hdfs://NN:8020/path/to/destination/folder --dest s3n://acceseKeyId:secretaccesskey@mybucket/destination/

but I'm getting error of:

INFO mapred.JobClient:  map 100% reduce 0%

INFO mapred.JobClient: Task Id : attempt_201312042223_10889_r_000001_0, Status : FAILED

Error: java.lang.ClassNotFoundException: com.amazonaws.services.s3.AmazonS3

at java.net.URLClassLoader$1.run(URLClassLoader.java:202)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

at java.lang.ClassLoader.loadClass(ClassLoader.java:306)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)

at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

at com.amazon.external.elasticmapreduce.s3distcp.CopyFilesReducer.executeDownloads(CopyFilesReducer.java:209)

at com.amazon.external.elasticmapreduce.s3distcp.CopyFilesReducer.reduce(CopyFilesReducer.java:196)

at com.amazon.external.elasticmapreduce.s3distcp.CopyFilesReducer.reduce(CopyFilesReducer.java:30)

at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax

INFO mapred.JobClient: Job Failed: NA

13/12/12 13:55:25 INFO s3distcp.S3DistCp: Try to recursively delete hdfs:/tmp/985ffdb0-1bc8-4d00-aba6-fd9b18e905f1/tempspace

Exception in thread "main" java.lang.RuntimeException: Error running job

at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:586)

at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:216)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

at com.amazon.external.elasticmapreduce.s3distcp.Main.main(Main.java:12)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Caused by: java.io.IOException: Job failed!

at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1388)

at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:568)

I then put access key and id into my core-site.xml on all the data and name node: 

  <property>

    <name>fs.s3.awsSecretAccessKey</name>

    <value>bippitybopityboo</value>

  </property>   

  <property>

    <name>fs.s3.awsAccessKeyId</name>

    <value>supercalifragilisticexpialadoscious</value>

  </property>

  <property>

    <name>fs.s3n.awsSecretAccessKey</name>

    <value>bippitybopityboo</value>

  </property>

  <property>

    <name>fs.s3n.awsAccessKeyId</name>

    <value>supercalifragilisticexpialadoscious</value>

  </property>

 

and still was getting same error when i tried this:

hadoop jar /usr/lib/hadoop/lib/s3distcp.jar  --src hdfs://NN:8020/path/to/destination/folder --dest s3n://mybucket/destination/

Is there some configuration I'm supposed to do, or am I missing some jar files or executing it incorrectly?

Thanks for any help.

[David Marin]

This is really more of a Hadoop question than a mrjob question. You'll probably have better luck on a Hadoop mailing list, StackOverflow, etc.

-Dave

> --
> You received this message because you are subscribed to the Google Groups "mrjob" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mrjob+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Nick Dimiduk]

It looks like you're missing some dependency jars at runtime. Maybe work out which jar is missing and use -libjars to include them. This post has some background info: http://grepalex.com/2013/02/25/hadoop-libjars/

Cheers,

Nick

--