S3distcp Download

0 views

Skip to first unread message

Athenna Jimenez

unread,

Jan 21, 2024, 9:42:41 AM1/21/24

to distprofontros

When I use s3distcp, I try to copy to the bucket and pass --s3ServerSideEncryption as per AWS documentation. However, I get error 403 when trying to write the objects. The --s3ServerSideEncryption parameter does not give the option to specify a key etc.

s3distcp download

Download Zip ===> https://t.co/zsZmJUpVUB

I've run a job on AWS's EMR, and stored the output in the EMR job's HDFS. I am then trying to copy the result to S3 via distcp or s3distcp, but both are failing as described below. (Note: the reason I'm not just sending my EMR job's output directly to S3 is due to the (currently unresolved) problem I describe in Where is my AWS EMR reducer output for my completed job (should be on S3, but nothing there)?

Here is the error:
Error: java.lang.RuntimeException: Reducer task failed to copy 272 files: s3://production-snowplow-processing-data/processing/EWBPWVW3GFOLK.2018-09-13-11.4d9fdf5f.gz etc
at com.amazon.elasticmapreduce.s3distcp.CopyFilesReducer.cleanup(CopyFilesReducer.java:67)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:635)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)