Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Qubole S3a support

118 views
Skip to first unread message

stefan.sc...@smaato.com

unread,
Sep 1, 2015, 9:10:56 AM9/1/15
to Qubole Public Forum
Hi,

does Qubole support the new S3a protocol ? 

It seems that S3a includes many improvements that Qubole already included in the other drivers before, so the second question would be what benefits one would gain from S3a respectively which benefits one would loose from not using the Qubole S3 drivers. 

Further, I have a job with correctly configured S3a credentials (I can run the job on a custom non-qubole cluster perfectly fine), but when I run it one qubole I get errors regarding the credential setup:

App > 15/09/01 12:46:13 main INFO SparkContext: Successfully stopped SparkContext
App > Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
App > at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
App > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
App > at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
App > at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
App > at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:154)
App > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2733)

Would be great to get some feedback on S3a on Qubole.

Thank you and best regards,
Stefan

Minesh Patel

unread,
Sep 1, 2015, 12:34:59 PM9/1/15
to stefan.sc...@smaato.com, Qubole Public Forum
Hi Stefan,

We currently don't support s3a in Hadoop (Spark runs on Yarn, so support would come through that).

The OS community is not considering s3a production ready yet. Support was added in Hadoop 2.6, which is the version Qubole runs. There were some stabilizations added in 2.7, and more to come in 2.8.

We would support s3a when we upgrade to Hadoop 2.8...

Here are some relevant Jiras:

S3A support added in Hadoop 2.6.0: HADOOP-10400 
S3A support stabilisation phase 1 in 2.7.0: HADOOP-11571
S3A support stabilisation phase 2 in 2.8.0: HADOOP-11694

regards,
minesh


--
You received this message because you are subscribed to the Google Groups "Qubole Public Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubole-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubole-users/a1d80f79-8df3-4f98-bdf3-dbddefd59cd3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mpatel

unread,
Jan 4, 2017, 2:27:30 PM1/4/17
to Qubole Public Forum, stefan.sc...@smaato.com
Support for s3a has been added to Qubole on Hadoop 2.7.

To unsubscribe from this group and stop receiving emails from it, send an email to qubole-users+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages