Hi,
does Qubole support the new S3a protocol ?
It seems that S3a includes many improvements that Qubole already included in the other drivers before, so the second question would be what benefits one would gain from S3a respectively which benefits one would loose from not using the Qubole S3 drivers.
Further, I have a job with correctly configured S3a credentials (I can run the job on a custom non-qubole cluster perfectly fine), but when I run it one qubole I get errors regarding the credential setup:
App > 15/09/01 12:46:13 main INFO SparkContext: Successfully stopped SparkContext
App > Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
App > at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
App > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
App > at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
App > at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
App > at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:154)
App > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2733)
Would be great to get some feedback on S3a on Qubole.
Thank you and best regards,
Stefan