Can't get AWSEC2IAMSessionCredentials to work

120 views
Skip to first unread message

jacek....@gmail.com

unread,
Oct 20, 2015, 5:37:38 PM10/20/15
to JetS3t Users
Hello,

I just tried using AWSEC2IAMSessionCredentials, but it doesn't work for me. Since it doesnt have an empty constructor (like com.amazonaws.auth.InstanceProfileCredentialsProvider) I've just used null for all arguments except the last one (automaticRefreshEnabled) which I did set to true. However this causes NPE. Nearly all arguments seems redundant for Instance Profile based authentication, so I have no idea what do I need to provide to get it to work. Can anybody provide a simple example of how AWSEC2IAMSessionCredentials should be created?

best regards,
Jacek Prucia

James Murty

unread,
Nov 15, 2015, 1:24:56 AM11/15/15
to jets3t...@googlegroups.com
Hi Jacek,

To create a new AWSEC2IAMSessionCredentials object you should call its #loadFromEC2InstanceData method instead of using the constructor – as you noted the constructor requires you to already know the parameter values that you are looking for.

For the general case, you would do the following:

    boolean automaticRefreshEnabled = true;
    AWSEC2IAMSessionCredentials credentials = AWSEC2IAMSessionCredentials.loadFromEC2InstanceData(automaticRefreshEnabled);

    S3Service service = new RestS3Service(credentials);

I have added a pointer to the load methods to the Javadoc for the AWSEC2IAMSessionCredentials class.

Hope this helps,
James


--
You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jets3t-users...@googlegroups.com.
To post to this group, send email to jets3t...@googlegroups.com.
Visit this group at http://groups.google.com/group/jets3t-users.
For more options, visit https://groups.google.com/d/optout.

abcd...@gmail.com

unread,
Dec 17, 2017, 5:43:50 PM12/17/17
to JetS3t Users
Hi Jacek and James,


I want to use IAM role credentials when application is running on an EC2 instnace. I am getting a 403.

Below I have pasted my code and the exception I am getting. I noticed that it is trying to use s3n instead of s3. 

Following is the code I use to setup things:

        conf.set("fs.s3.impl", NativeS3FileSystem.class.getCanonicalName());
        conf.set("fs.s3.buffer.dir", Env.getTempDirectory().getAbsolutePath());

        conf.set("fs.default.name", StringUtil.fmt("s3://%s", bucketName));
        if (creds != null && creds.isValid()) {
            conf.set("fs.s3.awsSecretAccessKey", creds.secretKey);
            conf.set("fs.s3.awsAccessKeyId", creds.accessKey);
        }
        else
        {
            conf.set("fs.s3.awsAccessKeyId", AWSEC2IAMSessionCredentials.loadFromEC2InstanceData(true).getAccessKey());
            conf.set("fs.s3.awsSecretAccessKey",AWSEC2IAMSessionCredentials.loadFromEC2InstanceData(true).getSecretKey());
        }

And I am getting the following exception:

Cant check existence of path-to-some-file.json org.apache.hadoop.security.AccessControlException: Permission denied: s3n://path-to-some-file.json ec=1E0378AC7734BABE org.apache.hadoop.security.AccessControlException: Permission denied: s3n://path-to-some-file.json at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:449) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleException(Jets3tNativeFileSystemStore.java:411) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:181) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at org.apache.hadoop.fs.s3native.$Proxy11.retrieveMetadata(Unknown Source) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:467) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)

... Caused by: org.jets3t.service.impl.rest.HttpException: 403 Forbidden at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:425) at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174)

Reply all
Reply to author
Forward
0 new messages