AWS S3: Unable to load credentials into profile

[Cinto Sunny]

Hi Team,

I have an AWS EKS cluster running Druid. When I access S3 from CLI from a pod, it returns the buckets correctly.

aws s3 ls druid-s3-bucket 

PRE devint/
PRE test-druid/
2022-03-15 21:41:36 4 tes

But from the SDK, it fails with the error

Unable to load credentials into profile [druidbotint]: AWS Access Key ID is not specified., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@2ff8d560: Failed to connect to service endpoint: , com.amazonaws.auth.InstanceProfileCredentialsProvider@3cad42b9: Failed to connect to service endpoint: ] at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:136) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713) ~[aws-java-sdk-core-1.12.37.jar:?]

I have a custom role and I have an external process to look up credentials. Here is the aws/credential file

[default] source_profile = druidbotint 

role_arn = arn:aws:iam::1233:role/worker-role 

role_session_name = druidbotsession 

 [druidbotint] credential_process = awsconnect -u druid_s -a 1233 -r custom_role -p conf

Here is the aws/config file

[profile conf] region = us-west-2

Anyone encountered this issue?

[Mark Herrera]

Hi Cinto,

I wonder if it might be your custom role and external process to look up credentials? Here are the S3 authentication methods, and they can be overridden by specifying an access key and secret key through the Properties Object in the ingestionSpec.

Best,

Mark

[Cinto Sunny]

yeah, it has something to do with the customer role and the external process. Since running as admin works fine. 

We do not have the access key and secret key as auth happens via external process.