AWS S3: Unable to load credentials into profile

48 views
Skip to first unread message

Cinto Sunny

unread,
Mar 16, 2022, 4:21:13 PMMar 16
to Druid User

Hi Team,

I have an AWS EKS cluster running Druid. When I access S3 from CLI from a pod, it returns the buckets correctly.

aws s3 ls druid-s3-bucket 

PRE devint/
PRE test-druid/
2022-03-15 21:41:36 4 tes

But from the SDK, it fails with the error

Unable to load credentials into profile [druidbotint]: AWS Access Key ID is not specified., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@2ff8d560: Failed to connect to service endpoint: , com.amazonaws.auth.InstanceProfileCredentialsProvider@3cad42b9: Failed to connect to service endpoint: ] at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:136) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753) ~[aws-java-sdk-core-1.12.37.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713) ~[aws-java-sdk-core-1.12.37.jar:?]

I have a custom role and I have an external process to look up credentials. Here is the aws/credential file

[default] source_profile = druidbotint 
role_arn = arn:aws:iam::1233:role/worker-role 
role_session_name = druidbotsession 

 [druidbotint] credential_process = awsconnect -u druid_s -a 1233 -r custom_role -p conf

Here is the aws/config file

[profile conf] region = us-west-2

Anyone encountered this issue?

Mark Herrera

unread,
Mar 16, 2022, 6:06:04 PMMar 16
to Druid User
Hi Cinto,

I wonder if it might be your custom role and external process to look up credentials? Here are the S3 authentication methods, and they can be overridden by specifying an access key and secret key through the Properties Object in the ingestionSpec.

Best,

Mark

Cinto Sunny

unread,
Mar 17, 2022, 9:04:47 AMMar 17
to Druid User
yeah, it has something to do with the customer role and the external process. Since running as admin works fine. 
We do not have the access key and secret key as auth happens via external process.

Reply all
Reply to author
Forward
0 new messages