AWS-IAM authentication Region error

Asif M

unread,

Mar 28, 2018, 2:11:34 AM3/28/18

to Vault

Hi All

I am trying to use EC2-IAM authentication and using the IAM credentials from ec2-metadata.

it looks like the vault is using us-east-1 as the default region and I can't see a way to override it. If I use us-east-1 in my client signature then it works but if I use any other such as us-west-2 etc. it fails.

Here is the error/exception:

error making upstream request: received error code 403 from STS: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">

<Error>

<Type>Sender</Type>

<Code>SignatureDoesNotMatch</Code>

<Message>Credential should be scoped to a valid region, not 'us-west-2'. </Message>

</Error>

</ErrorResponse>

error making upstream request: received error code 403 from STS: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">

<Error>

<Type>Sender</Type>

<Code>SignatureDoesNotMatch</Code>

<Message>Credential should be scoped to a valid region, not 'us-west-2'. </Message>

</Error>

</ErrorResponse>

thanks

ASif

Jeff Mitchell

unread,

Mar 28, 2018, 10:55:05 AM3/28/18

to Vault

Hi Asif,

How is your auth backend set up? Did you set https://www.vaultproject.io/api/auth/aws/index.html#inferred_aws_region ? I'm not sure if this is a client issue or something else, but you could try setting the standard AWS_REGION env var as well.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/21c94559-7903-4c7c-9702-634a7a1c712b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Joel Thompson

unread,

Mar 28, 2018, 10:58:29 AM3/28/18

to vault...@googlegroups.com

Hi Asif,

You need to update the auth/aws/config/client's sts_endpoint parameter (https://www.vaultproject.io/api/auth/aws/index.html#sts_endpoint) to specify the STS endpoint in the alternate region.

The reason for this is Vault is being a bit paranoid in ensuring it doesn't get used as an arbitrary HTTP proxy, and STS is really weird service relative to the rest of the AWS ecosystem in terms of regions (see my comment at https://github.com/hashicorp/vault-ruby/pull/161#issuecomment-355723269 if you want some light bedtime reading on this weirdness and how Vault handles it).

--Joel

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

Message has been deleted

Meet vadaria

unread,

Mar 18, 2019, 7:25:05 PM3/18/19

to Vault

Hi Joel,

Thanks for this. it helped me as well :)

but i am running into different issue. if we configure STS region specific endpoint, then we are limiting vault to that specific region. how can we have multi region support? my clients are in us-east-1 and us-west-2, my vault is sitting in us-east-1. is it possible to use vault for both region?

Thanks,

Meet

Joel Thompson

unread,

Mar 18, 2019, 10:06:08 PM3/18/19

to Vault

Hi Meet,

Glad you found this useful :)

Great question! tl;dr: The choice of an STS region doesn't actually limit Vault to only working with clients that specific region. It will work with clients in any AWS region in the same "partition" (the AWS partitions are Standard, GovCloud, and AWS China). So, I would recommend having all clients just use the region your Vault server is in, which in your use case is us-east-1.

A longer explanation:

With regards to regions, STS is a weird and confusing AWS service. It's global in the sense that credentials issued by IAM and STS are valid in any AWS region (in the same partition), e.g., credentials issued by STS in us-west-2 are valid in us-east-1 (but wouldn't be valid in regions in GovCloud or in China). However, STS is regional in the sense that STS endpoints exist in each AWS region. This means that all of your clients in every region (within the same AWS partition) can authenticate using a single STS region configured in Vault. They just all need to know to generate login requests for the region that the Vault server uses.

The way Vault does IAM authentication, the only impact the AWS region has on Vault clients is on the content of the login request sent to the Vault server. In the Vault IAM authentication login flow, the client doesn't communicate with STS at all. Rather, the Vault server is what communicates with STS. So, to minimize network latency, I would recommend choosing the STS region that is closest to your Vault server. In the example you gave, your Vault clients in us-west-2 would use their credentials to generate a Vault login request, send that to the Vault server in us-east-1, and then the Vault server would send that to STS in us-east-1, which would then respond to your Vault server and that would respond back to your clients. That's only a single hop across the US. If you chose an STS region of us-west-2 for your Vault server in us-east-1, then your clients in us-west-2 would send the login request to your Vault server in us-east-1, which would then send the request to the STS endpoint back in us-west-2, which would respond back to Vault in us-east-1, which would respond to your client in us-west-2, resulting in two hops across the US.

If for some reason you must use STS endpoints in different regions, as Vault is currently configured today, you would need to have multiple mounts of the AWS auth method and configure the appropriate region in each.

Hope this helps!

--Joel

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/544dd518-a625-46a2-b970-17f7a5fac832%40googlegroups.com.

Srinivas Anant

unread,

Jul 30, 2020, 5:32:40 AM7/30/20

to Vault

Hi Joel,

I am running into the same issue, I have added the sts_endpoint in the aws auth backend:

```

Key Value

--- -----

access_key n/a

endpoint n/a

iam_endpoint n/a

iam_server_id_header_value test.vault.com

max_retries -1

sts_endpoint https://sts.us-gov-west-1.amazonaws.com

sts_region us-gov-west-1

```

This is the role I am using to login:

```

Key Value

--- -----

allow_instance_migration false

auth_type iam

bound_account_id []

bound_ami_id []

bound_ec2_instance_id <nil>

bound_iam_instance_profile_arn []

bound_iam_principal_arn [redacted]

bound_iam_principal_id [redacted]

bound_iam_role_arn []

bound_region []

bound_subnet_id []

bound_vpc_id []

disallow_reauthentication false

inferred_aws_region us-gov-west-1

inferred_entity_type ec2_instance

max_ttl 800h

policies [vault-agent]

resolve_aws_unique_ids true

role_id redacted

role_tag n/a

token_bound_cidrs []

token_explicit_max_ttl 0s

token_max_ttl 800h

token_no_default_policy false

token_num_uses 0

token_period 0s

token_policies [vault-agent]

token_ttl 0s

token_type default
```

And the vault env is running in AWS dev gov in the us-gov-west-1 region.

Am I missing any other configuration setting here?

Thanks,
Anant

Srinivas Anant

unread,

Jul 30, 2020, 5:38:49 AM7/30/20

to Vault

And the command I am using to login is this

vault login -method=aws header_value=test.vault.com role=vault-agent-iam -region=us-gov-west-1

Error I get is this
```

* error making upstream request: received error code 403 from STS: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">

<Error>

<Type>Sender</Type>

<Code>SignatureDoesNotMatch</Code>

<Message>Credential should be scoped to a valid region, not 'us-east-1'. </Message>

</Error>

</ErrorResponse>

```

Reply all

Reply to author

Forward