Problems with cross-account aws-ec2 auth

[Alex Gottschalk]

Hi All,

I'm setting up vault 0.7.0 to authenticate EC2 instances across a few AWS accounts.  This is working inside the account where vault is installed.  I have also confirmed using boto and ipython that the vault instance is able to get temporary credentials from STS, and use them to describe instances in the target account.

The problem I'm running into is trying to do a login operation from an instance in the target AWS account.  Here is what happens:

alex@sandbox-:~
$ vault write auth/aws-ec2/login role=sandbox pkcs7=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/pkcs7 | tr -d '\n') nonce='nonce'
Error writing data to auth/aws-ec2/login: Error making API request.

URL: PUT https://vault-1.<REDACTED>:8200/v1/auth/aws-ec2/login
Code: 400. Errors:

* failed to verify instance ID: no reservations found in instance description

I've tried running this operation with vault logging debug messages, but it did not generate any.

FWIW, here's how I verified cross-account access is working in general for the vault instance:

In [1]: import boto
In [2]: conn_sts = boto.connect_sts()
In [3]: s = conn_sts.assume_role('arn:aws:iam::<REDACTED>:role/Vault', 'test-role-session')
In [4]: conn_ec2 = boto.connect_ec2(aws_access_key_id=s.credentials.access_key, aws_secret_access_key=s.credentials.secret_key, security_token=s.credentials.session_token)
In [5]: len(conn_ec2.get_all_instances())
Out[5]: 458
In [6]: conn_ec2 = boto.connect_ec2()
In [7]: len(conn_ec2.get_all_instances())
Out[7]: 30

[Alex Gottschalk]

Also, here is the "sandbox" role in vault I'm trying to let authenticate:

😊   $ vault read auth/aws-ec2/role/sandbox
Key                               Value
---                               -----
allow_instance_migration          false
bound_account_id
bound_ami_id
bound_iam_instance_profile_arn    arn:aws:iam::<REDACTED>:instance-profile/Sandbox
bound_iam_role_arn
bound_region
bound_subnet_id
bound_vpc_id
disallow_reauthentication         false
max_ttl                           0
period                            0
policies                          [default]
role_tag
ttl                               0

And here is the IAM info from the sandbox instance's meta-data service:

alex@sandbox:~
$ curl -s http://169.254.169.254/latest/meta-data/iam/info
{
  "Code" : "Success",
  "LastUpdated" : "2017-03-24T22:36:17Z",
  "InstanceProfileArn" : "arn:aws:iam::<REDACTED>:instance-profile/Sandbox",
  "InstanceProfileId" : "<REDACTED>"
}

[Alex Gottschalk]

Can anyone at least point me in the direction of enabling better logging so I can see where the auth backend is failing?

[Joel Thompson]

Hi Alex,

The error message indicates that Vault's attempt to make a DescribeInstances API request filtering on the instance ID in the instance identity document is returning no results. It sounds like the /auth/aws-ec2/config/sts config hasn't been set up properly for your mount point. You need to ensure that, for the account ID that your instance is in, Vault is configured, using config/sts, to assume a role in that account. Can you post details about how you've configured that, e..g., the output of GET /auth/aws-ec2/config/sts/<account_id> (where account_id is the account your target instance is in)?

--Joel

On Mon, Mar 27, 2017 at 6:16 PM Alex Gottschalk <alex.go...@gmail.com> wrote:

Can anyone at least point me in the direction of enabling better logging so I can see where the auth backend is failing?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/872d3999-f02d-4581-be45-c7e6dfd658c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Gottschalk]

Thanks Joel!  Adding that configuration was all I needed to get things working.  It would be nice if that were referenced from https://www.vaultproject.io/docs/auth/aws-ec2.html#cross-account-access

--Alex

[Vishal Nayak]

Joel, thanks for chiming in.

Alex, I've just pushed changes to the website linking the sts
configuration from "cross account access". It should get updated in a
while.

Regards,
Vishal

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/cc2fc1a6-6384-4bb1-b0fb-b067b84d63c8%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
vn