AWS IAM user cleanup

[Hridyesh Pant]

Hi,

Vault is creating IAM user entry, whenever user is creating new role using vault or generating dynamic credential.

e.g.

arn:aws:iam::****:user/vault-token-s3_bucket_role-1456567667-3300

Even through i set lease_duration is for 24h, user entry is not getting deleted from AWS IAM user section.

is there way IAM user get deleted automatically after lease_duration expire?

--Thanks

Hridyesh

[Jeff Mitchell]

Hi Hridyesh,

It should be getting cleaned up automatically. Can you share your
configuration (without any sensitive details) and the output you're
seeing when you fetch credentials?

Also, what version of Vault are you using?

Thanks,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/7388ff8c-2e1d-4a48-93c5-5c080df0f7a9%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Hridyesh Pant]

Hi Jeff,

curl -H "X-Vault-Token: adcd" -X GET https://vault.com/v1/test/creds/s3_bucket_role

{"lease_id":"test/creds/s3_bucket_role/****","renewable":true,"lease_duration":86400,"data":{"access_key":"***","secret_key":"***","security_token":null},"warnings":null,"auth":null}

The lease duration is showing 86400 ,which is 24 hr, but i can still see the user which was created in AWS IM section.

vault-token-s3_bucket_role-***    0   N/A    1 active       2016-03-03 14:56 UTC+1000

i can wait today to see if this user is getting deleted , but i suspect about it as it already crossed 24h interval.  

--Thanks

Hridyesh

[Hridyesh Pant]

vault -version

Vault v0.5.0

[Hridyesh Pant]

No Jeff ,user is not getting deleted . AM i missing any configuration ?

--Thanks

Hridyesh

[Jeff Mitchell]

Hi Hridyesh,

I just followed the AWS example at
https://www.vaultproject.io/docs/secrets/aws/index.html -- I created a
role with an inline policy, set the default/maximum TTL on the mount
to 30s for ease of testing, and then fetched credentials. A little
after 30 seconds later the lease was revoked and the user was removed
from AWS:

$ vault read aws/creds/deploy
Key Value
lease_id aws/creds/deploy/fffc90d7-a6d2-7c8c-4a91-7f8209172295
lease_duration 30
lease_renewable true
access_key AKIAITN663G6VMFFH7RQ
secret_key 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
security_token <nil>

(30 seconds later):
$ vault renew aws/creds/deploy/fffc90d7-a6d2-7c8c-4a91-7f8209172295
Renew error: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/renew/aws/creds/deploy/fffc90d7-a6d2-7c8c-4a91-7f8209172295
Code: 400. Errors:

* lease expired

and in the Vault logs:
2016/03/04 09:11:29 [ERR] sys: renew
'aws/creds/deploy/fffc90d7-a6d2-7c8c-4a91-7f8209172295' failed: lease
expired
2016/03/04 09:11:37 [INFO] expire: revoked
'aws/creds/deploy/fffc90d7-a6d2-7c8c-4a91-7f8209172295'

And indeed the user is gone from AWS.

Are you seeing anything relevant in the Vault logs? When you click on
a user you see within IAM, does it actually show you its attached
inline policies or does it show you errors? IAM is eventually
consistent, and I have seen cases where the user is deleted but it
takes a bit for the user to actually be removed from the console.

Thanks,
Jeff

> https://groups.google.com/d/msgid/vault-tool/37a19a8a-36a6-4b1b-818f-2404db139212%40googlegroups.com.

[Hridyesh Pant]

Hi Jeff,

below are the steps which i performed.

1. vault mount-tune  -default-lease-ttl=30s -max-lease-ttl=30s test  //where test is aws type

2. vault mounts

Path        Type       Default TTL  Max TTL  Description

cubbyhole/  cubbyhole  n/a          n/a      per-token private secret storage

prod/       aws        system       system

secret/     generic    system       system   generic secret storage

sys/        system     n/a          n/a      system endpoints used for control, policy and debugging

test/       aws        30           30

3. curl -H "X-Vault-Token: 7f2eeaf6-33b1-31be-0670-650e815f85ff" -X GET https://***/v1/test/creds/s3_bucket_role

{"lease_id":"test/creds/s3_bucket_role/a9776586-b0ed-d300-ba0e-2aa74eef0342","renewable":true,"lease_duration":30,"data":{"access_key":"**","secret_key":"**","security_token":null},"warnings":null,"auth":null}

4. after 30sec i am also getting same error

vault renew test/creds/s3_bucket_role/a9776586-b0ed-d300-ba0e-2aa74eef0342

Renew error: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/renew/test/creds/s3_bucket_role/a9776586-b0ed-d300-ba0e-2aa74eef0342

Code: 400. Errors:

* lease expired

5. but when i looked in AWS UI console , i can still see that user and able to see inline policy as well with no error.

--Thanks

Hridyesh

[Jeff Mitchell]

Hi Hridyesh,

Due to the AWS backend's need to handle rollback, it can take a short
time after the lease expires to actually be revoked (usually 5-10
seconds). Also, IAM is eventually consistent, and it can take a longer
time until the user actually disappears from the AWS console. If you
look immediately after the lease expires you're very likely to see the
user. Is the user still there after refreshing a few minutes later?

Thanks,
Jeff

> https://groups.google.com/d/msgid/vault-tool/9912d09c-cd8a-4fd7-a417-bf34e869a23f%40googlegroups.com.

[Hridyesh Pant]

Hi Jeff,

The user is not getting deleted even after 2 days and still present.

But when i run vault renew ,it is saying lease not found. 

vault renew test/creds/s3_bucket_role/a9776586-b0ed-d300-ba0e-2aa74eef0342

Renew error: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/renew/test/creds/s3_bucket_role/a9776586-b0ed-d300-ba0e-2aa74eef0342

Code: 400. Errors:

* lease not found or lease is not renewable

--Thanks

Hridyesh

[Hridyesh Pant]

Also did you checked the user is getting actually deleted from AWS UI console during your testing?

--Thanks

Hridyesh

[vishal nayak]

Hi Hridyesh,

I noticed the scheme difference in your previous post.

curl -H "X-Vault-Token: 7f2eeaf6-33b1-31be-0670-650e815f85ff" -X GET https://***/v1/test/creds/s3_bucket_role

URL: PUT http://127.0.0.1:8200/v1/sys/renew/test/creds/s3_bucket_role/a9776586-b0ed-d300-ba0e-2aa74eef0342

By any chance, are you pointing to different Vault instances?

Regards,

Vishal

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/c7fa0816-d05a-43ae-8629-fae9c34a64c4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

vn

[Hridyesh Pant]

Hi Vishal,

May be i pasted some wrong scheme, below are my steps. Please let me know if i am missing some configuration.

1. vault mount-tune  -default-lease-ttl=30s -max-lease-ttl=30s test

2. vault mounts

Path        Type       Default TTL  Max TTL  Description

cubbyhole/  cubbyhole  n/a          n/a      per-token private secret storage

secret/     generic    system       system   generic secret storage

sys/        system     n/a          n/a      system endpoints used for control, policy and debugging

test/       aws        30           30

3. now getting key which should expire in 30 sec.

curl -H "X-Vault-Token: 59b812d2-dcd9-0b7b-33be-56188feee3a1" -X GET https://***/v1/test/creds/s3_bucket_role

{"lease_id":"test/creds/s3_bucket_role/11bcddb6-7311-ca6f-52ff-c09f574f3a84","renewable":true,"lease_duration":30,"data":{"access_key":"**","secret_key":"**","security_token":null},"warnings":null,"auth":null}

4. 

vault token-lookup 59b812d2-dcd9-0b7b-33be-56188feee3a1 

Key           Value

creation_time 1.457414927e+09

creation_ttl 86400

display_name token

id           59b812d2-dcd9-0b7b-33be-56188feee3a1

meta         <nil>

num_uses     0

orphan       false

path         auth/token/create

policies     [default user_policy]

ttl           86135

5. after 30 sec ,when i try to renew the id , it say lease expired, so ideally this user should delete from AWS UI IAM's user section.

vault renew  test/creds/s3_bucket_role/11bcddb6-7311-ca6f-52ff-c09f574f3a84

Renew error: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/renew/test/creds/s3_bucket_role/11bcddb6-7311-ca6f-52ff-c09f574f3a84

Code: 400. Errors:

* lease expired

6.

after waiting 1 hour , i am still seeing that user present in AWS IAM user section ,so it is not deleting.

So i am not sure how to cleanup user, can you try if user is actually getting deleted from UI.

--Thanks

Hridyesh

[Hridyesh Pant]

except step 3 ,all command i am running in system where Vault server is configured.

i have placed nginx on top of vault ,so user can get key by calling nginx end point and forward request to vault server which is running 127.0.0.1.

i dont think this will cause issue to remove user from AWS IAM's user section?

--Thanks

Hridyesh

[Jeff Mitchell]

Hi Hridyesh,

What is in the Vault logs? Expirations are logged; if Vault is having trouble performing the revocation in AWS that error should be getting logged too.

Best,
Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/45c2190f-ac13-486f-abcf-dd111921f363%40googlegroups.com.

[Hridyesh Pant]

You are right.

i am seeing below error :

016/03/08 06:34:22 [ERR] expire: failed to revoke 'test/creds/s3_bucket_role/9cdb4d35-3118-4640-fa75-b4dc5fc9c796': failed to revoke entry: AccessDenied: User: arn:aws:iam::**:user/vault-server is not authorized to perform: iam:ListAttachedUserPolicies on resource: user vault-token-s3_bucket_role-1457414947-6258

I have added iam:ListAttachedUserPolicies in permission now , will see if that works , but i think https://www.vaultproject.io/docs/secrets/aws/ document need to updated then, if iam:ListAttachedUserPolicies i s required.

The root credentials need permission to perform various IAM actions. These are the actions that the AWS secret backend uses to manage IAM credentials. Here is an example IAM policy that would grant these permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateAccessKey",
        "iam:CreateUser",
        "iam:PutUserPolicy",
        "iam:ListGroupsForUser",
        "iam:ListUserPolicies",
        "iam:ListAccessKeys",
        "iam:DeleteAccessKey",
        "iam:DeleteUserPolicy",
        "iam:RemoveUserFromGroup",
        "iam:DeleteUser"
      ],
      "Resource": [
        "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/vault-*"
      ]
    }
  ]
}

[Hridyesh Pant]

yes after adding "iam:ListAttachedUserPolicies" , user are getting deleted.

Thanks a lot Jeff, is it possible to update below section in https://www.vaultproject.io/docs/secrets/aws/  doc as well ?

--Hridyesh

[Jeff Mitchell]

Glad it's working!

I have added it to the doc and it's deploying to the docs site now.

--Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/6fbaa521-7384-43c2-8c96-43cf2aa7a2fe%40googlegroups.com.

[Hridyesh Pant]

Great . Thanks Jeff