AWS backend - cleanup users NoSuchEntityException

[Andrew Backhouse]

Hello Vault Community,

Background

We use Vault for temporary AWS access amongst other things. Previously, we had Vault create IAM users then we attached Console Login permissions to this IAM user. We realised this caused Vault to fail at revoking the IAM user created because of the Console Login blocks it as a dependency. As an iteration, we have successfully moved over to using AWS STS for temporary AWS access using Vault to handle leases. 

Vault version running: 0.7.3

The problem

Upon investigating our AWS CloudTrail logs we see;

errorCode      NoSuchEntityException

errorMessage The user with name vault-ldap-foo-automation-1459260091-9947 cannot be found.

eventName     ListGroupsForUser

eventSource     iam

eventType     AwsApiCall

eventVersion     1.02

requestParameters.userName     vault-ldap-foo-automation-1459260091-9947

sourceIPAddress     ***{our Vault IP}***

userAgent     aws-sdk-go/1.8.34 (go1.8.3; linux; amd64)

userIdentity.arn     arn:aws:iam::{aws-account-number}:user/vault

In fact, we see a lot of these events for users of the previous IAM setup. 

This tells me. Vault is still trying to revoke the IAM users it created before we moved to STS. However, these IAM users have been deleted in AWS so it cannot clean up what it can't find. I believe it's stuck in a loop. 

Attempts to fix

A)

I've tried doing a force revoke on the aws backend's under the path; "aws/test/creds" 

This returned ok in the CLI. However, looking at the Vault logs I see;

[WARN ] revocation from the backend failed, but in force mode so ignoring: error=failed to revoke entry: resp:(*logical.Response)(nil) err:NoSuchEntity: The user with name vault-ldap-foo-automation-1459260091-9947 cannot be found.

The above has not stopped the logs from CloudTrail logs. 

B)

I have looked within; 

/sys/leases/lookup/auth/ldap/login/foo

This lists a lot of lease id's for this user. But, I'm not convinced if this is the right place to investigate further.

C)

Looking through some of the Vault code, this looks like the place of the API call within Vault when attempting to rollback (same as revoke?) an IAM user.  https://github.com/hashicorp/vault/blob/master/builtin/logical/aws/path_user.go#L65

Call to help

I'm looking for ideas or potentially something I've missed or if you think is a bug in Vault? Any help appreciated. 

Many thanks,

Andrew

[Jeff Mitchell]

Hi Andrew,

Step A is the right way to go. I wonder if the issue is that the AWS SDK is retrying the request? I'm not really sure, since the very next thing after that line printed as a warning to the logs is to delete the lease from Vault's tracking. (https://github.com/hashicorp/vault/blob/master/vault/expiration.go#L399)

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/04aa6639-c761-46a3-a90b-196386c6daf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Backhouse]

Hi Jeff,

We have had the most success with Step A. Upon retrying the force-revoke Vault it completes much quicker that the first time doing it and results in no spike of Vault warnings in the log. However, it doesn't resolve the flood of CloudTrail logs trying to list IAM resources which no longer exist by Vault. 

The code you've linked looks to do as it should, I'm not sure what to do next to look into AWS SDK retrying. Happy to help with any investigation work or provide more information. 

Many thanks

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

[Chris Minton]

Hi all
I am seeing something very similar with aws/creds running Vault 0.8.3. A check every minute through the AWS CLI for ListGroupsForUser failing for already-revoked leases, and yet a number of leases remaining:

$ vault list /sys/leases/lookup/aws/creds/cred-endpoint

Keys

----

069ffa73-12cc-092a-607a-0ca3ed74b3fa

721a6490-ca9c-ef43-47b5-3a7c060cd1e7

79d1e8bb-5b27-6fd6-99e5-7060ece6d35c

900a2d45-6fef-5e0b-71e2-ce2f1cc64599

dc52a208-4c16-37f4-242b-dd90ac562711

f317e146-e35d-6e2e-16c0-f0d5e79e72d6

These leases are all already revoked and yet we continue to see failing checks on no longer extant IAM users in CloudTrail. Force-revoke does not remove these lease keys, is that a possible issue?

Chris

[Jeff Mitchell]

Hi Chris,

Likely the leases weren't revoked, but rather the users were deleted via some other mechanism. So Vault is refusing to give up knowledge of the credential unless you force it to.

Note that forced revocation can't be used on individual leases until 0.10.2+

Best,

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/56226030-1ac5-4c31-84a8-e9054aa2bbcf%40googlegroups.com.