curl command to list vault ids in consul backing store

[Julian Gamble]

Hi Everyone, 

I see a large number of keys under here:

http://consul.domain.com/ui/#/my-app/kv/vault/sys/expire/id/auth/app-id/login/

http://consul.domain.com/ui/#/my-app/kv/vault/sys/token/id

I'd like to get a list of the ids using curl. Is there standard way to do this?

I'm running vault 0.6.4 and consul 0.6.4. 

Cheers

Julian

[Jeff Mitchell]

Hi Julian,

If you have appropriate permissions you can use the sys/raw endpoint with the Consul (not Vault) paths - minus the Vault prefix - to introspect entries. That's the only way you can "list" token IDs; there is no other method, for safety.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/da765a01-9a42-463d-b8fb-125c893c1b66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Gamble]

Hi Jeff, 

Thanks for being so helpful. 

I'm running this to get the ids:

curl http://localhost:8500/v1/kv/vault/sys/token/id/?recurse |  jq '. | length'

5462

and this to get the expired IDs:

curl http://localhost:8500/v1/kv/vault/sys/expire/id/auth/app-id/login/?recurse |  jq '. | length'

5521

What is the process to clean these up?

Cheers

Julian

On Saturday, 6 May 2017 21:12:29 UTC+10, Jeff Mitchell wrote:

Hi Julian,

If you have appropriate permissions you can use the sys/raw endpoint with the Consul (not Vault) paths - minus the Vault prefix - to introspect entries. That's the only way you can "list" token IDs; there is no other method, for safety.

Best,

Jeff

On May 6, 2017 6:50 AM, "Julian Gamble" <julian...@gmail.com> wrote:

Hi Everyone, 

I see a large number of keys under here:

http://consul.domain.com/ui/#/my-app/kv/vault/sys/expire/id/auth/app-id/login/

http://consul.domain.com/ui/#/my-app/kv/vault/sys/token/id

I'd like to get a list of the ids using curl. Is there standard way to do this?

I'm running vault 0.6.4 and consul 0.6.4. 

Cheers

Julian

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

[Vishal Nayak]

Hi Julian,

If you are not creating too many tokens and still notice that there
are a lot many keys, it could be because of a bug, which was fixed
(for the most part) in 0.6.4. However, the fix only ensures that no
new dangling keys are added and it doesn't cleanup the already left
overs. So, there was an endpoint (auth/token/tidy) exposed to address
this.

Lately, we discovered that leases also needed some cleanup. In the
latest version of Vault, we have fixed some problems in the already
existing tidy API, enhanced the scope of the tidy operation to clean
up leases as well.

Its recommended that you first upgrade to 0.7.1 (wouldn't work
otherwise), invoke `sys/leases/tidy` and then `auth/token/tidy`. Note
that each of these API might take a long time to return depending on
the number of keys in the data store. You might also want to enable
trace level logging and observe the API progress.

Let us know how it goes.

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/6cf27d63-1d5f-4fff-84bc-4ae3cdd08430%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
vn

[Julian Gamble]

Hi Vishal and Jeff, 

I upgraded my vault to 0.7.2

I've run the following prior to running the tidy endpoints:

curl http://0.0.0.0:8500/v1/kv/vault/sys/expire/id/auth/app-id/login/?recurse |  jq '. | length'

6386->6385->6381

curl http://localhost:8500/v1/kv/vault/sys/token/id/?recurse |  jq '. | length'

6329->6328->6234

curl http://localhost:8500/v1/kv/vault/sys/token/accessor?recurse |  jq '. | length'

6329->6328->6324

I've then run the tidy endpoints:

curl -X POST -H "X-Vault-Token:$VAULT_TOKEN" --insecure https://0.0.0.0/v1/sys/leases/tidy

curl -X POST -H "X-Vault-Token:$VAULT_TOKEN" --insecure https://0.0.0.0/v1/auth/token/tidy

I've then run the checks again:

curl http://0.0.0.0:8500/v1/kv/vault/sys/expire/id/auth/app-id/login/?recurse |  jq '. | length'

6386-->6381

curl http://localhost:8500/v1/kv/vault/sys/token/id/?recurse |  jq '. | length'

6329-->6234

curl http://localhost:8500/v1/kv/vault/sys/token/accessor?recurse |  jq '. | length'

6329-->6324

These got about 4 or 5 down out of 6000 expired. 

Any recommendations?

Cheers

Julian

[Vishal Nayak]

Hi Julian,

Can you please provide the log summaries of the tidy operations?

Are we sure about 6000 entries being expired? The entries under
`expire/` are intended to not contain "expired" entries, instead it is
meant for those that will eventually get expired. Can you check the
validities of the entries there?

You could do it in 2 ways:
1) curl -X GET -H "X-Vault-Token:<token>"
http://0.0.0.0/v1/sys/raw/sys/expire/id/auth/app-id/login/<entry> | jq
.
2) vault write sys/leases/lookup lease_id=auth/app-id/login/<entry>

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/163e4a08-80cb-4d08-bd36-041c8912164a%40googlegroups.com.

[Julian Gamble]

Hi Vishal, 

Here is an example of an entry:

$ curl -X GET -H "X-Vault-Token:$VAULT_TOKEN" --insecure https://127.0.0.1:443/v1/sys/raw/sys/expire/id/auth/app-id/login/00387665cc0752c1e8243411229d3e4527fb5195 | jq .

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  1825  100  1825    0     0  18466      0 --:--:-- --:--:-- --:--:-- 18622

{

  "value": "{\"lease_id\":\"auth/app-id/login/00387665cc0752c1e8243411229d3e4527fb5195\",\"client_token\":\"8d4cd04a-1a9e-18b5-f988-b98da0d12082\",\"path\":\"auth/app-id/login\",\"data\":null,\"secret\":null,\"auth\":{\"lease\":2592000000000000,\"renewable\":true,\"InternalData\":{\"app-id\":\"go-agent\",\"user-id\":\"642d4648-4063-4beb-8ea4-54e519484762\"},\"DisplayName\":\"app-id-go-agent\",\"Policies\":[\"default\",\"go-agent\"],\"Metadata\":{\"app-id\":\"sha1:4301beadfd251789dc4c68af4466c414c2234230\",\"user-id\":\"sha1:d52f1e58a6630d94c49521e67f73a7aa75d347d4\"},\"ClientToken\":\"8d4cd04a-1a9e-18b5-f988-b98da0d12082\",\"Accessor\":\"0faa2864-86cb-cc5a-f2ac-6d8b2a05b5bc\"},\"issue_time\":\"2017-04-21T02:41:41.535567371Z\",\"expire_time\":\"2017-05-21T02:41:41.535568088Z\",\"last_renewal_time\":\"0001-01-01T00:00:00Z\"}",

  "request_id": "a1c2e3e7-8fca-aeeb-f389-2b2d559f4757",

  "lease_id": "",

  "renewable": false,

  "lease_duration": 0,

  "data": {

    "value": "{\"lease_id\":\"auth/app-id/login/00387665cc0752c1e8243411229d3e4527fb5195\",\"client_token\":\"8d4cd04a-1a9e-18b5-f988-b98da0d12082\",\"path\":\"auth/app-id/login\",\"data\":null,\"secret\":null,\"auth\":{\"lease\":2592000000000000,\"renewable\":true,\"InternalData\":{\"app-id\":\"go-agent\",\"user-id\":\"642d4648-4063-4beb-8ea4-54e519484762\"},\"DisplayName\":\"app-id-go-agent\",\"Policies\":[\"default\",\"go-agent\"],\"Metadata\":{\"app-id\":\"sha1:4301beadfd251789dc4c68af4466c414c2234230\",\"user-id\":\"sha1:d52f1e58a6630d94c49521e67f73a7aa75d347d4\"},\"ClientToken\":\"8d4cd04a-1a9e-18b5-f988-b98da0d12082\",\"Accessor\":\"0faa2864-86cb-cc5a-f2ac-6d8b2a05b5bc\"},\"issue_time\":\"2017-04-21T02:41:41.535567371Z\",\"expire_time\":\"2017-05-21T02:41:41.535568088Z\",\"last_renewal_time\":\"0001-01-01T00:00:00Z\"}"

  },

  "wrap_info": null,

  "warnings": null,

  "auth": null

}

Given that this expire time may be in the future - is it possible to update the expire time of a lease to a point in time in the past?

Cheers

Julian

[Julian Gamble]

Hi Vishal, 

That one wasn't a great example - here is an example of a token (showing up in the startup logs with blank token, unable to expire errors). You can see the token has expired - but it still exists. 

Cheers

Julian

$ curl -X GET -H "X-Vault-Token:$VAULT_TOKEN" --insecure https://localhost/v1/sys/raw/sys/expire/id/auth/app-id/login/6488f9f0bb248c4d8cd0fe8dcf0f2f1900e036a4 | jq .

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  1821  100  1821    0     0  15224      0 --:--:-- --:--:-- --:--:-- 15302

{

  "value": "{\"lease_id\":\"auth/app-id/login/6488f9f0bb248c4d8cd0fe8dcf0f2f1900e036a4\",\"client_token\":\"7d2d8d72-b46d-5a68-b992-6e6488303dcb\",\"path\":\"auth/app-id/login\",\"data\":null,\"secret\":null,\"auth\":{\"lease\":2592000000000000,\"renewable\":true,\"InternalData\":{\"app-id\":\"go-agent\",\"user-id\":\"642d4648-4063-4beb-8ea4-54e519484762\"},\"DisplayName\":\"app-id-go-agent\",\"Policies\":[\"default\",\"go-agent\"],\"Metadata\":{\"app-id\":\"sha1:4301beadfd251789dc4c68af4466c414c2234230\",\"user-id\":\"sha1:d52f1e58a6630d94c49521e67f73a7aa75d347d4\"},\"ClientToken\":\"7d2d8d72-b46d-5a68-b992-6e6488303dcb\",\"Accessor\":\"77fa8cb1-7edc-80e6-e8ff-9c87533feeae\"},\"issue_time\":\"2017-04-04T07:14:15.46872081Z\",\"expire_time\":\"2017-05-04T07:14:15.46872168Z\",\"last_renewal_time\":\"0001-01-01T00:00:00Z\"}",

  "request_id": "3a558cdd-38dc-4562-ba5a-af2a1d81d62a",

  "lease_id": "",

  "renewable": false,

  "lease_duration": 0,

  "data": {

    "value": "{\"lease_id\":\"auth/app-id/login/6488f9f0bb248c4d8cd0fe8dcf0f2f1900e036a4\",\"client_token\":\"7d2d8d72-b46d-5a68-b992-6e6488303dcb\",\"path\":\"auth/app-id/login\",\"data\":null,\"secret\":null,\"auth\":{\"lease\":2592000000000000,\"renewable\":true,\"InternalData\":{\"app-id\":\"go-agent\",\"user-id\":\"642d4648-4063-4beb-8ea4-54e519484762\"},\"DisplayName\":\"app-id-go-agent\",\"Policies\":[\"default\",\"go-agent\"],\"Metadata\":{\"app-id\":\"sha1:4301beadfd251789dc4c68af4466c414c2234230\",\"user-id\":\"sha1:d52f1e58a6630d94c49521e67f73a7aa75d347d4\"},\"ClientToken\":\"7d2d8d72-b46d-5a68-b992-6e6488303dcb\",\"Accessor\":\"77fa8cb1-7edc-80e6-e8ff-9c87533feeae\"},\"issue_time\":\"2017-04-04T07:14:15.46872081Z\",\"expire_time\":\"2017-05-04T07:14:15.46872168Z\",\"last_renewal_time\":\"0001-01-01T00:00:00Z\"}"

  },

  "wrap_info": null,

  "warnings": null,

  "auth": null

}

[Vishal Nayak]

Hi Julian,

This looks strange. It looks like an expired lease is not getting
cleaned up. When Vault comes up, the lease restoration logic will
iterate through all the leases and loads them up. It should revoke
those leases that are expired and ideally the tidy operation should
never need to bother about expired leases.

For the lease you mentioned above, could you please look for log
entries of the following format? It could give some visibility on the
type of error encountered while revoking it during startup.
"expire: failed to revoke lease:
lease_id=auth/app-id/login/6488f9f0bb248c4d8cd0fe8dcf0f2f1900e036a4
error=<err>"

Regards,
Vishal

> https://groups.google.com/d/msgid/vault-tool/7b98a781-264a-4c23-b99d-7d4f99693511%40googlegroups.com.

[Julian Gamble]

Hi Vishal, 

I can confirm I am seeing those "failed to revoke lease" entries in the vault logs. 

Cheers

Julian