PKI backend - creating and deleting roles causing problems for creating credentials

[Steve Perkins]

I've been working on some Java wrapper code for the PKI backend restful API.  To run local automated tests, I'm starting up a dev Vault server instance and issuing the following CLI commands to enable the backend and generate an internal cert:

vault mount -path=pki pki

vault write pki/root/generate/internal common_name=myvault.com ttl=99h

I then run a series of various automated commands that rapidly create and delete a role (using "/v1/pki/roles"), and issue a cert (using "/v1/pki/issue").

When I run the same steps manually, at human-speed, they all work fine.  But when run in rapid succession in an automated test, the "/v1/pki/issue" command returns a 500 error with the following body:

{

  "errors": [

    "Error fetching CA certificate: stored CA information not able to be parsed"

  ]

}

At this point, something inside of Vault is trashed... and even slow manual calls to "/v1/pki/issue" fail with the same message.  Normal functionality resumes after I re-run the second CLI command to regenerate another internal cert.

Any ideas on what might be causing this behavior?  When creating or deleting a role through the PKI backend endpoint, does something within Vault need time to "settle" before you attempt to issue a certificate for that role?

Thanks!

[Jeff Mitchell]

Hi Steve,

Sorry you're having this issue. There should be no way for those endpoints to affect the CA cert. A few questions:

* Can you provide the logs? There might be more info. It'd also be good to know your config, which will be printed out at the beginning
* Any chance you can provide the test code?
* If you restart Vault, do things return to normal or stay screwed up?

It might be best to open an issue on GH to track this.

Thanks,
Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/3fccf796-eeed-4b1f-8986-7758acd781df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Perkins]

Thanks, Jeff.  I'll open a GH issue, but all of the test code is publicly visible in this GitHub repo (in the "pki" branch):

https://github.com/BetterCloud/vault-java-driver/tree/pki

The relevant test classes are under the "src/test-integration/java/com/bettercloud/vault/api/pki" directory.  The setup steps I'm using are spelled up verbatim in:

https://github.com/BetterCloud/vault-java-driver/blob/pki/src/test-integration/README.md

Since I'm running a local dev server, I assume that console output in the Vault terminal window counts as "logs".  The messages tend to trickle in unpredictable, not in real-time as the tests are running, but here is some recent output:

2016/05/03 18:29:19 [INFO] expire: revoked 'auth/token/renew-self/6877d685bbf5a172362712707ff6d66a1ebabacf'

2016/05/03 18:29:19 [INFO] expire: revoked 'auth/token/create/6877d685bbf5a172362712707ff6d66a1ebabacf'

2016/05/03 18:29:46 [INFO] expire: revoked 'auth/token/create/b3db36849aa584c57295a4d218a053fd8acb6a21'

2016/05/03 18:29:46 [INFO] expire: revoked 'auth/token/renew-self/b3db36849aa584c57295a4d218a053fd8acb6a21'

2016/05/03 19:12:53 [ERR] expire: failed to revoke 'pki/issue/testRole/176fec84-c89c-8a92-b915-2199c70da16b': failed to revoke entry: Error encountered during CRL building: Error fetching CA certificate: stored CA information not able to be parsed

2016/05/03 19:13:03 [ERR] expire: failed to revoke 'pki/issue/testRole/176fec84-c89c-8a92-b915-2199c70da16b': failed to revoke entry: Error encountered during CRL building: Error fetching CA certificate: stored CA information not able to be parsed

2016/05/03 19:13:23 [ERR] expire: failed to revoke 'pki/issue/testRole/176fec84-c89c-8a92-b915-2199c70da16b': failed to revoke entry: Error encountered during CRL building: Error fetching CA certificate: stored CA information not able to be parsed

2016/05/03 19:14:03 [ERR] expire: failed to revoke 'pki/issue/testRole/176fec84-c89c-8a92-b915-2199c70da16b': failed to revoke entry: Error encountered during CRL building: Error fetching CA certificate: stored CA information not able to be parsed

2016/05/03 19:15:23 [ERR] expire: failed to revoke 'pki/issue/testRole/176fec84-c89c-8a92-b915-2199c70da16b': failed to revoke entry: Error encountered during CRL building: Error fetching CA certificate: stored CA information not able to be parsed

2016/05/03 19:16:35 [INFO] expire: revoked 'auth/token/create/eb1f0d1f398cc6adecab529712b58bdf60925077'

2016/05/03 19:17:20 [INFO] expire: revoked 'auth/token/create/629d1ebf58f1adb2f91370913978af768287b133'

2016/05/03 19:17:33 [INFO] expire: revoked 'auth/token/create/79f241f9a1eb082171ab8ff47ef450781ec1019d'

2016/05/03 19:18:03 [ERR] expire: failed to revoke 'pki/issue/testRole/176fec84-c89c-8a92-b915-2199c70da16b': failed to revoke entry: Error encountered during CRL building: Error fetching CA certificate: stored CA information not able to be parsed

2016/05/03 19:20:36 [INFO] expire: revoked 'auth/token/create/5f7ff032c21bdb219755b8916c692fe030cfc8b3'

2016/05/03 19:22:22 [INFO] expire: revoked 'auth/token/create/bd4feb3bba5616e33df86cfd1f48ab3c56019151'

2016/05/03 19:23:23 [ERR] expire: maximum revoke attempts for 'pki/issue/testRole/176fec84-c89c-8a92-b915-2199c70da16b' reached

2016/05/03 19:26:20 [INFO] expire: revoked 'auth/token/create/6cc4eecbf5df6abcaf3bc1434a5c57212b2a9610'

2016/05/03 19:27:56 [INFO] expire: revoked 'auth/token/create/02542d327918b13808686d5e683c02b28ab09529'

2016/05/03 19:28:20 [INFO] expire: revoked 'auth/token/renew-self/f391bb49c180cd8f1cf2afc8e0d5a769b3d791c3'

2016/05/03 19:28:20 [INFO] expire: revoked 'auth/token/create/f391bb49c180cd8f1cf2afc8e0d5a769b3d791c3'

2016/05/03 19:28:42 [INFO] expire: revoked 'auth/token/create/db7cc237b94b90468a3885fbd7a404218fb6f2c0'

2016/05/03 19:28:59 [INFO] expire: revoked 'auth/token/create/e2110a705c855fc7f51397eb354d84025736e8b0'

2016/05/03 19:29:26 [INFO] expire: revoked 'auth/token/create/7cab98774934c4ebc162be8995e7bbb767a15de4'

You received this message because you are subscribed to a topic in the Google Groups "Vault" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vault-tool/8yXMEPPQDoc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GEUwcECDLGnEXrmbgATzUqPQfYnZNA8vyk87eqeeDcmpA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Steve Perkins | Software Architect | m. 404-997-3018

3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 

Subscribe to the BetterCloud Monitor - Get IT delivered to your inbox

[Jeff Mitchell]

Oh -- that's the info I needed. It's running in dev mode. The inmem storage used in dev mode is explicitly not safe for use like this -- it uses a map under the hood and has no locking around it, and maps can get corrupt in all sorts of weird ways by concurrent reading/writing, even to keys not explicitly being written to.

There are some locking inmem storages that should work but aren't exposed now as they're used for tests that have concurrency. I'll raise an issue internally -- maybe we should just default to that in dev mode.

So you can wait for that to appear in a branch/master (will keep you updated), or you can use a non dev-mode workflow and switch to a different backend. I think that should fix your problem!

--Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAC37mywsHy88q8%3Dk4MH%2B_5m628Qr7Mgo3Ast5zmipmNjPncyAg%40mail.gmail.com.

[Jeff Mitchell]

A few updates:

First, it's not the inmem storage; I forgot that there are two inmem
storages. One is a physical one, and the other is used mostly for
testing. The physical inmem storage properly locks.

Second, I took a look at your code, and I think I know what's going
on. In addition to writing/deleting roles and issuing certificates, I
see a call in there to pki/intermediate/generate (in
testWriteWithContentReturned).

Each PKI mount can only contain a single CA certificate. From the docs
for the endpoint:

"Generates a new private key and a CSR for signing. If using Vault as
a root, and for many other CAs, the various parameters on the final
certificate are set at signing time and may or may not honor the
parameters set here. This will overwrite any previously existing CA
private key..."

I think what's happening here is that somewhere along the way that
function is called. When it is, a CSR is being returned (but ignored)
and the cert/key are being overwritten. The normal workflow is to have
one mount be the root CA and another mount be the intermediate. You
call pki/root/generate to generate the root on the root mount -- this
is likely a very long-lived cert so with a high max TTL -- then you
call pki/intermediate/generate to generate the intermediate CSR on the
intermediate mount, sign the CSR via the root CA on the root mount,
then use pki/intermediate/set-signed to upload the signed cert.

I can try to find a way to make this error clearer, but it's basically
saying, "I don't have both a certificate and private key for a CA
stored right now". This makes sense if they've been ovewritten via a
call to pki/intermediate/generate.

Does that fit into your mental model of what's going on in the tests?

Best,
Jeff

[Steve Perkins]

Thank you so much!  That test you are referencing came from a community-submitted pull request, that I merged prior to even starting on the PKI code.  I had forgotten about it when I started work on the PKI wrapper.  Your explanation makes a lot of sense.

I removed that particular test (that functionality already has coverage elsewhere), and the problem has been completely resolved.  Thanks for checking it out so thoroughly.  Really above and beyond the call of duty, Jeff!

Steve

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to a topic in the Google Groups "Vault" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vault-tool/8yXMEPPQDoc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GFdbi65O00o%2BRQNXwp46R69jErrKEKXQ3Fv4ikVgk7j7Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Jeff Mitchell]

No problem. Glad it's working!

--Jeff

You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAC37myxXDacHS3Nr8t%3DfKrG%2BVXWODzq7JpFKkSTDMzo0%3Dy%2BdYA%40mail.gmail.com.