Upgrade broke backend

[george....@mx.com]

Hi,

I'm running vault 0.6.2 on a postgresql storage backend and zookeeper HA backend. I upgraded to 0.7.0 by upgrading two of my three vault nodes, unsealing them and then stepping down the last one. On step down, the following error came up in one of the upgraded nodes:

2017/04/20 14:55:08.319231 [ERROR] core: post-unseal setup failed: error=expiration state restore failed: failed to read lease entry: pq: unnamed prepared statement does not exist

2017/04/20 14:55:08.552974 [INFO ] core: acquired lock, enabling active operation

2017/04/20 14:55:08.563094 [ERROR] core: error performing key upgrades: error=error scheduling upgrade cleanup: failed to list upgrades: pq: bind message supplies 1 parameters, but prepared statement "" requires 2

2017/04/20 14:55:08.567253 [INFO ] core: vault is sealed

After which, I shut down all of the vault nodes and now I'm getting: 2017/04/20 15:45:13.429732 [INFO ] core: security barrier not initialized

I have had this issue before and just had to restore from backup to fix it as I think that the 'core/keyring' entry was obliterated from the database.

Once again, I search for the 'core/keyring' entry, but cannot find it in the database.

I rolled back the vault nodes to 0.6.2 and still getting the core: security barrier not initialized error.

I'm not sure what is going on here. It looks to me like the upgrade lost my keyring. Am I crazy?

Thanks in advance.

George

[Jeff Mitchell]

On Thu, Apr 20, 2017 at 6:00 PM, <george....@mx.com> wrote:

Hi,

I'm running vault 0.6.2 on a postgresql storage backend and zookeeper HA backend. I upgraded to 0.7.0 by upgrading two of my three vault nodes, unsealing them and then stepping down the last one. On step down, the following error came up in one of the upgraded nodes:

2017/04/20 14:55:08.319231 [ERROR] core: post-unseal setup failed: error=expiration state restore failed: failed to read lease entry: pq: unnamed prepared statement does not exist

2017/04/20 14:55:08.552974 [INFO ] core: acquired lock, enabling active operation

2017/04/20 14:55:08.563094 [ERROR] core: error performing key upgrades: error=error scheduling upgrade cleanup: failed to list upgrades: pq: bind message supplies 1 parameters, but prepared statement "" requires 2

2017/04/20 14:55:08.567253 [INFO ] core: vault is sealed

My guess is that you need to go through https://www.vaultproject.io/docs/configuration/storage/postgresql.html and make sure the tables/indexes/procedures are in place -- note that different versions of postgres require some different setup. 

I rolled back the vault nodes to 0.6.2 and still getting the core: security barrier not initialized error.

Not sure about this, but potentially the later code made some changes to the schema but were unable to make all?

 

I'm not sure what is going on here. It looks to me like the upgrade lost my keyring. Am I crazy?

Honestly not sure and I have no experience with the postgres backend, but possibly it's just not being indexed properly. 

Best,

Jeff

[george....@mx.com]

Thanks once again for such a quick response.

We went through the database and checked the indexing. I then rolled back to 0.6.2 and put the keyring back from a backup. I then did another upgrade to 2 of 3 nodes by shutting down vault, putting 0.7.0 in place and then starting it up and unsealing. As soon as I did a vault step-down on the last node, I got a bunch of stuff in the logs and the security barrier not initialized error. I checked the DB again and the keyring had once again been wiped out. I'm sure this is probably a postgresql specific error, so I'm digging into code now to find the issue. If anyone else has upgrade issues, lemme know. Thanks!

[Jeff Mitchell]

Do please let us know!

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/b9db31f3-3dba-4e12-8be8-31bcf5c098f0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[george....@mx.com]

We found why the keyring was being deleted. We are running pgbouncer in transaction pooling mode with postgresql.The pq lib for go is using unnamed prepared procedures and parameters, but these aren't being handled properly and one DELETE query would get applied with the parameters from another SELECT query, causing the keyring to be deleted. We switched the pgbouncer to session pooling mode and that should prevent wires from getting crossed. This was not directly related to the upgrade, but when two nodes were contending for leadership, a lot of queries started flying and caused the issue.

[Jeff Mitchell]

Thanks for getting back to us with the resolution!

If there is some change that can be made on the Vault side to prevent this please let us know or open a PR.

Best,

Jeff

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/e5878322-ba01-494c-8efa-38a981265e97%40googlegroups.com.