Vault reports as successfully unsealed, try to write secret, get "503 vault is sealed" error

[Amy Brown]

Good afternoon all,

I'm running vault with a file backend on ArchLinux.

I was able to write secrets successfully to vault earlier this afternoon. 

I have vault set to start up with systemd.  I had an ExecStartPost command in my service file to unseal vault but it didn't work, so I took it out. (This is just a demo, not production ready at all.)

Now, when I reboot, vault starts in sealed status. I unseal vault manually with three keys and it reports sealed=false.

Then, when I try to write, I get a 503 error that vault is sealed.

I've tried stopping the service on systemd, then rebooting. No luck. Not sure what to try next. 

[Michael Fischer]

Vault always starts up sealed.  Any logs to suggest that vault is stopping or restarting in the background between the time you unseal it and the time you try to write to it?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/051b89bc-b433-422c-8f0a-6e46af0a85ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Amy Brown]

At this point I can't even do vault audit-enable because of the "sealed" 503 error. I'll see if there's anything of interest in journalctl or /var/log, but I'm not hopeful. 

I may just blow away the directory, create a new empty one, and start from scratch. I don't have much data in there.

I'm not sure how stateful vault is between system shutdown/restart so I hope it would just recognize a new empty directory as its filesystem. Or maybe I'll point to another directory. 

Will also check the permissions on the directory I'm using as the file backend ... maybe something went amiss there. 

[Amy Brown]

It turns out that journalctl was my friend here. Due to my systemd service file settings, vault was restarting every couple of seconds. Type=notify and Restart=Always don't play well together, at least not for vault.

I'm now happily writing and reading secrets again even after a reboot. 

Onward with my multiple learning curves. Thanks so much to this community for responding to my queries. :-)

[Jeff Mitchell]

Hi Amy,

Was just responding to your earlier email when I noticed this one
after it -- glad it's working! I was going to ask about logs, since it
sounded like something was restarting Vault between unseal and write.

To answer your previous question, Vault has a data store, and with the
file backend that's a directory. If you point it to an empty directory
it will just treat it as an uninitialized data store. (This does also
mean that you can point Vault instances at different data stores, or
backups of data stores.)

Best,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/8f175754-1698-4675-acc2-07b9f2c9f729%40googlegroups.com.