Vault-backup

[Niks]

Hi,

I don't want Vault HA as of now. Experimenting my use cases with Vault using "file" backend.

Use case: Vault server has an issue, worst case, I cannot use this vault any more. 

How do I back up Vault, Is there a way to backup this vault(before or after it stopped working)? So that another vault, on same or other machine can point to the copied backed data and be brought up?

Niks

[David Adams]

I haven't used the file backend, but I would assume you could stop the Vault server, then tar or zip the data directory and use it anywhere else. So long as the other place has the keys needed to unseal it.

For what it's worth, for pure experimentation, I've found it's really useful to just spin up  vault in dev mode (ie with the inmem backend) "vault server -dev". You can specify a fixed root token with "-dev-root-token-id <token>" so that it's consistent every time. And then I write a bash script to do all my setup--adding backends, users, policies, secrets, mounts, tunings, etc. Then it takes just a few seconds to start up, and I can play around. Then if it's messed up, I can just shut it off and start it back up and re-init back to where I was. Then as I add more things to test, I add those to the script. Then to work with others I can just share the setup script. Super handy.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/e4000fe4-60ac-4784-a027-962b3e34964d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

David Adams | Systems Administrator

E: dad...@instructure.com

[Niks]

Hi David,

Thanks a lot.

I am using production mode, but just experimenting on my machine.

I just tried. Shutting down my vault instance. Trying to mock use case: copy by backend storage file directory to another location and point it to a new Vault instance.

Issues:

1.  Start vault with following config

backend "file" {

   path = "<new location of storage>"

}

listener "tcp" {

 address = "127.0.0.1:8200"

 tls_disable = 1

}

getting error: Vault already initialised. 

2. To fix this I pointed to a new storage location(with no data). Init worked and gave me new set of keys. This creates the backend storage directory structure similar to previous one.

Now if I replace this storage directory with back up one, I suspect vault will have issues.

Need some pointers to back up Vault please

[David Adams]

You should only run "vault init" the very very first time you set up a vault data store. When you start the new server pointed at the new file location, you should only do the unseal steps, not init.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/7c5dace8-88d2-44d2-8f76-84cb2650ac3e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Niks]

Thats right. But I am trying to mock the use case where a new instance of vault, say in another machine is able to come up with this backed up storage backend.

If physical backend is deleted, isn't vault as good as a new instance?

On Tuesday, May 17, 2016 at 3:03:07 PM UTC+1, Niks wrote:

[David Adams]

What I meant to say is: if you have the vault files backed up, and restore them to a new machine, and then start Vault pointing to those data files, you do not run "vault init" again. You said you were "getting error: Vault already initialised." So I assume you are trying to run "vault init" But you should not do that if you are trying to use restored files.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/b0d21b5f-dc5f-445e-8696-cc4c547d1f61%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alex Ku]

(So we use the consul backend and I think the extra thing we did to the backup might apply to your case here)

Basically, when we backup the vault data from Consul (through the consul API), and we want to restore the data onto a new blank vault server, we had to remove a couple pieces of data from the backups, namely stuff under these two paths:

- core/lock

- core/leader

Now, I haven't looked at how the data is laid out in the filesystem with the file backend (though it's quick to try out, I know), I'd imagine that the data is likely laid out the same way and the concern around removing a couple things from the backup might apply.

Hope this helps.

[Jeff Mitchell]

On Tue, May 17, 2016 at 2:23 PM, Alex Ku <ale...@opower.com> wrote:
> (So we use the consul backend and I think the extra thing we did to the
> backup might apply to your case here)
>
> Basically, when we backup the vault data from Consul (through the consul
> API), and we want to restore the data onto a new blank vault server, we had
> to remove a couple pieces of data from the backups, namely stuff under these
> two paths:
> - core/lock
> - core/leader
>
> Now, I haven't looked at how the data is laid out in the filesystem with the
> file backend (though it's quick to try out, I know), I'd imagine that the
> data is likely laid out the same way and the concern around removing a
> couple things from the backup might apply.

It's laid out the same way!

Thanks for pointing this out -- this is indeed a requirement. And as I
said on the other thread, I strongly recommend atomic snapshots of the
file tree, then backing up the snapshot. Of course, stopping Vault,
taking a backup, and restarting it works fine too!

--Jeff

[Niks]

Thanks David,Alex,Jeff.

@David, 

My requirement is to bring up a new instance of Vault with backed up data(file backend as of now ). So I wanted to initialize Vault again(since I am playing around on same machine), Trying to make new instance of vault work with this backed up data.

You are right, for same instance of vault, we should not initialize again to use backed up data.

Alex,Jeff,David

For above requirement  :bring up a new instance of Vault with backed up data(file backend) . Primary function of Vault: encrypt/decrypt with transit backend.

File system storage lay out when vault is initialised/configured is as below:

auth,

      8a8e9681-adf2-1d70-033b-263286384638\struct\map\app-id

      8a8e9681-adf2-1d70-033b4684638462384\struct\map\user-id

core

  _audit

  _auth

  _keyring

  _master

 _mounts

 _seal-config

logical

  7c88be47-92b0-1e54-c92c-8790683648346834683\archive

     _demo1

    _vaultKey

    ***** demo1,vaultKey are my encryption keys from previous vault

sys

   expire

   policy

   token

------------------------------------

1) Can I bring up a new instance of vault by copying some of the folders from above to make transit backend work as before?

If yes, which ones?

2) I tried copying auth and logical(folders were copied after initializing new vault instance, else it was giving error, think it expects no data during init)

But when I try to read my policies created before, in new vault instance

>vault policies my-policy

Code: 400. Errors:

* failed to read policy: decryption failed: cipher: message authentication failed

Same gets listed though

>vault policies

default

my-policy

root

3) http://127.0.0.1:8200/v1/auth/app-id/login   end point gives error with backed up app-id,user-id

{

  "errors": [

    "invalid user ID or app ID"

  ]

}

4) Permission denied error when using old tokens

5) decrypt fails with ciphertext created from previous vault instance

{

  "errors": [

    "invalid ciphertext"

  ]

}

All use cases fail in this scenario. Am I doing something totally wrong? I have to prove this work else I cannot Use Vault at all since consul is not getting approved as it requires installation and set up whereas file system do not.

Will really appreciate any suggestion/workaround here please?

Niks

On Tuesday, May 17, 2016 at 3:03:07 PM UTC+1, Niks wrote:

[David Adams]

On Wed, May 18, 2016 at 3:55 AM, Niks <niharik...@gmail.com> wrote:

@David, 

So I wanted to initialize Vault again ... Trying to make new instance of vault work with this backed up data.

You are right, for same instance of vault, we should not initialize again to use backed up data.

To start vault with restored data you do not initialize it. It doesn't matter what machine you are on or whether you've used Vault on that machine or not. Initializing is the process of generating new encryption keys and writing an empty data structure to the backend storage. So if you point the Vault program to a config file that points to some restored files, you only need to unseal (using the original keys). Do not run `vault init` or call the equivalent API. It correctly fails, because the restored data was initialized when it was first created.

I've made a gist containing a bash script that sets up a file-backed Vault, inits and unseals it, writes some secrets, shuts it down, then "makes a backup" by copying the data directory of the original vault to a new location. It then creates a new config file, and starts a new instance of Vault pointed at the new data. It tries to run `vault init` but expects to fail (and it does), so it then moves on to unseal with the original keys, and reads the secrets and finds they are correctly restored.

See: https://gist.github.com/daveadams/c028c1019f9a9738cbe99d5ee9fa992b

1) Can I bring up a new instance of vault by copying some of the folders from above to make transit backend work as before?

If yes, which ones?

Copy everything to the new location. You can't copy just certain secrets to another instance of Vault by copying the backend storage. If you could, that would mean vault was completely insecure.

If you don't want to copy everything and only want to copy the transit config from one vault instance to another vault instance, you should just read the transit config out of one vault and write it into another vault using the vault API.

-dave

[Alex Ku]

Just to clarify further. You need to remove these from the backup only if you're using >1 instances of Vault (which I would think is predominantly the case for everyone):

- core/lock

- core/leader

If this is agreed upon from the kind Vault folks, I can push a PR to update the docs (since it wasn't clear to us and we had to experiment and test this out ourselves).

[Niks]

Thanks All for quick help.

I could make the files system backup work.

As a POC, I am currently working on only 1 vault server and making it work with another new Vault set up(from back up of filesystem) .

Big mistake I was doing was calling init on my new vault instance and copying the backed up files after init of new vault system.

Followed exactly what David mentioned and worked like a charm.

Thanks again !!!

Since I don't have a cluster set up, Don't have following in my backup

- core/lock

- core/leader

Yet to try cluster scenario.

Niks

On Tuesday, May 17, 2016 at 3:03:07 PM UTC+1, Niks wrote:

[Jeff Mitchell]

This would be neat, although I'm not sure where it should go. Most likely we need an actual backup section, with this as part of it. I'll take that as a TODO unless someone interested gets to it before me.

Best,
Jeff

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/257902d5-56e7-4448-baec-bb644ce64b1b%40googlegroups.com.

[Niks]

Thanks Jeff.

I will get back if I have any more issues. This was just a POC. Still trying to push folks in my team for consul.

Niks

On Tuesday, May 17, 2016 at 3:03:07 PM UTC+1, Niks wrote: