consul snapshot: Remote execution, backup

800 views
Skip to first unread message

Randy Fay

unread,
Dec 8, 2016, 1:01:10 PM12/8/16
to Consul
Experimenting with consul backups, I've found the "consul snapshot" save and restore to be excellent. 

I'm aware that consul enterprise contains automated snapshotting and remote storage of the snapshot, but don't want to do that at this point. Also aware that my cluster probably won't need this because I have 3 servers running and have been careful about how they're set up.

Does anybody have suggestions about:

* Remote triggering of the snapshot (doesn't seem to be an api?) - the context for me is a pod in a Kubernetes cluster
* Sending the resulting file off to an external location
* Gotchas with using the consul snapshot as a backup/restore technique?

Thanks,
-Randy

James Phillips

unread,
Dec 8, 2016, 1:21:51 PM12/8/16
to consu...@googlegroups.com
Hi Randy,

Snapshot and restore are available as an HTTP endpoint if you wanted
to create your own tool (this is the same API that the Consul
Enterprise tool uses, as well as the "consul snapshot" CLI commands).
Details are here:

https://www.consul.io/docs/agent/http/snapshot.html

You basically treat what you get from the snapshot endpoint as an
opaque blob and push it up to offsite storage. You can also restore
with this API, or using the "consul snapshot" CLI.

-- James
> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/consul-tool/c0d1ab2c-e87c-4faf-82aa-631667a5452b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Randy Fay

unread,
Dec 8, 2016, 1:33:33 PM12/8/16
to consu...@googlegroups.com
Thanks - I discovered the HTTP endpoint at just the wrong time *after* sending, and it seems to work fine, thanks.

Since consul snapshot is really quite excellent, a mention of it should probably be added to the Outage Recovery page, https://github.com/hashicorp/consul/blob/master/website/source/docs/guides/outage.html.markdown - When I get farther down the line I can do a PR there, but someone more expert in the tradeoffs might be better at this point.

Thanks,
-Randy

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CAGoWc06oYt5j4K-x0p-q%2BwW57yLkqKB5wnmzbXJ1yYC%2B3ZRccA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

James Phillips

unread,
Dec 8, 2016, 1:47:49 PM12/8/16
to consu...@googlegroups.com
Oh that's an oversight and it should definitely be mentioned there.
Created https://github.com/hashicorp/consul/issues/2583 to track that.
Thanks!

-- James
>> > email to consul-tool...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/consul-tool/c0d1ab2c-e87c-4faf-82aa-631667a5452b%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> This mailing list is governed under the HashiCorp Community Guidelines -
>> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
>> of those guidelines may result in your removal from this mailing list.
>>
>> GitHub Issues: https://github.com/hashicorp/consul/issues
>> IRC: #consul on Freenode
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Consul" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to consul-tool...@googlegroups.com.
> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/consul-tool/CAGxQj1GJdR4iNHenKOyA515XRr480K7Zqsf1ky1ZEZzWhSr1Jw%40mail.gmail.com.

Randy Fay

unread,
Dec 10, 2016, 10:33:10 PM12/10/16
to consu...@googlegroups.com
On a related note, I see snapshot creation (for raft) regularly, "snapshot: Creating new snapshot at /var/lib/consul/raft/snapshots/2-135633-1481417100545.tmp" - Is that snapshot related to the one that you can create manually? It looks like they're actually the same files you'd find in a tarball from a captured snapshot. 

-Randy


>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/consul-tool/c0d1ab2c-e87c-4faf-82aa-631667a5452b%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> This mailing list is governed under the HashiCorp Community Guidelines -
>> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
>> of those guidelines may result in your removal from this mailing list.
>>
>> GitHub Issues: https://github.com/hashicorp/consul/issues
>> IRC: #consul on Freenode
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Consul" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/consul-tool/CAGoWc06oYt5j4K-x0p-q%2BwW57yLkqKB5wnmzbXJ1yYC%2B3ZRccA%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Randy Fay
> ra...@randyfay.com
> +1 970.462.7450
>
> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> To view this discussion on the web visit
>
> For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CAGoWc05EeV2yVbu39B0KBzLjCSmUbU1WOeLRNWisG8ZL74X0%3Dw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

James Phillips

unread,
Dec 11, 2016, 2:11:25 AM12/11/16
to consu...@googlegroups.com
Yep - it uses the same mechanism under the hood, and you'll even see
that log line when you make a manual snapshot. The snapshot API takes
that snapshot file and packages with some metadata and SHA256 sums
inside of a tarball and gzips it.

-- James
>> >> > email to consul-tool...@googlegroups.com.
>> >> > To view this discussion on the web visit
>> >> >
>> >> >
>> >> > https://groups.google.com/d/msgid/consul-tool/c0d1ab2c-e87c-4faf-82aa-631667a5452b%40googlegroups.com.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> This mailing list is governed under the HashiCorp Community Guidelines
>> >> -
>> >> https://www.hashicorp.com/community-guidelines.html. Behavior in
>> >> violation
>> >> of those guidelines may result in your removal from this mailing list.
>> >>
>> >> GitHub Issues: https://github.com/hashicorp/consul/issues
>> >> IRC: #consul on Freenode
>> >> ---
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Consul" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to consul-tool...@googlegroups.com.
>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/consul-tool/CAGoWc06oYt5j4K-x0p-q%2BwW57yLkqKB5wnmzbXJ1yYC%2B3ZRccA%40mail.gmail.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> >
>> >
>> > --
>> > Randy Fay
>> > ra...@randyfay.com
>> > +1 970.462.7450
>> >
>> > --
>> > This mailing list is governed under the HashiCorp Community Guidelines -
>> > https://www.hashicorp.com/community-guidelines.html. Behavior in
>> > violation
>> > of those guidelines may result in your removal from this mailing list.
>> >
>> > GitHub Issues: https://github.com/hashicorp/consul/issues
>> > IRC: #consul on Freenode
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Consul" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to consul-tool...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/consul-tool/CAGxQj1GJdR4iNHenKOyA515XRr480K7Zqsf1ky1ZEZzWhSr1Jw%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> This mailing list is governed under the HashiCorp Community Guidelines -
>> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
>> of those guidelines may result in your removal from this mailing list.
>>
>> GitHub Issues: https://github.com/hashicorp/consul/issues
>> IRC: #consul on Freenode
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Consul" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to consul-tool...@googlegroups.com.
> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/consul-tool/CAGxQj1E7tWzM%3DFOL2Gc3QeAz%3DQTtX8QHc2w28Q%2Be%2B3n%3DNmmzkQ%40mail.gmail.com.

Randy Fay

unread,
Dec 11, 2016, 9:55:32 AM12/11/16
to consu...@googlegroups.com
Thanks - The "Outage Recovery" page probably should get a full revamp then, because if you have a snapshot on the filesystem (even if you didn't manually create it) you should be able to get everything back from the time of the snapshot. IMO it would be better for the auto-snapshotting to use the same packaging, so it could be used in a restore as is. It does look like the auto-snapshot directory contains state.bin and meta.json, but not the checksum file SHA256SUMS, but that can be created manually. 

cd /var/lib/consul/raft/snapshots/2-146226-1481465408058
[in snapshot directory]
# sha256sum * >SHA256SUMS
# tar -czf /tmp/recreated.snap *
# consul snapshot restore -token=... /tmp/recreated.snap
Restored snapshot

It would be nice to just package these auto-snapshots in a way they could be used immediately as-is in a desperate restore situation.

Thanks,
-Randy



>> >> > To view this discussion on the web visit
>> >> >
>> >> >
>> >> > https://groups.google.com/d/msgid/consul-tool/c0d1ab2c-e87c-4faf-82aa-631667a5452b%40googlegroups.com.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> This mailing list is governed under the HashiCorp Community Guidelines
>> >> -
>> >> https://www.hashicorp.com/community-guidelines.html. Behavior in
>> >> violation
>> >> of those guidelines may result in your removal from this mailing list.
>> >>
>> >> GitHub Issues: https://github.com/hashicorp/consul/issues
>> >> IRC: #consul on Freenode
>> >> ---
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Consul" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an

>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/consul-tool/CAGoWc06oYt5j4K-x0p-q%2BwW57yLkqKB5wnmzbXJ1yYC%2B3ZRccA%40mail.gmail.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> >
>> >
>> > --
>> > Randy Fay
>> > ra...@randyfay.com
>> > +1 970.462.7450
>> >
>> > --
>> > This mailing list is governed under the HashiCorp Community Guidelines -
>> > https://www.hashicorp.com/community-guidelines.html. Behavior in
>> > violation
>> > of those guidelines may result in your removal from this mailing list.
>> >
>> > GitHub Issues: https://github.com/hashicorp/consul/issues
>> > IRC: #consul on Freenode
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Consul" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/consul-tool/CAGxQj1GJdR4iNHenKOyA515XRr480K7Zqsf1ky1ZEZzWhSr1Jw%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> This mailing list is governed under the HashiCorp Community Guidelines -
>> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
>> of those guidelines may result in your removal from this mailing list.
>>
>> GitHub Issues: https://github.com/hashicorp/consul/issues
>> IRC: #consul on Freenode
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Consul" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/consul-tool/CAGoWc05EeV2yVbu39B0KBzLjCSmUbU1WOeLRNWisG8ZL74X0%3Dw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Randy Fay
> ra...@randyfay.com
> +1 970.462.7450
>
> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> To view this discussion on the web visit
>
> For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CAGoWc05vNOvDddVKge6WX_rsnkJp9w3Fmr5zOHsCECh-cnaXtg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

James Phillips

unread,
Dec 19, 2016, 10:57:31 PM12/19/16
to consu...@googlegroups.com
> Thanks - The "Outage Recovery" page probably should get a full revamp then, because if you have a snapshot on the filesystem (even if you didn't manually create it) you should be able to get everything back from the time of the snapshot.

If you still have the data directory with the Raft info, you should be
able to do a peers.json recovery even with just that server to get it
to start up with all its data intact. There's a line in the outage
guide about that:

> In extreme cases, it should be possible to recover with just a single remaining server by starting that single server with itself as the only peer in the raft/peers.json recovery file.

You shouldn't ever need to make a snapshot on your own if you've got
an intact data-dir.

Your example steps are valid, though, so I'll think about whether we
should include something like that for dire emergencies (maybe if you
are trying to make the best out of a corrupted data dir and using one
of the previous snapshots).
>> >> >> > email to consul-tool...@googlegroups.com.
>> >> >> > To view this discussion on the web visit
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > https://groups.google.com/d/msgid/consul-tool/c0d1ab2c-e87c-4faf-82aa-631667a5452b%40googlegroups.com.
>> >> >> > For more options, visit https://groups.google.com/d/optout.
>> >> >>
>> >> >> --
>> >> >> This mailing list is governed under the HashiCorp Community
>> >> >> Guidelines
>> >> >> -
>> >> >> https://www.hashicorp.com/community-guidelines.html. Behavior in
>> >> >> violation
>> >> >> of those guidelines may result in your removal from this mailing
>> >> >> list.
>> >> >>
>> >> >> GitHub Issues: https://github.com/hashicorp/consul/issues
>> >> >> IRC: #consul on Freenode
>> >> >> ---
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "Consul" group.
>> >> >> To unsubscribe from this group and stop receiving emails from it,
>> >> >> send
>> >> >> an
>> >> >> email to consul-tool...@googlegroups.com.
>> >> >> To view this discussion on the web visit
>> >> >>
>> >> >>
>> >> >> https://groups.google.com/d/msgid/consul-tool/CAGoWc06oYt5j4K-x0p-q%2BwW57yLkqKB5wnmzbXJ1yYC%2B3ZRccA%40mail.gmail.com.
>> >> >> For more options, visit https://groups.google.com/d/optout.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Randy Fay
>> >> > ra...@randyfay.com
>> >> > +1 970.462.7450
>> >> >
>> >> > --
>> >> > This mailing list is governed under the HashiCorp Community
>> >> > Guidelines -
>> >> > https://www.hashicorp.com/community-guidelines.html. Behavior in
>> >> > violation
>> >> > of those guidelines may result in your removal from this mailing
>> >> > list.
>> >> >
>> >> > GitHub Issues: https://github.com/hashicorp/consul/issues
>> >> > IRC: #consul on Freenode
>> >> > ---
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "Consul" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > email to consul-tool...@googlegroups.com.
>> >> > To view this discussion on the web visit
>> >> >
>> >> >
>> >> > https://groups.google.com/d/msgid/consul-tool/CAGxQj1GJdR4iNHenKOyA515XRr480K7Zqsf1ky1ZEZzWhSr1Jw%40mail.gmail.com.
>> >> >
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> This mailing list is governed under the HashiCorp Community Guidelines
>> >> -
>> >> https://www.hashicorp.com/community-guidelines.html. Behavior in
>> >> violation
>> >> of those guidelines may result in your removal from this mailing list.
>> >>
>> >> GitHub Issues: https://github.com/hashicorp/consul/issues
>> >> IRC: #consul on Freenode
>> >> ---
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Consul" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to consul-tool...@googlegroups.com.
>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/consul-tool/CAGoWc05EeV2yVbu39B0KBzLjCSmUbU1WOeLRNWisG8ZL74X0%3Dw%40mail.gmail.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> >
>> >
>> > --
>> > Randy Fay
>> > ra...@randyfay.com
>> > +1 970.462.7450
>> >
>> > --
>> > This mailing list is governed under the HashiCorp Community Guidelines -
>> > https://www.hashicorp.com/community-guidelines.html. Behavior in
>> > violation
>> > of those guidelines may result in your removal from this mailing list.
>> >
>> > GitHub Issues: https://github.com/hashicorp/consul/issues
>> > IRC: #consul on Freenode
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Consul" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to consul-tool...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/consul-tool/CAGxQj1E7tWzM%3DFOL2Gc3QeAz%3DQTtX8QHc2w28Q%2Be%2B3n%3DNmmzkQ%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> This mailing list is governed under the HashiCorp Community Guidelines -
>> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
>> of those guidelines may result in your removal from this mailing list.
>>
>> GitHub Issues: https://github.com/hashicorp/consul/issues
>> IRC: #consul on Freenode
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Consul" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to consul-tool...@googlegroups.com.
> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/consul-tool/CAGxQj1F-82id6q8AbPbrZ0k7KehqMXHWJLDRe%2BF5k4%3Dy_jYUCg%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages