hot backup with prometheus 2.0?

[californi...@gmail.com]

I can find the followings in the article "Sneak Peak of Prometheus 2.0" at https://prometheus.io/blog/2017/04/10/promehteus-20-sneak-peak/

"It also paves the way to add features such as hot backups."

I would like to know what this means. I need to backup metrics data that prometheus scraped without stopping prometheus itself. I understand I need to stop prometheus to keep consistency of data store.

[Brian Brazil]

On 15 November 2017 at 06:13, <californi...@gmail.com> wrote:

I can find the followings in the article "Sneak Peak of Prometheus 2.0" at https://prometheus.io/blog/2017/04/10/promehteus-20-sneak-peak/

"It also paves the way to add features such as hot backups."

Try doing:

echo -n | POST http://localhost:9090/api/v2/admin/tsdb/snapshot 

with the admin API flag enabled.

There's no longer a need to stop Prometheus.

--

Brian Brazil

www.robustperception.io

[californi...@gmail.com]

Thank you. I have some questions.

1. Each snapshot seems to include all files in /data directory. Out prometheus will have over 500GB data to snapshot. Question is how the size of data affects snapshot. Duration of hot backup will be longer and that might lead to some performance issue such as memory usage. If it is not unwise to take snapshot of that size, I should think of using remote read/write that is in experimental state in production.

2. How can I resotre data from snapshot? just put files into data directory and start prometheus?

[Brian Brazil]

On 16 November 2017 at 02:47, <californi...@gmail.com> wrote:

Thank you. I have some questions.

1. Each snapshot seems to include all files in /data directory. Out prometheus will have over 500GB data to snapshot. Question is how the size of data affects snapshot. Duration of hot backup will be longer and that might lead to some performance issue such as memory usage. If it is not unwise to take snapshot of that size, I should think of using remote read/write that is in experimental state in production.

It's using hardlinks, so it's much cheaper than that.

 

2. How can I resotre data from snapshot? just put files into data directory and start prometheus?

 

Yes.

Brian 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/72a6e716-addc-43ad-beef-093afb6498fa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[californi...@gmail.com]

I misunderstood what hard link is. let me deleted former questions and post newer ones as follows:

How can I know job to create hard link on shapshot directory? I would like to make sure that hard links are fully created before starting a file copy job to move snapshot to backup disk.

[Ben Kochie]

On Fri, Nov 17, 2017 at 7:19 AM, <californi...@gmail.com> wrote:

I misunderstood what hard link is. let me deleted former questions and post newer ones as follows:

How can I know job to create hard link on shapshot directory? I would like to make sure that hard links are fully created before starting a file copy job to move snapshot to backup disk.

The snapshot API call is blocking, when the HTTP request completes, the snapshot is ready.

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c5c93eab-dc15-49cf-8813-06c86f601280%40googlegroups.com.

[Goutham Veeramachaneni]

Hi,

This video will hopefully make it clear: https://www.youtube.com/watch?v=15uc8oTMgPY When you call the snapshot API, it will create the snapshot and returns the foldername. The API call won't take more than a few seconds to complete and once the API call returns, the snapshotting is complete and you can go ahead and move the snapshot.  

Thanks,

Goutham

[californi...@gmail.com]

Ben and Goutham,

Thank you for your comment. I could understand how it works.