crash recovery

[Meher Garda]

Hi

Prometheus got OOM'd. 

I restarted Prometheus on a diff server with more resources.

I see a lot of messages in prometheus log like so

fingerprint 0afa517d833c582b: recovered 54 chunks from series file, recovered 1 chunks from checkpoint.","source":"crashrecovery.go:363","time":"2016-11-18T00:12:45Z

for the last 2 hours.

Prometheus port 9090 is not up.

How long does crash recovery take?

Is there a way for prometheus to begin scraping and make new monitoring data available while crash recovery is ongoing?

I ask because since the last 2 hours while crash recovery is happening, prometheus is not listening on port 9090 and we basically have no monitoring available.

[Björn Rabenstein]

On 18 November 2016 at 01:19, Meher Garda <meher...@gmail.com> wrote:
>
> How long does crash recovery take?

Usually only a few minutes, but that depends on your storage device.
As you have started Prometheus on a different server but on the same
data, it sounds you are using some kind of network device. That might
explain the slowness in your case.

> Is there a way for prometheus to begin scraping and make new monitoring
> data available while crash recovery is ongoing?

No.

> I ask because since the last 2 hours while crash recovery is happening, prometheus
> is not listening on port 9090 and we basically have no monitoring available.

The immediate remedy is to start another server in parallel with the
same config but a fresh data directory. You won't have history, but
you get fresh metrics for the time being.

That's the good thing with pull-based monitoring. You can do those
things in an instant.

Also, in general, I would run every important Prometheus server twice
with the same config and separate data directory for redundancy.





--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Meher Garda]

I am on AWS and the data directory is on EBS.

I attached the same volume to a diff server with more resources.

If you consider EBS as a network attahced device then fine but if you are referring to an NFS or something similar thats not true

If I run two Prometheus servers simultaneously then would it not scrape every target twice. If I recall somewhere I read that scraping a target twice is not recommended or somethin along those lines, 

[Björn Rabenstein]

On 18 November 2016 at 18:11, Meher Garda <meher...@gmail.com> wrote:
> If you consider EBS as a network attahced device then fine but if you are
> referring to an NFS or something similar thats not true

Crash recovery performs loads of I/O ops, and EBS is limited on that.
So that would be the most likely explanation why the crash recovery
takes so long in your case.

> If I run two Prometheus servers simultaneously then would it not scrape
> every target twice. If I recall somewhere I read that scraping a target
> twice is not recommended or somethin along those lines,

Running two or more Prometheus servers scraping the same target is
wholeheartedly recommended. The whole Prometheus ecosystem is designed
with that in mind.
Multiple scrapes are fine and encouraged, that's the core of the
Prometheus data model.

[Meher Garda]

Ok. We can use another disk type on AWS.

What I fail to understand is that when the OOm happened there would be at most "9409708" bytes in memory 

which is defined by storage.local.memory-chunks. Thats apprx. 9.5 mb in memory . The initial system had 64 gb of memory. 20 gb was assigned to prometheus.

Now it should not take more than a few mins to recover 9.5 mb of data, unless the crash recovery goes though the entire data directory ( in our case 100 gb in data folder). Does the crash recovery go through all the data stored in the persistance layer?

[Björn Rabenstein]

On 18 November 2016 at 20:38, Meher Garda <meher...@gmail.com> wrote:
> What I fail to understand is that when the OOm happened there would be at
> most "9409708" bytes in memory
> which is defined by storage.local.memory-chunks. Thats apprx. 9.5 mb in
> memory . The initial system had 64 gb of memory. 20 gb was assigned to
> prometheus.

Where is that number "9409708" coming from? If that's the setting of
storage.local.memory-chunks, then Prometheus tries to have at most
that number of chunks in RAM. Each chunk is 1kiB, but there is
overhead. And your server is doing many more things. A Prometheus
server with that number of configured chunks will usually take at
least ~30GiB of RAM, potentially much more depending on the number of
memory times series, service discovery, used queries... Please read
https://prometheus.io/docs/operating/storage/

> Now it should not take more than a few mins to recover 9.5 mb of data,
> unless the crash recovery goes though the entire data directory ( in our
> case 100 gb in data folder). Does the crash recovery go through all the data
> stored in the persistance layer?

It has to check the consistency of teh files on disk. Depending on the
nature of your crash, it will have to look through more or fewer time
series files.