Is there any way to give memory specifications by ourselves in prometheus?

[adi garg]

Hello experts, 

Is there any way to give memory specifications by ourselves in prometheus?

[Julien Pivotto]

On 19 Feb 08:08, adi garg wrote:
> Hello experts,
>
> Is there any way to give memory specifications by ourselves in prometheus?

Hello,

No, this is not possible. Prometheus will use the memory it needs.

Regards,

>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/0fb6ec18-c2a8-437f-9d2b-5cc2b9f5b330%40googlegroups.com.


--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[adi garg]

So what will happen if prometheus crosses the RAM limit, will it die? Or is it gonna affect the other processes running on the system?

On Wednesday, February 19, 2020 at 9:41:36 PM UTC+5:30, Julien Pivotto wrote:

On 19 Feb 08:08, adi garg wrote:
> Hello experts,
>
> Is there any way to give memory specifications by ourselves in prometheus?

Hello,

No, this is not possible. Prometheus will use the memory it needs.

Regards,

>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

[Julien Pivotto]

On 19 Feb 08:24, adi garg wrote:
> So what will happen if prometheus crosses the RAM limit, will it die? Or is
> it gonna affect the other processes running on the system?

It is the operating system that will chose. Prometheus will probably be
terminated by the operating system.

>
> On Wednesday, February 19, 2020 at 9:41:36 PM UTC+5:30, Julien Pivotto
> wrote:
> >
> > On 19 Feb 08:08, adi garg wrote:
> > > Hello experts,
> > >
> > > Is there any way to give memory specifications by ourselves in
> > prometheus?
> >
> > Hello,
> >
> > No, this is not possible. Prometheus will use the memory it needs.
> >
> > Regards,
> >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "Prometheus Developers" group.
> > > To unsubscribe from this group and stop receiving emails from it, send

> > an email to prometheus-devel...@googlegroups.com
> > <javascript:>.

> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-developers/0fb6ec18-c2a8-437f-9d2b-5cc2b9f5b330%40googlegroups.com.
> >
> >
> >
> > --
> > (o- Julien Pivotto
> > //\ Open-Source Consultant
> > V_/_ Inuits - https://www.inuits.eu
> >
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/df7bdc80-b95f-4f02-88d1-664d3cf3c651%40googlegroups.com.

[Julius Volz]

While an explicit memory limit is not configurable, there are a number of knobs in Prometheus that one can configure that limit resource usage along certain dimensions, for example https://www.robustperception.io/limiting-promql-resource-usage.

There's also a setting that prevents a maximum number of samples ingested per scrape.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200219163423.GA6819%40oxygen.

[adi garg]

Thanks Julius and Julien. Awesome answers. This is related to RAM, but what will happen if the secondary storage is not sufficient to take the metrics. What will happen in that case?

On Wednesday, February 19, 2020 at 10:32:43 PM UTC+5:30, Julius Volz wrote:

While an explicit memory limit is not configurable, there are a number of knobs in Prometheus that one can configure that limit resource usage along certain dimensions, for example https://www.robustperception.io/limiting-promql-resource-usage.

There's also a setting that prevents a maximum number of samples ingested per scrape.

On Wed, Feb 19, 2020 at 5:34 PM Julien Pivotto <roidel...@inuits.eu> wrote:

On 19 Feb 08:24, adi garg wrote:
> So what will happen if prometheus crosses the RAM limit, will it die? Or is
> it gonna affect the other processes running on the system?


It is the operating system that will chose. Prometheus will probably be
terminated by the operating system.

>
> On Wednesday, February 19, 2020 at 9:41:36 PM UTC+5:30, Julien Pivotto
> wrote:
> >
> > On 19 Feb 08:08, adi garg wrote:
> > > Hello experts,
> > >
> > > Is there any way to give memory specifications by ourselves in
> > prometheus?
> >
> > Hello,
> >
> > No, this is not possible. Prometheus will use the memory it needs.
> >
> > Regards,
> >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "Prometheus Developers" group.
> > > To unsubscribe from this group and stop receiving emails from it, send

> > an email to prometheus-developers+unsub...@googlegroups.com

> > <javascript:>.
> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-developers/0fb6ec18-c2a8-437f-9d2b-5cc2b9f5b330%40googlegroups.com.
> >
> >
> >
> > --
> >  (o-    Julien Pivotto
> >  //\    Open-Source Consultant
> >  V_/_   Inuits - https://www.inuits.eu
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/df7bdc80-b95f-4f02-88d1-664d3cf3c651%40googlegroups.com.


--
 (o-    Julien Pivotto
 //\    Open-Source Consultant
 V_/_   Inuits - https://www.inuits.eu

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

[Stuart Clark]

If you are meaning the disk storage, Prometheus would stop working and you might also encounter some corruption.

You can control disk usage by setting the retention period or by setting the maximum space to remain available.

I'd strongly suggest putting the storage on a separate mount and also using the node exporter with alerts to ensure you don't run out of space.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[adi garg]

Thanks Stuart. Is it possible to restart the prometheus without data loss from that point?

[Stuart Clark]

Probably not. Prometheus will write scraped data to the WAL and if that fails it won't be stored anywhere. Equally if a file is only partially written (either a full block or something in the WAL) you may end up losing all the data in that file (Prometheus will check for corruption on startup).