Tuning Prometheus memory usage in Kubernetes

[Khusro Jaleel]

Hi there,

I have a second question about running Prometheus (1.6.2) in our Kubernetes cluster (1.7.8). I am using GKE for this.

I have used Helm to install it and given the pods 2Gi of memory (both request and limits are the same), and I've told Prometheus to use a heap size of around 1.4Gi only as the maximum. However I do sometimes see Kubernetes killing the pod because it suddenly spiked up to 2.6Gi of memory for example, I think that might be because someone did an expensive query? Is that possible? And if so, is there a way to tune it so it doesn't try to burst to such a high usage? 

On a separate note, I'm wondering if it's recommended to run Prometheus inside a K8 cluster at all? Is it generally more stable to run it externally, as long as it can access all the pods directly? 

Thanks for any insights.

[Ben Kochie]

Yes, a large query can load a lot of data into memory for a short period of time.  This is unavoidable in Prometheus 1.x.

Running Prometheus in K8s is just fine.  I would recommend giving your Prometheus server a little more memory slack to avoid the crashes.

I would also remove the limit, or set it much higher and rely on the normal request size for scheduling the Prometheus pod(s).

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6dd76cb9-1f91-43c8-8ca2-51a6503deded%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Khusro Jaleel]

Thanks Ben, you mentioned "Prometheus 1.x" - so is it a good idea to switch to 2.x now or soon when it's released? How does that handle memory differently? 

On Monday, 6 November 2017 16:40:31 UTC, Ben Kochie wrote:

Yes, a large query can load a lot of data into memory for a short period of time.  This is unavoidable in Prometheus 1.x.

Running Prometheus in K8s is just fine.  I would recommend giving your Prometheus server a little more memory slack to avoid the crashes.

I would also remove the limit, or set it much higher and rely on the normal request size for scheduling the Prometheus pod(s).

On Mon, Nov 6, 2017 at 3:38 PM, Khusro Jaleel <kerne...@gmail.com> wrote:

Hi there,

I have a second question about running Prometheus (1.6.2) in our Kubernetes cluster (1.7.8). I am using GKE for this.

I have used Helm to install it and given the pods 2Gi of memory (both request and limits are the same), and I've told Prometheus to use a heap size of around 1.4Gi only as the maximum. However I do sometimes see Kubernetes killing the pod because it suddenly spiked up to 2.6Gi of memory for example, I think that might be because someone did an expensive query? Is that possible? And if so, is there a way to tune it so it doesn't try to burst to such a high usage? 

On a separate note, I'm wondering if it's recommended to run Prometheus inside a K8 cluster at all? Is it generally more stable to run it externally, as long as it can access all the pods directly? 

Thanks for any insights.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

There was just another thread on prometheus-users about 2.0 today.

It's currently a release candidate.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ec738855-aefe-4941-8783-146854f21a24%40googlegroups.com.

[khusro...@holidayextras.com]

Hi, I set my memory limits as discussed in the post below but our Prometheus server (1.6.2) crashed a couple days ago and was never able to perform a full crash recovery. What was happening was the following:

Our Kubernetes node has 3.5GB of memory, some of which is reserved for the OS. The Allocatable amount is only around 2500Mi, and when I try to create a Prometheus pod, the max I can give it is 2Gi, no more than that, so I set the "Requests" to a max of 2Gi and I did NOT set any Limits so that it could burst to as much as it would need. I set the target heap size to around 1.4Gi.

What happened then was that the K8s node "killed" my Prometheus pod because the Node itself was low on memory resource. Why this was the case, I'm still not sure. I'm suspecting that something else OS-level on the Node (like fluentd) was taking up too much memory and that's why the low memory condition on the node happened. Prometheus itself does have an RSS of around 1.9Gi when running.

When the Prometheus pod crashed, it tried to do a crash recovery but K8s kept on killing it repeatedly because it tried to use as much memory on the Node as it could, so K8s would kill it. I then tried to add Limits to it so it wouldn't try to "burst" and use more than it needed, but it still continued to crash because then the app itself would run of of memory and crash so the recovery essentially never completed.

Is there a way to recover from this situation? Is there more info on what the recovery process actually does, or is there a way to "skip" it so it's not needed at all? As far as I could tell it was unable to start up for about 8 hours so what exactly was it trying to do when starting up? What was it trying to recover? 

It seems I might have hit this LevelDB issue and there doesn't seem to be a fix for it?

https://github.com/prometheus/prometheus/issues/3038

https://github.com/prometheus/prometheus/issues/2139

In the end I made a snapshot of the GCP mounted disk, deleted all the data on it and then I was able to bring Prometheus up properly.

This might happen again at any time now so I'm keen to avoid it, I'd appreciate any ideas or tips. I have attached a few screenshots of what memory usage looks like at the moment, along with the number of samples we are currently ingesting. Number of chunks seems to be consistently <800K.

On Monday, 6 November 2017 16:40:31 UTC, Ben Kochie wrote:

Yes, a large query can load a lot of data into memory for a short period of time.  This is unavoidable in Prometheus 1.x.

Running Prometheus in K8s is just fine.  I would recommend giving your Prometheus server a little more memory slack to avoid the crashes.

I would also remove the limit, or set it much higher and rely on the normal request size for scheduling the Prometheus pod(s).

On Mon, Nov 6, 2017 at 3:38 PM, Khusro Jaleel <kerne...@gmail.com> wrote:

Hi there,

I have a second question about running Prometheus (1.6.2) in our Kubernetes cluster (1.7.8). I am using GKE for this.

I have used Helm to install it and given the pods 2Gi of memory (both request and limits are the same), and I've told Prometheus to use a heap size of around 1.4Gi only as the maximum. However I do sometimes see Kubernetes killing the pod because it suddenly spiked up to 2.6Gi of memory for example, I think that might be because someone did an expensive query? Is that possible? And if so, is there a way to tune it so it doesn't try to burst to such a high usage? 

On a separate note, I'm wondering if it's recommended to run Prometheus inside a K8 cluster at all? Is it generally more stable to run it externally, as long as it can access all the pods directly? 

Thanks for any insights.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6dd76cb9-1f91-43c8-8ca2-51a6503deded%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

It sounds like you need a larger node to operate your Prometheus server.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/51131f98-edd4-4c58-9ff6-1a9a76c42b0b%40googlegroups.com.