How to get the disk usage of a PVC in Kubernetes to alert on it?

[Khusro Jaleel]

Hello,

I'm running Prometheus in a K8s cluster (1.7.6). I installed it using Helm and it creates a PVC/PV and I'm trying to alert if the disk space on that PV is below a certain threshold using AlertManager, but I'm not sure where I can query this from? 

"node_filesystem_avail" seems the right fit for this but it doesn't have info about the PV which is mounted as "sdb" on the host node.

I have CAdvisor and Node Exporter running in the same cluster but they don't seem to have any metrics for "externally" mounted disks, like "sdb", etc. Maybe there's a parameter I could add to Node Exporter or enable a certain "collector" to get this info? 

Thanks.

[Khusro Jaleel]

Anyone got any ideas about this? I wouldn't think it would be recommended to just run Prometheus in K8s without any disk space monitoring? How do people approach this? 

[kondar...@gmail.com]

Hello Khusro ,

Good Day !!

Did you fix the issue , since I  have also same request ,

can you please help me  if you have already fix the issue ,

Regards

Konda Reddy  

[Khusro Jaleel]

Hi Konda,

I ended up running Prometheus outside of Kubernetes instead, using a GCP instance and it was a lot easier to run node_exporter on that and get the correct metrics.

However, I did come across this tool [1] which I tried and it DID work from inside the cluster and I was successfully able to get PV metrics, however I found that I had to sometimes restart the pods that this tool creates, otherwise I wouldn't be able to get metrics successfully which was a downside.

Give it a try if you'd like and see if it works for you? 

[1] - https://github.com/joar/k8s-pv-disk-usage-exporter

[Martin Chodúr]

Hi, actually metrics about PV size has the kubelet by default check that out.