chunks_head space issue

[Senthil]

Hi All,

I am having issues with chunks_head directory. I have the tsdb retention size to 175 GB, but the chunks_head size is increasing rapidly. now it's  689 GB.

689G    chunks_head
4.0K    queries.active
18G     wal

ExecStart=/apps/prometheus/prometheus_server/prometheus \
  --config.file=/apps/prometheus/prometheus_server/prometheus.yml \
  --storage.tsdb.path=/apps/prometheus/prometheus_data \
  --storage.tsdb.retention.size=500GB \
  --storage.tsdb.retention.time=180d

Is there any setting I can make to control the chunks_head  space?

Prometheus version is

prometheus, version 2.31.0 (branch: HEAD, revision: d4c83da6d252d4edfdbd639fb817ebdb8e9ab2e4)
build user: root395d507f537a
build date: 20211102-10:28:54
go version: go1.17.2
platform: linux/amd64

Every time I have to remove all directories under the chunks_head and restart the Prometheus service on that server. 

[Brian Candler]

I am having issues with chunks_head directory. I have the tsdb retention size to 175 GB

... 

  --storage.tsdb.retention.size=500GB

That doesn't appear to be consistent with what you said above.

However, in any case, the "chunks_head" directory should only contain data created in the last 2-4 hours, before it gets moved to its final location.

Is it possible that you're generating 689GB of data in 2-4 hours?  That's a *lot* of metrics, and it seems unlikely - but if so, this would be expected behaviour.

If not, then there's some problem moving the head chunks into their final location.  Your storage directory should look something like this (from v2.30.3):

root@prometheus:~# ls /var/lib/prometheus/data/
01FRDMW4G1T19JM989NE1VKZ32  01FSGD7TFJEWA6FJ08PJ2TZZPG  01FTK5KH12W0RDWZA85NSTP4TX  chunks_head
01FRKE8RPDW2ZYSJFXEDD7QK3S  01FSP6MG73NQQ996C7EWH49MQF  01FTRZ05RH3C8RKJ4PQ162QJT8  lock
01FRS7NBJST1MW7R3XDBE9592M  01FSW012JNK10VV5WWZ4C2H3V9  01FTSKK60DX4D8C908BYHZTR8F  queries.active
01FRZ120RQQZJW24T3BVP9NFTT  01FT1SDN5R9JYEQFQ90JCS2Y2Z  01FTT869E3DW1V0QYSPZ921CKQ  wal
01FS4TEKSMZ3NJ2M6NNEAH9K63  01FT7JTABCH3D1JDQYZAPJ9QXK  01FTT86AK2N6VA00RSCYB9AC6J
01FSAKV7VSS6VMPDCJAWNZGT6V  01FTDC6WGMY25WCQ3JTKZECGPX  01FTTF20P47NVE06W79PKKFXFJ

The majority of storage will be in those other directories, which are created automatically, and are what the retention setting will control.  If this isn't happening, then you have a problem.

I suggest you check the log output from prometheus for errors (e.g. "journalctl -eu prometheus" if you're running it under systemd).  It could be that it's failing to create these directories, and data is sticking around in the head.  Are permissions on /apps/prometheus/prometheus_data correct?  Are you storing data on a block filesystem (e.g. local disk or a filesystem in Amazon EBS)?  Prometheus does not support writing data to any sort of NAS filesystem, e.g. NFS / Samba / EFS.

There were a couple of minor fixes in v2.31.1 and v2.31.2, but I don't think they are relevant here.

 

[Senthil]

Thanks for the response Brain, it was 175GB and with the last restart, I increased it to 500 GB, sorry I did a typo there. 

It's not generating that much data in 2 ~ 4 hrs, but it is doing over a week. The data is stored on the local filesystem ext4. not on nas. 

# ls
01FTRRKBEQV1QX0HAM2MXG1W3Q  01FTTWSEGZ4MQN3NXAR12R8GSD  01FTTX6H5GM758FVPHK1DTSH9S  chunks_head  lock  queries.active  wal
# ls chunks_head/
000175  000176

I don't see any errors in the logs, but still checking on the issue. 

Thanks

[Brian Candler]

On Tuesday, 1 February 2022 at 15:52:08 UTC Senthil wrote:

# ls
01FTRRKBEQV1QX0HAM2MXG1W3Q  01FTTWSEGZ4MQN3NXAR12R8GSD  01FTTX6H5GM758FVPHK1DTSH9S  chunks_head  lock  queries.active  wal
# ls chunks_head/
000175  000176

How long after a restart is that?  I suggest you compare how it is a day later.

Also you can try periodically:

du -sck chunks_head/*

to see if it's any one particular subdirectory which is getting very big (in which case, drill down to that one) or lots of subdirectories that aren't getting cleaned up.

Is all of /apps/prometheus/prometheus_data on a single filesystem?  You don't have any separate mountpoints underneath it for chunks_head or wal, or symlinks pointing to another location?

mount | grep /apps/prometheus

find /apps/prometheus/prometheus_data/ -type l

[Senthil]

I started on Jan 31, so it's a day.

# du -sck chunks_head/*
54140   chunks_head/024326
4       chunks_head/024327
54144   total

mount | grep /apps/prometheus

/dev/mapper/vg_apps-lv_apps on /apps type ext4 (rw,nodev,relatime,data=ordered)

# find /apps/prometheus/prometheus_data/ -type l - for this I didn't find any. 

[Brian Candler]

On Tuesday, 1 February 2022 at 21:52:30 UTC Senthil wrote:

I started on Jan 31, so it's a day.

# du -sck chunks_head/*
54140   chunks_head/024326
4       chunks_head/024327
54144   total

That's perfectly reasonable: it's only 54MB (which is a long way from 689GB!)

Here's what I see on a moderately busy system:

root@ldex-prometheus:~# du -sck /var/lib/prometheus/data/chunks_head/*
81004        /var/lib/prometheus/data/chunks_head/006831
77824        /var/lib/prometheus/data/chunks_head/006832
158828        total

That's comparable to yours.

Therefore, I think you need to keep an eye on this periodically.  If only you had a monitoring system which could do this for you :-)

If it does start to rise, that's when you'll need to check prometheus log output and find out what's happening.  But this is very strange, and it does seem to be something specific to your system.

[Senthil]

The issue started again. 

629G    chunks_head
0       lock
4.0K    queries.active
9.3G    wal

There is numerous restart of Prometheus

Feb 17 09:02:02 kernel: Out of memory: Kill process 36580 (prometheus) score 844 or sacrifice child
Feb 17 09:08:36 kernel: Out of memory: Kill process 39001 (prometheus) score 846 or sacrifice child
Feb 17 09:16:02 kernel: Out of memory: Kill process 41074 (prometheus) score 845 or sacrifice child
Feb 17 09:22:17 kernel: Out of memory: Kill process 44665 (prometheus) score 844 or sacrifice child
Feb 17 09:29:25 kernel: Out of memory: Kill process 47234 (prometheus) score 844 or sacrifice child
Feb 17 09:36:06 kernel: Out of memory: Kill process 48970 (prometheus) score 846 or sacrifice child
Feb 17 09:43:21 kernel: Out of memory: Kill process 50661 (prometheus) score 844 or sacrifice child

but there is plenty of mem available in the servers.

              total        used        free      shared  buff/cache   available
Mem:             47           5          31           0          10          40
Swap:             5           1           3
Total:           52           7          35

[Brian Candler]

Now would be a good time to do:

ls -l /var/lib/prometheus/data/chunks_head/

du -sck /var/lib/prometheus/data/chunks_head/*

My suspicion is your out-of-memory condition is messing up the writing of chunks.  Are you using cgroups/containers?

Also, is prometheus continually crashing and being restarted by systemd? Try looking in "journalctl -eu prometheus".  That might explain why you see lots of free memory most of the time (when prometheus is stopped).

[Senthil]

Yes, it continuously crashes for OOM, 10 to 15 min once. 
Unfortunately, this time also someone deleted those files, to recover the prometheus, still it crashes and auto restarts. 

Feb 18 05:49:18 kernel: Out of memory: Kill process 61845 (prometheus) score 844 or sacrifice child
Feb 18 05:52:26 kernel: Out of memory: Kill process 63185 (prometheus) score 844 or sacrifice child
Feb 18 05:55:47 kernel: Out of memory: Kill process 64500 (prometheus) score 844 or sacrifice child
Feb 18 05:58:51 kernel: Out of memory: Kill process 875 (prometheus) score 844 or sacrifice child
Feb 18 05:58:51 kernel: Out of memory: Kill process 1754 (prometheus) score 844 or sacrifice child
Feb 18 06:02:05 kernel: Out of memory: Kill process 2328 (prometheus) score 845 or sacrifice child
Feb 18 06:05:39 kernel: Out of memory: Kill process 3155 (prometheus) score 844 or sacrifice child
Feb 18 06:09:06 kernel: Out of memory: Kill process 5273 (prometheus) score 845 or sacrifice child
Feb 18 06:12:24 kernel: Out of memory: Kill process 6549 (prometheus) score 844 or sacrifice child
Feb 18 06:15:29 kernel: Out of memory: Kill process 6756 (prometheus) score 845 or sacrifice child
Feb 18 06:18:28 kernel: Out of memory: Kill process 8474 (prometheus) score 844 or sacrifice child
Feb 18 06:21:36 kernel: Out of memory: Kill process 8649 (prometheus) score 845 or sacrifice child
Feb 18 06:24:41 kernel: Out of memory: Kill process 9708 (prometheus) score 844 or sacrifice child
Feb 18 06:27:52 kernel: Out of memory: Kill process 11003 (prometheus) score 844 or sacrifice child
Feb 18 06:30:50 kernel: Out of memory: Kill process 11189 (prometheus) score 844 or sacrifice child
Feb 18 06:33:47 kernel: Out of memory: Kill process 12210 (prometheus) score 844 or sacrifice child