Raspberry Pi: Died after 2 months with "panic: mmap, size 134217728: cannot allocate memory"

[Kok-Yan Lo]

Hey Everyone, my Prometheus instance died with the dreaded error:

"panic: mmap, size 134217728: cannot allocate memory"

Just wondering if anyone managed to solve it?

It is exactly the same issue (and log pattern too) as reported here - I'm on 32-bit Raspbian 10, on a Pi 4 (4 GB model):

https://github.com/prometheus/prometheus/issues/8661

I tried removing all files in "wal" directory, which brought it back but then failed again with the similar error.
ts=2021-12-22T17:57:22.806Z caller=compact.go:459 level=info component=tsdb msg="compact blocks" count=3 mint=1639375206018 maxt=1639958400000 ulid=01FQHMMJYJQK9B3NQD5P7PGA6Q sources="[01FPZGK37J5MZ80SGREDQ6J87Q 01FQ533ZRMFYPNGG9MYZC36KHC 01FQAWGKH8XE52XXFE6NSJAWXX]" duration=5.925005803s
ts=2021-12-22T17:57:23.063Z caller=db.go:830 level=error component=tsdb msg="compaction failed" err="reloadBlocks blocks: corrupted block 01FQHMMJYJQK9B3NQD5P7PGA6Q: mmap files: mmap, size 206139815: cannot allocate memory"
ts=2021-12-22T17:59:24.390Z caller=stdlib.go:105 level=error component=web caller="http: TLS handshake error from 172.19.0.1:60974" msg=EOF
panic: mmap, size 134217728: cannot allocate memory

goroutine 428 [running]:
github.com/prometheus/prometheus/tsdb.(*memSeries).mmapCurrentHeadChunk(0x6b38a50, 0x6710510)
        /app/tsdb/head_append.go:565 +0x220
github.com/prometheus/prometheus/tsdb.(*memSeries).cutNewHeadChunk(0x6b38a50, 0x17de34cdd97, 0x6710510)
        /app/tsdb/head_append.go:536 +0x24
github.com/prometheus/prometheus/tsdb.(*memSeries).append(0x6b38a50, 0x17de34cdd97, 0x3ff0000000000000, 0x392, 0x6710510)
        /app/tsdb/head_append.go:503 +0x320
github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0x78fb620)
        /app/tsdb/head_append.go:439 +0x3f0
github.com/prometheus/prometheus/tsdb.dbAppender.Commit({{0x2bfce60, 0x78fb620}, 0x6710000})
        /app/tsdb/db.go:871 +0x24
github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit(0x83ac920)
        /app/storage/fanout.go:176 +0x28
github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport.func1(0x7505e18, 0x7505e20, 0x5fb4280)
        /app/scrape/scrape.go:1250 +0x38
github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport(0x5fb4280, 0xdf8475800, 0x2540be400, {0xc068f7ba5031d0cd, 0x57dc6cc6a0d, 0x3e8c588}, {0x10293dea, 0xed9555ea5, 0x3e8c588}, 0x0)
        /app/scrape/scrape.go:1317 +0xe58
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x5fb4280, 0x0)
        /app/scrape/scrape.go:1203 +0x324
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
        /app/scrape/scrape.go:584 +0x86c

I don't have any special configs to my container - just this docker-compose.yml:

version: '3.8'

services:

prometheus:

image: prom/prometheus:v2.31.1

container_name: prometheus

volumes:

# Config

- /srv/prometheus/etc/prometheus/:/etc/prometheus/:ro

# Data

- /srv/prometheus/prometheus/:/prometheus/

command:

# Default commands and paths from https://github.com/prometheus/prometheus/blob/main/Dockerfile

- "--config.file=/etc/prometheus/prometheus.yml"

- "--storage.tsdb.path=/prometheus"

- "--web.console.libraries=/usr/share/prometheus/console_libraries"

- "--web.console.templates=/usr/share/prometheus/consoles"

# Keep data for 3 months

- --storage.tsdb.retention.time=90d

# Listen on port 9090 inside container

- --web.listen-address=:9090

# Use TLS with Basic Auth

- --web.config.file=/etc/prometheus/web-config.yml

ports:

- 443:9090

restart: on-failure:3

Any idea / suggestions welcome :-)

Thanks!

Kok-Yan

[Kok-Yan Lo]

In case anyone came across this, I resolve it by moving to Raspberry Pi OS 11 64-bit.

All other configs remains the same.

Moral of the story - don't run Prometheus in 32-bit...