Re: [prometheus-users] failures dismissed by incompetent developer - maybe others have a solution

[Richard Hartmann]

Hi,

while I do understand that using software can be stressful, in particular when it's not acting as you want it to, and that it's human to have strong emotions sometimes, those emotions should not swap over onto GitHub or mailing lists. There are humans on the other side as well.

Please be aware that Prometheus is part of the CNCF and we do follow the CNCF CoC.

Still, best,

Richard

[Julius Volz]

Hi,

The experimental flag to allow limiting disk usage was mostly meant for larger disk usages than a couple of hundred MB, but there's an opportunity to clarify that in the flag description. Prometheus will need to store at least a couple of hundred MB due to the way it segments and preallocates data (current TSDB devs would be able to provide more detail here). Indeed this behavior changed with 2.x, as the entire storage engine got swapped out for a much better one (except for tiny use cases like this).

Please refrain from personal insults, as it's against our code of conduct (https://github.com/cncf/foundation/blob/master/code-of-conduct.md).

Regards,

Julius

On Mon, Jan 25, 2021 at 2:59 PM prommyonapi <j.a.st...@gmail.com> wrote:

https://github.com/prometheus/prometheus/issues/8406

pi@raspberrypi:~ $ cat /etc/fstab

proc            /proc           proc    defaults          0       0

PARTUUID=6c586e13-01  /boot vfat    defaults,noatime         0       2

PARTUUID=6c586e13-02  / ext4    defaults,noatime,commit=600,errors=remount-ro 0 1

tmpfs /tmp tmpfs defaults,noatime,nosuid 0 0

tmpfs /var/log tmpfs defaults,noatime,nosuid,size=50M 0 0

tmpfs /prometheus-tmpfs tmpfs defaults,nosuid,size=200M

What did you do?
Prometheus + node_exporter on Buster
saving prometheus in TMPFS to save SD writes

What did you expect to see?
It working

What did you see instead? Under which circumstances?
faliures:

Jan 25 03:12:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:12:23.027Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:13:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:13:23.069Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:14:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:14:23.150Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:15:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:15:23.186Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:16:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:16:23.217Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:17:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:17:23.254Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:18:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:18:23.316Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:19:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:19:23.391Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:20:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:20:23.414Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
Jan 25 03:21:23 raspberrypi prometheus[6999]: level=error ts=2021-01-25T03:21:23.430Z caller=db.go:745 component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"

Environment
Raspbian Buster

• System information:

Linux 5.4.83-v7l+ armv7l

• Prometheus version:

prometheus, version 2.24.1 (branch: HEAD, revision: e448727)
build user: root@0b5231a0de0f
build date: 20210120-01:04:17
go version: go1.15.6
platform: linux/arm

• Prometheus configuration file:

- job_name: 'node' scrape_interval: 60s scrape_timeout: 10s static_configs: - targets: ['localhost:9100'] - job_name: 'dump1090' scrape_interval: 60s scrape_timeout: 10s static_configs: - targets: ['localhost:9105'] =/usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path="/prometheus-tmpfs/prometheus" \ --storage.tsdb.min-block-duration=30m --storage.tsdb.max-block-duration=1h \ --storage.tsdb.wal-compression \ --storage.tsdb.retention.time=1d --storage.tsdb.retention.size=50MB \ --storage.tsdb.no-lockfile \ --web.enable-admin-api 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/510fc30b-c42f-4884-b9e7-b3257540db5dn%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[prommyonapi]

Let me get this straight, poor software design choices result in destroying any local tmpfs IoT device use case for Prometheus and anyone that points this out is banned and given some bullshit code of conduct reason.  Anyone using Prometheus on flash memory is writing hundreds of MB of data an hour or a day, or worse to stop kilobytes of   data thus obliterating the majority of IoT storage in the process.

Got it.

Well done.   Well done.

I guess there's no need to use Prometheus at this point for local IoT.

 Good job.   

[prommyonapi]

It's only taken 18 months to get to this answer from the devs.

Perhaps someone should be less woke and fucking DOCUMENT THIS SHIT.

[Ben Kochie]

In fact, the good design choices made in 2.x were intentional to help embedded use cases. The updated version reduces IO by a factor of 100. Eliminating the damaging write patterns that existed in 1.x. It is no longer required to write to tmpfs in order to protect flash storage from poor write patterns that existed in Prometheus 1.x.

This change was well documented in the 2.0.0 release notes and announcements in 2017.

https://github.com/prometheus/prometheus/blob/master/CHANGELOG.md#200--2017-11-08

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a62b0b9c-60d3-4595-8d62-26c6180fe11cn%40googlegroups.com.

[prommyonapi]

Let's see, no mention of requiring 100's of MB of TMPS that totals to more RAM than the most widely used IoT devices have on your links.  One page is 404, and the other page makes only a passing mention of disk i/o with no queries.

Again,

How does requiring hundreds of megabytes of disk write to store a 100 kilobyte of data not thrash flash storage?
Is there a way to disable this massive disk space requirement?