Does prometheus use compression to store its data ?

[Mega Rajan]

hi 

we are using prometheus 2.3.1 .

Does prometheus use compression to store its data ?

From the documentation I see that its being compacted but no reference to compression and any compression algorithm as such .

could anyone please clarify on the compression and compression algorithm used by prometheus if any.

thanks

Megarajan

[Ben Kochie]

Yes, Prometheus uses compression.

See: https://www.youtube.com/watch?v=b_pEevMAC3I

Also, I highly recommend upgrading to the latest release. The version you are running is from 2018 and has numerous bugs.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1a63ac93-4e31-4df5-9e35-3803bd137f0en%40googlegroups.com.

[Mega Rajan]

Thanks for the reply. 

I digged in and saw that Prometheus uses Gorilla compression.

Do we have option to change the compression method to something else . I see that Gorilla compression is an efficient one but wanted a confirmation on whether there is a way to change it to something else 

thanks

Megarajan

[Ben Kochie]

PRs welcome.

But remember, any changes need to support mmap for efficient caching of stored data.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6e850231-f284-49ba-b0bb-76156340c0f0n%40googlegroups.com.

[Brian Candler]

If you're worried about compression of long-term archival storage, have a look at Thanos.

If you want to compare disk usage to a system with a similar design to Prometheus but different compression, have a look at VictoriaMetrics:

https://faun.pub/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932

However, disk space is cheap.  Other considerations may be more important for many users.  And Prometheus in 2022 may be different to Prometheus in 2017.

[Ben Kochie]

On Fri, Aug 26, 2022 at 9:31 AM Brian Candler <b.ca...@pobox.com> wrote:

If you're worried about compression of long-term archival storage, have a look at Thanos.

If you want to compare disk usage to a system with a similar design to Prometheus but different compression, have a look at VictoriaMetrics:

https://faun.pub/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932

Does VictoriaMetrics still cast float64 to int to round off bits and improve compression?

 

However, disk space is cheap.  Other considerations may be more important for many users.  And Prometheus in 2022 may be different to Prometheus in 2017.

For example, WAL compression was introduced in 2019 and became the default in 2020.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a2562fc6-a660-4ee1-9af2-cf58a5d73e99n%40googlegroups.com.