Thanos vs TimescaleDB

[ppan...@gmail.com]

Hi team,

I have been reading about TimeScale DB and prometheus with federation approach.

If we designing a metrics collection platform for 3k+ nodes, which one is better from scalability, longterm storage, HA and Operational efficiency standpoint?

Is there good bench marking or articles related to this?

[Chris Bulleri]

I just saw the Thanos information and would love any information as well.

This is the only thing I've seen so far.

https://improbable.io/games/blog/thanos-prometheus-at-scale

[Aliaksandr Valialkin]

Take into account that TimescaleDB requires way more disk storage comparing to Prometheus and, consequently, to Thanos, which is built on Prometheus' storage engine. TimescaleDB requires 30 bytes per each (timestamp, value) data point, while Prometheus (Thanos) usually requires 1-4 bytes per data point, i.e. 8 - 30 times less disk space comparing to TimescaleDB.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ee995063-f67b-43d8-83c0-a61057322186%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best Regards,

Aliaksandr

[Harkishen Singh]

That is not true at all. After compression in TimescaleDB, the timestamp undergoes double delta compression (like in facebook's gorilla papers) in a more efficient manner, fusing the rows and achieving a much higher compression ratio. That clearly means the stats you mentioned is false!

[Brian Candler]

On Sunday, 4 October 2020 19:14:53 UTC+1, Harkishen Singh wrote:

That is not true at all. After compression in TimescaleDB, the timestamp undergoes double delta compression (like in facebook's gorilla papers) in a more efficient manner, fusing the rows and achieving a much higher compression ratio. That clearly means the stats you mentioned is false!

Would you like to provide evidence to back that up, i.e. an actual experimental setup which reproducibly supports your claim?

[David Kohn]

Just to clarify here, the original post from Aliaksandr was from before Timescale released compression, so that was true at the time, in fact the large amount of storage space used was one of the main reasons we developed compression. With compression it is no longer true. You can take a look at blog posts here: https://blog.timescale.com/blog/building-columnar-compression-in-a-row-oriented-database/ and https://blog.timescale.com/blog/time-series-compression-algorithms-explained/  for some more information on the ways we do that. We also have Promscale now: https://github.com/timescale/promscale which automatically compresses data, uses a specially developed schema for storing Prometheus data and with which you can easily set up and run your own test to see how well it does in terms of space utilization. I'd let it run for a little bit (say a couple days) before comparing as we do maintain a region of uncompressed data in the recent past. Hope that helps clarify. 

-David