Better file system

[Евгений Смышляев]

Hi!

I use prometheus federation for scraping large set of my clusters. It generate much metrics and now size of database is increasing to 350Gb. So, what file system is better for this purposes? In this moment I use btrfs. In this case real data use 190Gb and overhead for fragmentation about 160Gb.

What filesystem (or other storage parameters) you recommended for this size of database?

[Ben Kochie]

First, Federation is not mirroring large amounts of metrics.  It's for aggregating small numbers (1000 or fewer) cluster summary metrics.

For sending all data to a central Prometheus storage, I would recommend remote write and Cortex[0].

As for filesystems, for Prometheus 1.x, we typically recommend ext4 or xfs.  The only down side to ext4 is the static inode allocations, so when you get into millions of metrics with high label churn you can run out of inodes.

Prometheus 2.x does not suffer from the inode issues due to the new TSDB design.

[0]: https://github.com/weaveworks/cortex/

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e08e61c3-1acf-49c0-8b05-296ec472858b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Евгений Смышляев]

In this way you recommend architecture like this?

why it better than store all in federated prometheus internal storage?

четверг, 18 января 2018 г., 18:42:56 UTC+3 пользователь Ben Kochie написал:

First, Federation is not mirroring large amounts of metrics.  It's for aggregating small numbers (1000 or fewer) cluster summary metrics.

For sending all data to a central Prometheus storage, I would recommend remote write and Cortex[0].

As for filesystems, for Prometheus 1.x, we typically recommend ext4 or xfs.  The only down side to ext4 is the static inode allocations, so when you get into millions of metrics with high label churn you can run out of inodes.

Prometheus 2.x does not suffer from the inode issues due to the new TSDB design.

[0]: https://github.com/weaveworks/cortex/

On Thu, Jan 18, 2018 at 3:39 PM, Евгений Смышляев <amp...@gmail.com> wrote:

Hi!

I use prometheus federation for scraping large set of my clusters. It generate much metrics and now size of database is increasing to 350Gb. So, what file system is better for this purposes? In this moment I use btrfs. In this case real data use 190Gb and overhead for fragmentation about 160Gb.

What filesystem (or other storage parameters) you recommended for this size of database?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.