Storage: Huge number of files

[Martin Schröder]

Hi,
we are using prometheus for ca. 40 machines with a retention
period of two weeks (all other storage settings are defaults)
and see that our prometheus server creates a huge number of
files (>1 million) as storage.

Because of this we already had to change the partition (ext4) to
allow for more inodes (IT insists on using ext4 :() and now the
backup process (bacula) for the prometheus data partition is timing
out.

AFAIK there is currently no other option for storage.

How do others cope with this?

Best
Martin

[Björn Rabenstein]

On 21 November 2016 at 11:40, Martin Schröder <mar...@oneiros.de> wrote:
>
> we are using prometheus for ca. 40 machines with a retention
> period of two weeks (all other storage settings are defaults)
> and see that our prometheus server creates a huge number of
> files (>1 million) as storage.

Yes. This is deliberate design.

> How do others cope with this?

The only time we had trouble with this at SoundCloud was when we
changed our filesystem default to ext4 with number of inodes on the
low side (to save overhead). So we create partitons for Prometheus
with a higher inode count. The default setting is usually enough
(which was also our benchmark when we made the design decision for
"one file per time series').

There are situations of extreme churn (like deploy a service with many
instances every half hour or something), but in that case, you run
into other problems (e.g. with indexing), too.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Martin Schröder]

2016-11-21 12:55 GMT+01:00 Björn Rabenstein <bjo...@soundcloud.com>:
> On 21 November 2016 at 11:40, Martin Schröder <mar...@oneiros.de> wrote:
>> and see that our prometheus server creates a huge number of
>> files (>1 million) as storage.
>
> Yes. This is deliberate design.

Understood. IMHO this should be documented better.

Would it be useful to add another dir level in metrics?
Currently the dirs have ca. 8k files.

> The only time we had trouble with this at SoundCloud was when we
> changed our filesystem default to ext4 with number of inodes on the
> low side (to save overhead). So we create partitons for Prometheus

In your opinion is ext4 a good choice or would xfs be much better?

Best
Martin

[Björn Rabenstein]

On 21 November 2016 at 13:12, Martin Schröder <mar...@oneiros.de> wrote:
> Understood. IMHO this should be documented better.

It's in the first paragraph of https://prometheus.io/docs/operating/storage/ .
It's in the FAQs:
https://prometheus.io/docs/introduction/faq/#why-does-prometheus-use-a-custom-storage-backend-rather-than-some-other-storage-method-isn-t-the-one-file-per-time-series-approach-killing-performance

As far as I can see, we don't have an explicit warning anywhere to
adjust the inode settings. On the other hand, it is usually only an
issue on ext4,. and only if you set the inode count deliberately below
the default settings.

And indeed, the number of people reporting trouble with inodes is
relatively low.

Happy to be proven otherwise, and we'll add something more explicit somewhere.

> Would it be useful to add another dir level in metrics?
> Currently the dirs have ca. 8k files.

This should be fine for modern file systems.

We are prepared to introduce another directory fan-out layer, but I'd
expect it will not improve performance (or might even degrade it) in
most situations.

Have you tried benchmarks?

> In your opinion is ext4 a good choice or would xfs be much better?

I'm emotionally attached as I used XFS even before I used Linux for
the first time. :)
I think it's awesome, and not needing to think about inode counts is a
plus here, obviously.
Having said that, we didn't really notice any striking performance
differences WRT Prometheus.
Just make sure you have enough inodes...

[dya...@gmail.com]

From practical experience, there is very little difference these days between XFS and ext4 on heavily used filesystems, both have minor advantages. Depending on why backula is failing, there may be advantages to tuning the filesystem variables at the kernel level, instead of looking at new filesystem types. On CentOS/RHEL, tuned can be a good starting point, as can looking at controlling whether or not you are tracking access times, etc on a mounted partition.

I have huge filesystems with millions of files on them that are manageable around backups, but it took a fair amount of tuning the Linux system to be efficient.

David

Best
   Martin