Size of Prometheus data folder grows even with identical samples

[Charles Dayan]

I am performing some tests recording 1000 identical samples every 10 seconds. Besides, there are 4 other variables from Prometheus: up, scrape_samples_scraped, scrape_duration_seconds and scrape_samples_post_metric_relabeling.

 

The gauge value is always 123.456 for all variables. From what I understood of the value compression approach (https://promcon.io/2016-berlin/talks/the-prometheus-time-series-database/) the storage should be mininal. But the rate of increase of the data folder size indicates that almost 2 bytes are been used per sample. When I use sines and cosines with 5% noise this figure jumps to 8.6 bytes per sample!

Am I missing something? A flag or some other configuration? 

Is there a better way to estimate the required storage per sample? 

Thanks...

[Ben Kochie]

Are you looking at the wal or the compacted TSDB dirs?

The wal directory takes up quite a bit more storage, as it is not well compressed.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a131d573-9b67-404a-b5d9-eb2edd66c488%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charles Dayan]

Sorry for the late response.

I am recordind the size of the "data" directory. It increases at rate of ~2 bytes per sample. After the "2 hours update" almost 1.6 MB is added. I monitored along two full days and this value was very reproducible.

The main question is that kind of data (constant, with few significant numbers and with fixed scrape time) should take a negligible amount of storage. But 2 bytes per sample doesn't seems right!

[Ben Kochie]

You really do need to look more closely than just the `data/` directory as a single unit. Examining the wal, the TSDB directories, the index and chunks separately within each of those directories.

WIthout the output of `find data/ -ls`, I can't tell you what's going on.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/16d6dcdb-8a74-4041-8ed0-308aeea9e599%40googlegroups.com.

[Charles Dayan]

Thank you for the attention. These are some information about the test.

The scraped text is always the same (1000 gauge variables every 10 seconds; "only" 6 significant digits and no Prometheus variables, but 4):

# HELP Var0Unit0 Informative/explanatory text about the variable

# TYPE Var0Unit0 gauge

Var0Unit0 123.456

# HELP Var1Unit1 Informative/explanatory text about the variable

# TYPE Var1Unit1 gauge

Var1Unit1 123.456

...

... ...

# HELP Var998Unit998 Informative/explanatory text about the variable

# TYPE Var998Unit998 gauge

Var998Unit998 123.456

# HELP Var999Unit999 Informative/explanatory text about the variable

# TYPE Var999Unit999 gauge

Var999Unit999 123.456

I restarted the test some days ago and these are the files inside the "data" directory:

The content of the meta.json of the first two (the biggest ones):

{

"ulid": "01CK78AN71HA96JCVPBK9B7EV8",

"minTime": 1532404800000,

"maxTime": 1532455200000,

"stats": {

"numSamples": 4495912,

"numSeries": 1004,

"numChunks": 37148

},

"compaction": {

"level": 3,

"sources": [

"01CK5HCTVQY6A2ME51J08PFPPV",

"01CK5R8J3E02ZCFHQVPD7R8Z30",

"01CK5Z49B8S05H0DVV24V4AHZ8",

"01CK6600N006JRK0961YVVZP4F",

"01CK6CVQVGEHA97XBNY9CQWM3X",

"01CK6KQF3JV3H8Y2XDRD505QJM",

"01CK6TK6B2GX1ZWPSNTH5HENVX"

]

},

"version": 1

}

And

{

"ulid": "01CK8Z8NAWXF4RQEWDAWNQSJAB",

"minTime": 1532455200000,

"maxTime": 1532520000000,

"stats": {

"numSamples": 4669384,

"numSeries": 1004,

"numChunks": 39212

},

"compaction": {

"level": 3,

"sources": [

"01CK71EXKADASB438GYEMFT15B",

"01CK78AMV4KX5PKP28JRHQ5113",

"01CK7F6C2XMY501KXBCHACH63G",

"01CK7P23B7NV935XWHVQXH9P8E",

"01CK7WXTK2RE21V4AV989828B7",

"01CK84QNB5NTPSK5CWBNYNJEKK",

"01CK8ANFFQ9MGYNDW0SZXM79HF",

"01CK8HH6Q6QEX8H4F2P3352YX4",

"01CK8RCXZSCZXKMP9X4F5ET1MT"

]

},

"version": 1

}

And, finally, the following graph shows the behavior of the size of the "data" directory:

I put attention on the "whole required storage" because, at the end of the day, is this figure that I have to use to estimate the storage needed on the client's HD!

Thanks. 

[Charles Dayan]

I am still striving to cope this subject.

My simplistic test is very easy to replicate (as described in the first message). If you try, could you inform if your results are consistent to those present in the graph ("data" folder grows ~0.75MB every hour)?

Putting my question in another form:

How do you estimate the storage needed when dealing with values that comes from industrial sensors (figures with few significant numbers and an inerent noise, i.e. ever changing)? Is there a way to force float32 representation?

Thanks! ;)

[Matthias Rampke]

Hey,

in any case, a synthetic test is only going to give you so much information about the real world usage, especially as you say the real data is noisy.

From your screenshot, the WAL is indeed the biggest part of the disk space used. It isn't compressed in the same way the finished chunks are, and because constant values compress so well it is much much bigger than anything else.

On real-world large-volume data, I would expect 1.5-2 bytes per sample, but again, only testing with real data will get you reliable data here.

There is no way to use float32. It wouldn't make a significant difference in the compressed chunks.

0.75MB/hour convert to 126MB/week, I don't think you'll get anything less. Prometheus is designed as a server-side application where storage is not *quite* as limited, so I don't think we have micro-optimised it to the extent that you are trying to benchmark here.

I hope this helps!

Matthias

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f044043e-8695-4799-95c3-92b164f5c76b%40googlegroups.com.

[Charles Dayan]

Matthias Rampke, I understood your comments, but the main point is that in this test I am storing 1000 variables always with the same value (123.456) in a regular time interval (10s). The storage should be minimal (as desbribed in https://promcon.io/2016-berlin/talks/the-prometheus-time-series-database/  - at 21:38 ==> 0 bit/sample )!! And even in this "quite-easy-to-compact" case, the "actual" storage takes ~2 bytes per sample. (much worst, 8.6, if the values change!)

For float point numbers, there is no simpler case then this! 

If my procedure is sort of wrong, please share how would be the "right way". But my pratical vision always moves towards monitoring the size of the "data" folder. 

PS: I have no complain about the size of WAL folder  ;) 

Thanks!

 

[Julius Volz]

So the bytes-per-sample value estimate is for fully-persisted finished blocks, which are kept around forever. There is some fixed-time-range overhead of the WAL, but that only ever covers the most recent 2-3h of samples at most, so for long retention times it doesn't have much weight.

I ran your example for a couple of hours and for a full 2h chunk of data I'm getting a "du" disk usage for the block dir of 480KB for 722880 samples, which comes out to 0.68 bytes per sample. Once blocks get compacted into larger ones, that will likely even become less.

So yeah. Basically you have to ignore the WAL overhead as it's fixed-time-range and won't matter much if you have any kind of long-ish retention time.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8841205b-fb05-481e-90f4-ef0f972ac0a5%40googlegroups.com.

[Charles Dayan]

It seems we are (actually "I am"!) near to finish this subject, but I still have some points to comment. 

    When I put this graph, I am not concerned about the size of the WAL folder. What is taking my attention is the slope 0.7434 MB/h 

      I should be deadlocked in a wrong perspective (but how to ignore the ever growing size of the folder - I run for days and the slope was constant) !!  

     Could you shortly explain how is your procedure (calculation or which file are you monitoring the size) ?

      Thank you very much!

       

 

[Julius Volz]

Well looking at your block dirs, you have some that are 8MB or 9MB large, but you also have some very small ones. Are those from times when your Prometheus was scraping more or fewer samples? In any case, the blocks in the screenshot you shared don't look like they fit the test data you mentioned at all.

If I look at my storage as configured with your stated 1000 test series, it looks like this with "du":

-----------------------

$ du -hcs *     

156K 01CM2GC20GQ0MB4AKQPJ43NZC8

480K 01CM2Q7S8VVCPX6V93YNKYD4HN

480K 01CM2Y3GGM6SNE10JNCY28R3C2

0 lock

279M wal

280M total

-----------------------

The first 156K block is one that was only partially filled before it was persisted because it was the first block to be persisted and the 2h range does not start with the TSDB's first initialization, but is grided against some absolute-time 2h grid. So only the other (480KB) ones are representative of how much your storage should grow every 2h (until retention time is hit, which removes old blocks again). Compaction isn't factored in yet, which later on combines smaller blocks into larger ones (in levels, always with a factor of 3x) and will likely also save more space.

So every subsequent 2h just takes up 480KB. And you can see that one of the larger blocks has the right number of samples:

-----------------------

$ cat 01CM2Q7S8VVCPX6V93YNKYD4HN/meta.json 

{

"ulid": "01CM2Q7S8VVCPX6V93YNKYD4HN",

"minTime": 1533384000000,

"maxTime": 1533391200000,

"stats": {

"numSamples": 722880,

"numSeries": 1004,

"numChunks": 6024

},

"compaction": {

"level": 1,

"sources": [

"01CM2Q7S8VVCPX6V93YNKYD4HN"

]

},

"version": 1

}

-----------------------

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/34aca717-b7fc-419c-ae19-2fe030d5fe2e%40googlegroups.com.

[Charles Dayan]

Thank you for the directions. 

I will restart everything with an empty setup and will inform what cames out of this. 

:)

[celia.m...@gmail.com]

Hi! I am here again! 

Same situation; 1000 variables equals to 123.456 sampled every 10s. Retrieving data for Var999Unit999:

http://localhost:9090/api/v1/query?query=Var999Unit999[3d]

{"status":"success","data":{"resultType":"matrix","result":[{"metric":

{"__name__":"Var999Unit999","instance":"localhost:9898","job":"plant_data"},

"values":[

[1533430000.184,"123.456"],[1533430010.184,"123.456"],

[1533430020.185,"123.456"],[1533430030.184,"123.456"],

[1533430040.185,"123.456"],[1533430050.186,"123.456"],

...

   ...

[1533614340.184,"123.456"],[1533614350.185,"123.456"],

[1533614360.184,"123.456"],[1533614370.184,"123.456"],

[1533614380.184,"123.456"],[1533614390.184,"123.456"]

]}]}}

This table shows the sum up of 51h13m10s :

Same result! As you can see, I am still stucked in ~2 bytes/sample.

Far, far away from 0.066 bits per sample:

As stressed by Björn Rabenstein in his speach: "bits not bytes"!! (Did anybody know how to reproduce it?)

My goal now is to reproduce Julius Volz's 0.68 bytes per sample (some answers above)!!! "My" present value is three times that!

What could cause the diference? I am using Prometheus 2.3.2 on Windows 10. 

Maybe some configuration tweak or flag in "yml" file? I am using the following to start the test:

prometheus.exe --web.enable-admin-api --config.file=.\prometheus_size_test.yml --storage.tsdb.path=.\data --storage.tsdb.retention=280d

I noticed in the "meta.json" files the entry "version":1 . Is there another version ("2"?) with better compaction? 

That's it. I feel myself trapped in a maze!

[Charles Dayan]

Up to now, I don't know yet if the storage required to keep a constant value sampled at a regular interval should take just few bits (or less!) or two bytes per sample.

Could anyone suggest any reliable testing setup? Or share your own results to benchmark against the reported value.

Thanks!