Prometheus performance characteristics

[lavrinen...@gmail.com]

Hi Prometheus developers,

I'm compiling an overview of TSDBs available on the market. I could not find enough materials on Prometheus performance, so wonder if you have them and could share?

The ones I'm interested in are:
- For some deployment examples
Cores and RAM used
Number of Time Series
Ingestion rate (samples and MB)
Ingestion latency (data available for query)
QPS
Query Latency (90 and 99%%)
#Shards
- Biggest known deployment
- Average memory used per data sample

Thank you,
Sergii

[Brian Brazil]

On Fri, Oct 9, 2015 at 5:02 PM, <lavrinen...@gmail.com> wrote:

Hi Prometheus developers,

I'm compiling an overview of TSDBs available on the market. I could not find enough materials on Prometheus performance, so wonder if you have them and could share?

The ones I'm interested in are:
- For some deployment examples
  Cores and RAM used
  Number of Time Series
  Ingestion rate (samples and MB)

http://prometheus.io/docs/introduction/faq/#why-does-prometheus-use-a-custom-storage-backend-rather-than-some-other-storage-method-isn-t-the-one-file-per-time-series-approach-killing-performance are the main real numbers we have on the limits of that.

 

  Ingestion latency (data available for query)
  QPS
  Query Latency (90 and 99%%)

  #Shards
- Biggest known deployment

 

- Average memory used per data sample

There are around 3.5 bytes per sample on disk.

Brian

 

Thank you,
Sergii

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Björn Rabenstein]

On Fri, Oct 9, 2015 at 6:09 PM, Brian Brazil
<brian....@robustperception.io> wrote:
> On Fri, Oct 9, 2015 at 5:02 PM, <lavrinen...@gmail.com> wrote:
>>
>> The ones I'm interested in are:
>> - For some deployment examples
>> Cores and RAM used
>> Number of Time Series
>> Ingestion rate (samples and MB)
>
>
> http://prometheus.io/docs/introduction/faq/#why-does-prometheus-use-a-custom-storage-backend-rather-than-some-other-storage-method-isn-t-the-one-file-per-time-series-approach-killing-performance
> are the main real numbers we have on the limits of that.

And you can already see that a lot of parameters go into the equation.

The ingestion rate in MB depends on the used format and if compression
is applied or not.

>> Ingestion latency (data available for query)

There is no delay in the design, although the indexing of new time
series may lag behind a tiny bit.

>> QPS
>> Query Latency (90 and 99%%)

That depends so heavily on the nature of the query that it cannot be
answered in general.

>> #Shards
>> - Biggest known deployment

I guess exact data will be difficult to get due non-disclosure requirements.
SoundCloud has dozens of Prometheus servers (but less than 100).

>> - Average memory used per data sample
>
>
> There are around 3.5 bytes per sample on disk.

That also depends heavily on the nature of the data. In a recent spot
check, SoundCloud data was more like 4.5 bytes per sample.

Note that the compression used applies to samples in memory as well as
on disk, i.e. no special compression kicks in when writing to disk.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[lavrinen...@gmail.com]

Hi Bjorn,

Thank you for additional information.

I understand that many parameters vary significantly on the type of data (like number of key-value pairs and compressibility of samples). That is why it would be good to have perf measured on some prod or synthetic deployment with additional knowledge of dimensions and string lengths statistics.

Regarding ingestion latency (data available for query), say at speed 20k events/sec/core how big lag with indexing might be? Again, likely depends on the number of dimensions, but we could take some average numbers like 6-8 dimensions, 40 bytes key and value size.

As far as I understand (and please let me know if this makes sense) QPS and Query Latency metrics can be separated to two each:
1. For E2E user queries with provided statistics how many time series and shards user query hits.
2. For individual time series queries: what QPS a shard can support given certain ingestion rate and what is its latency.

Do you have any of mentioned numbers that you could share?

Thanks again,
Sergii

[Björn Rabenstein]

On Sat, Oct 10, 2015 at 9:16 AM, <lavrinen...@gmail.com> wrote:
>
> I understand that many parameters vary significantly on the type of data (like number of key-value pairs and compressibility of samples). That is why it would be good to have perf measured on some prod or synthetic deployment with additional knowledge of dimensions and string lengths statistics.

Definitely. https://github.com/prometheus/prometheus/issues/168 is in
the same area. We want a nice, realistic, automated benchmark suite
for A/B testing of different Prometheus versions and also things like
comparing different storage backends.

> Regarding ingestion latency (data available for query), say at speed 20k events/sec/core how big lag with indexing might be? Again, likely depends on the number of dimensions, but we could take some average numbers like 6-8 dimensions, 40 bytes key and value size.

Prometheus exposes metrics about that. There is
`prometheus_local_storage_indexing_queue_length`, which tells you how
many batches of samples to index are waiting in the queue. This should
be 0 most of the time. (I just checked our meta monitoring. Among the
score of Prometheus servers we run, only one has occasional peaks
here, and it is the beefiest and busiest.) Then there is
`prometheus_local_storage_indexing_batch_duration_milliseconds`, which
tells you how long it took to process a batch of metrics to index. The
90th percentile is usually below 500ms. On our outlier mentioned
above, it is 4s.

So I guess it is safe to assume that a new metric will be available
for querying after 500ms in most cases.

But note that this is only relevant for newly appearing metrics. For
existing metrics (one for which we have ingested before), the ingested
sample will be available for queries immediately.

> As far as I understand (and please let me know if this makes sense) QPS and Query Latency metrics can be separated to two each:
> 1. For E2E user queries with provided statistics how many time series and shards user query hits.
> 2. For individual time series queries: what QPS a shard can support given certain ingestion rate and what is its latency.

Sounds about right, but we have no precise measurements for those.
Note that there are no cross-shard queries in Prometheus. The
dashboard builder might query different Prometheus servers, but
otherwise, you would need to set up a federating server to combine
time series from different servers in one server and query them there.

The "Graph" tab on the Prometheus Web UI gives you timing numbers you
can use as a guideline. If I do just random queries on a mid-size
production server right now:

Single point in single time series: ~100ms
1d range of samples (5min resolution) in single time series: ~180ms
Single point in 120 time series: ~120ms
1d range of samples (5min resolution) in 120 time series: ~200ms

This is all if serving from memory. If data is accessed that has been
evicted to disk, disk loads come into the game.

Probably a large part of the above is just overhead from marshalling,
the HTTP round trip, work in the browser, ...

We never really loadtested the query part as the queries are usually
not the heavy-lifting (which is ingestion). Even if many dashboards
access the same server all the time, problems only arise with very
complex queries, for which then recording rules need to be created,
which removes the problem.