High cardinality labels

[Karthik Subraveti]

Hello all,

I have an application where i want to use lots of labels around 100k+. I don't have too many metrics otherwise and we can assume that total number of timeseries 

caused due to these labels might remain under 2million. I was wondering apart from querying on the top level metric(without label filter), are there any

other things to be aware of which can cause poor behavior in prometheus. 

Second question is i'm running a test in the scenario described above and i see RSS going up constantly. This is the output from the tsdb status 

curl -s localhost:9090/api/v1/status/tsdb | python3 -m json.tool

{

    "status": "success",

    "data": {

        "seriesCountByMetricName": [

            {

                "name": "testAttr",

                "value": 500000

            },

            {

                "name": "prometheus_http_request_duration_seconds_bucket",

                "value": 80

            },

            {

                "name": "prometheus_http_response_size_bytes_bucket",

                "value": 72

            },

            {

                "name": "prometheus_sd_kubernetes_events_total",

                "value": 15

            },

            {

                "name": "prometheus_tsdb_compaction_chunk_samples_bucket",

                "value": 13

            },

            {

                "name": "prometheus_tsdb_compaction_chunk_size_bytes_bucket",

                "value": 13

            },

            {

                "name": "net_conntrack_dialer_conn_failed_total",

                "value": 12

            },

            {

                "name": "prometheus_tsdb_tombstone_cleanup_seconds_bucket",

                "value": 12

            },

            {

                "name": "prometheus_engine_query_duration_seconds",

                "value": 12

            },

            {

                "name": "prometheus_tsdb_compaction_duration_seconds_bucket",

                "value": 11

            }

        ],

        "labelValueCountByLabelName": [

            {

                "name": "sid",

                "value": 500000

            },

            {

                "name": "__name__",

                "value": 187

            },

            {

                "name": "le",

                "value": 66

            },

            {

                "name": "quantile",

                "value": 9

            },

            {

                "name": "handler",

                "value": 8

            },

            {

                "name": "role",

                "value": 5

            },

            {

                "name": "reason",

                "value": 4

            },

            {

                "name": "code",

                "value": 4

            },

            {

                "name": "slice",

                "value": 4

            },

            {

                "name": "event",

                "value": 3

            }

        ],

        "memoryInBytesByLabelName": [

            {

                "name": "sid",

                "value": 4888890

            },

            {

                "name": "__name__",

                "value": 7032

            },

            {

                "name": "le",

                "value": 307

            },

            {

                "name": "handler",

                "value": 109

            },

            {

                "name": "slice",

                "value": 43

            },

            {

                "name": "revision",

                "value": 40

            },

            {

                "name": "instance",

                "value": 32

            },

            {

                "name": "reason",

                "value": 31

            },

            {

                "name": "role",

                "value": 30

            },

            {

                "name": "quantile",

                "value": 28

            }

        ],

        "seriesCountByLabelValuePair": [

            {

                "name": "job=myapp",

                "value": 500043

            },

            {

                "name": "instance=192.168.0.124:2112",

                "value": 500043

            },

            {

                "name": "__name__=testAttr",

                "value": 500000

            },

            {

                "name": "instance=localhost:9090",

                "value": 517

            },

            {

                "name": "job=prometheus",

                "value": 517

            },

            {

                "name": "__name__=prometheus_http_request_duration_seconds_bucket",

                "value": 80

            },

            {

                "name": "__name__=prometheus_http_response_size_bytes_bucket",

                "value": 72

            },

            {

                "name": "handler=/api/v1/status/tsdb",

                "value": 24

            },

            {

                "name": "handler=/metrics",

                "value": 24

            },

            {

                "name": "handler=/api/v1/label/:name/values",

                "value": 24

            }

        ]

    }

}

..

Thanks

Karthik

[Brian Candler]

On Thursday, 2 April 2020 12:49:32 UTC+1, Karthik Subraveti wrote:

Second question is i'm running a test in the scenario described above and i see RSS going up constantly.

Does it stabilise after 2 hours?

There's a memory calculator here:

https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion

You could expect at least 3GB of RAM usage for 500K time series, 500K unique label pairs, 10 bytes per label pair, and 15-second scrapes.

[Karthik Subraveti]

I let the test run overnight and usage kept growing to around 4.4G. I have a screenshot at a previous timeslot.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/90b786ae-c6cd-4f03-9552-59cfca66c52d%40googlegroups.com.