Regex pattern matching on metric label values

[Ketan Talreja]

Hello All, 

I am new to prometheus space, and am currently using it do some poc's and understand its concepts. I have a use case where from the source application I am receiving metrics in the json string format and the source is a third party library. 

Below is the sample metric output from source: - 

{

    "status": "success",

    "data": {

        "resultType": "vector",

        "result": [

            {

                "metric": {

                    "__name__": "workflow_executions",

                    "instance": "localhost:9920",

                    "job": "exporter-application",

                    "result": "[{'_index': 'abc', '_type': 'workflow', '_id': '4b01fe28-3c80-4595-a1be-3861b037f9ba', '_score': 1.0, '_source': {'type': '512156fd-e18f-4563-981b-2a1e1d24ecdf', 'version': 2, 'startTime': '2020-10-01T12:20:03.629Z', 'status': 'COMPLETED', 'input': '{executionId=c134e4e0-62e2-4190-9d9e-755fef48faed, orgId=7e04226a-1a82-4844-ba36-357a35f6677e}', 'output': '{result={\"version\":1,\"priority\":0,\"status\":\"COMPLETED\"}}', 'executionTime': 4284, 'failedReferenceTaskNames': '', 'priority': 0, 'inputSize': 1115, 'outputSize': 6944, 'updateTime': '2020-10-01T12:20:07.913Z', 'endTime': '2020-10-01T12:20:07.913Z'}}, {'_index': 'abc', '_type': 'workflow', '_id': '28ddb129-97ea-49c3-a074-a61a739905a7', '_score': 1.0, '_source': {'type': 'a6212938-7de8-42e9-a9c5-8b5b97b4a284', 'version': 1, 'workflowId': '28ddb129-97ea-49c3-a074-a61a739905a7', 'startTime': '2020-09-24T21:20:17.362Z', 'status': 'COMPLETED', 'input': '{executionId=2e81bdcb-d369-4892-84fc-4a5d33a8fb0b, orgId=7e04226a-1a82-4844-ba36-357a35f6677e}', 'output': '{result=success}', 'executionTime': 3021, 'failedReferenceTaskNames': '', 'priority': 0, 'outputSize': 16, 'inputSize': 1115, 'updateTime': '2020-09-24T21:20:20.383Z', 'endTime': '2020-09-24T21:20:20.383Z'}}, {'_index': 'xyz', '_type': 'workflow', '_id': '4e6c257e-04a7-4377-a072-1b8df9658f2f', '_score': 1.0, '_source': {'type': '4b10a1d4-47b8-419e-842c-694fbe3a9985', 'version': 1, 'workflowId': '4e6c257e-04a7-4377-a072-1b8df9658f2f', 'startTime': '2020-09-30T20:51:10.590Z', 'status': 'COMPLETED', 'input': '{executionId=a22c16ef-b1b2-4049-a095-5cc088275625,orgId=7e04226a-1a82-4844-ba36-357a35f6677e}', 'output': '{result=success}', 'executionTime': 3601, 'failedReferenceTaskNames': '', 'priority': 0, 'inputSize': 1115, 'outputSize': 16, 'updateTime': '2020-09-30T20:51:14.191Z', 'endTime': '2020-09-30T20:51:14.191Z'}}]"

                },

                "value": [

                    1602027104.962,

                    "1"

                ]

            }

        ]

    }

}

Here, I would like to read and count number of workflows with "status: COMPLETED" ? 

I have tried few things using promQL but no luck. e.g. 

workflow_executions{result=~"'status': 'COMPLETED'"} 

workflow_executions{result=~"result=success"}

I am using grafana for visualization, and tried above queries but didn't work. 

Can anyone advise or guide me where i am going wrong?     

Thanks in advance, Ketan!

[l.mi...@gmail.com]

The short but very important line on https://prometheus.io/docs/prometheus/latest/querying/basics/ is "Regex-matches are fully anchored."
Which means that when you query result=~"result=success" promql will run a regex match using "^result=success$" as regex value, so it's not gonna match "result=success" as a substring. You need to query with result=~".*result=success.*" to allow for that.

[l.mi...@gmail.com]

Obviously that's ignoring the fact that using JSON blob as a label value is not recommended. As a starting point labels should have concrete values rather than be any complex data structure like JSON, that way you'll be able to query it easily and avoid having problems like the one above.

First problem will be cardinality - each unique label value will be a new time series.

Second it's just too much information, it looks like an even rather than a metric, maybe you are after a logging system (unless you're going to trim away most of fields from that blob) instead.