Querying prometheus data/series.

[akshay sharma]

Hi,

I'm using prometheus golang library to push metrics to prometheus in prometheus format.

For ex:

# HELP STAT unistats
# TYPE STAT gauge
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="31",time="1608712842316890688",txByte="16"}    1
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="32",time="1608712842316941111",txByte="17"}     2
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="33",time="1608712842316954395",txByte="18"}    3
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="34",time="1608712842316966079",txByte="19"}    4
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="35",time="1608712842316976959",txByte="20"}    5
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="36",time="1608712842316988021",txByte="21"}    6

STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="37",time="1608712842316999306",txByte="22"}    7
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="38",time="1608712842317010782",txByte="23"}    8
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="39",time="1608712842317021443",txByte="24"}    9
STAT{IDF="false",Interval="0",Type="NT",ResourceID="t",rxByte="40",time="1608712842317030897",txByte="25"}   10

In prometheus GUI, I'm able to see metrics.

ex : 

ONTSTAT{IDF="false",Interval="0",METype="NT",ResourceID="t",instance="172.27.173.103:8999",job="prometheus",rxByte="31",time="1608711202557174342",txByte="16"}	1
ONTSTAT{IDF="false",Interval="0",METype="NT",ResourceID="t",instance="172.27.173.103:8999",job="prometheus",rxByte="32",time="1608711202557246491",txByte="17"}	2
ONTSTAT{IDF="false",Interval="0",METype="NT",ResourceID="t",instance="172.27.173.103:8999",job="prometheus",rxByte="33",time="1608711202557257990",txByte="18"}	3

1) I want to perform some action on the labels of metrics above, how can we achieve this?

ex: rxbyte-txbyte/timestamp

2) Above metrics that support multiple values?? like list or map 

3) As prometheus uses a TSDB, like influx, how can we query series from prometheus DB? 

4) Does promql support operations on labels?  

5) while storing metrics in Prometheus db, does it support multiple fields?? like influx. 

thanks!

[Christian Hoffmann]

On 2020-12-23 09:53, akshay sharma wrote:
> ONTSTAT{IDF="false",Interval="0",METype="NT",ResourceID="t",instance="172.27.173.103:8999
> ",job="prometheus",rxByte="33",time="1608711202557257990",txByte="18"}
> 3
>
>
> 1) I want to perform some action on the labels of metrics above, how can
> we achieve this?
> ex: rxbyte-txbyte/timestamp

You seem to have created a single metric with several labels which
contain values. This is problematic, because that's not what Prometheus
was designed for. You will likely get performance issues and will miss
methods to work with your data.

> 2) Above metrics that support multiple values?? like list or map

Not directly. This is solved using multiple metrics in Prometheus.

> 3) As prometheus uses a TSDB, like influx, how can we query series from
> prometheus DB?

Prometheus contains a TSDB. It can be queried using PromQL. There is no
other, secret interface or something.

> 4) Does promql support operations on labels?

Yes, it supports some operations (such as label_replace()), but this is
probably not what you are looking for. All values which you want to be
part of calculations should be the value of a metric, not the label content.

To provide more explicit guidance how your metrics should look like, we
would need some more context. I assume that you are monitoring some kind
of network interface. I guess it will have some kind of ID (maybe that's
the numbers 1 to 10 which you currently store as a value?). I'll just
assume it can be called device_id.

The metrics for one such device could look like this:

ontstat_rx_bytes{device_id="1"} 31
ontstat_tx_bytes{device_id="1"} 16
ontstat_idf{device_id="1"} 0 # 0 for false
ontstat_info{device_id="1",type="NT",resource_id="t"} 1 # always 1

If the value time= denotes the time when the metric was gathered, it
should be dropped entirely. Prometheus will figure this out on its own.
If this is some other timestamp (timestamp of last administrative
modification or something), it should be added as another metric:

ontstat_last_admin_action_timestamp_seconds{device_id="1"}
1608712842316890688

The value doesn't look like a unix timestamp. Maybe it should converted
to one (in seconds).

The following documents provide further background:
https://prometheus.io/docs/concepts/data_model/
https://prometheus.io/docs/practices/naming/

In fact, I just provided some examples for the hopefully right
direction. I'm probably missing something as well. Depending on context
you will also want to do some things differently (e.g. if it makes sense
semantically to sum up rx/tx metrics, you might consider making it a
single metric with a label to distinguish, e.g.
ontstat_traffic_bytes{device_id="1",queue="tx"}.

The above examples are the examples for a single device_id. You would
have additional such lines for every other device.

It may also make sense to look at the (network) metrics of an existing
exporter such as node_exporter.

Kind regards,
Christian

[akshay sharma]

Hi Christian,

Thanks for you reply,

ONTSTAT{IDF="false",Interval="0",METype="NT",ResourceID="t",instance="localhost:8999 ",job="prometheus",rxByte="33",time="1608711202557257990",txByte="18"}   4

what i've understood from your reply is , you are saying to break the metric into two metrics, one refers to rxbyte info with value and other one with txbyte info with value.

and then perform action (like rx-tx) on using both of them.?????

ex: as you mentioned.. 

ontstat_rx_bytes{device_id="1"} 31
ontstat_tx_bytes{device_id="1"} 16  

I've one query, suppose I'm receiving the same metrics with different rx and tx values, but labels are the same(deviceid), and I want to perform some calculation between 1st (rx tx metrics ) and last (tx rx metrics).

How can we achieve this?

 

can you please elaborate on the below comment??   

semantically to sum up rx/tx metrics, you might consider making it a
single metric with a label to distinguish, e.g.
ontstat_traffic_bytes{device_id="1",queue="tx"}. 

On Wed, Dec 23, 2020 at 7:28 PM Christian Hoffmann <ma...@hoffmann-christian.info> wrote:

On 2020-12-23 09:53, akshay sharma wrote:

> ONTSTAT{IDF="false",Interval="0",METype="NT",ResourceID="t",instance=":8999

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/54936447-5f4e-464a-6db7-b9359b8f3ac9%40hoffmann-christian.info.

[Stuart Clark]

On 23/12/2020 15:18, akshay sharma wrote:

Hi Christian,

Thanks for you reply,

ONTSTAT{IDF="false",Interval="0",METype="NT",ResourceID="t",instance="localhost:8999 ",job="prometheus",rxByte="33",time="1608711202557257990",txByte="18"}   4

what i've understood from your reply is , you are saying to break the metric into two metrics, one refers to rxbyte info with value and other one with txbyte info with value.

and then perform action (like rx-tx) on using both of them.?????

That is correct. Metrics are the values you are wanting to store over time. Labels attached to those metrics are purely descriptions of those metrics (device name, location, service, etc.) and not values which change over time themselves.

ex: as you mentioned.. 

ontstat_rx_bytes{device_id="1"} 31
ontstat_tx_bytes{device_id="1"} 16  

I've one query, suppose I'm receiving the same metrics with different rx and tx values, but labels are the same(deviceid), and I want to perform some calculation between 1st (rx tx metrics ) and last (tx rx metrics).

How can we achieve this?

It depends what calculation you are wanting to do. For example if you wanted to add the rx & tx values you could just do ontstat_rx_bytes + onststat_tx_bytes. For the example above you would get the result {device_id="1"} 47

can you please elaborate on the below comment??   

semantically to sum up rx/tx metrics, you might consider making it a
single metric with a label to distinguish, e.g.
ontstat_traffic_bytes{device_id="1",queue="tx"}. 

The design of your metrics should reflect their meaning. As mentioned labels are there to describe different subsets of a metric (e.g different devices). It should be possible to remove all labels via an aggregation such as sum() and still have a value which is meaningful.

For example if you had a metric counting number of requests it would make sense to have labels for status code or method. However it would not make sense to have a metric where the values with a label of "type=time" are durations and "type=number" are counts - that would be two different metrics.

[akshay sharma]

Thanks for the clarification,

What if,there is multiple metrics with same labels but different values

ex: 

STAT{METype="NT",ResourceID="t",queue="rx",time="1608786795151775007"}	31
STAT{METype="NT",ResourceID="t",queue="rx",time="1608786795151813374"}	32
STAT{METype="NT",ResourceID="t",queue="rx",time="1608786795151855424"}	33

STAT{METype="NT",ResourceID="t",queue="tx",time="1608786795151775007"}	1
STAT{METype="NT",ResourceID="t",queue="tx",time="1608786795151813374"}	2
STAT{METype="NT",ResourceID="t",queue="tx",time="1608786795151855424"}	3

And i've to perform operations on specific metrics( 2nd and 5th, how can we fetch that particular series for operation.

Note: Just to clarify, above time labels are added,to push the same metrics multiple times.

[Stuart Clark]

On 24/12/2020 05:53, akshay sharma wrote:

Thanks for the clarification,

What if,there is multiple metrics with same labels but different values

ex: 

STAT{METype="NT",ResourceID="t",queue="rx",time="1608786795151775007"}	31
STAT{METype="NT",ResourceID="t",queue="rx",time="1608786795151813374"}	32
STAT{METype="NT",ResourceID="t",queue="rx",time="1608786795151855424"}	33

STAT{METype="NT",ResourceID="t",queue="tx",time="1608786795151775007"}	1
STAT{METype="NT",ResourceID="t",queue="tx",time="1608786795151813374"}	2
STAT{METype="NT",ResourceID="t",queue="tx",time="1608786795151855424"}	3

And i've to perform operations on specific metrics( 2nd and 5th, how can we fetch that particular series for operation.

Note: Just to clarify, above time labels are added,to push the same metrics multiple times.

You can't have multiple entries for the same metric (same name and set of labels) during a single scrape.

What is the actual difference between the lines you listed above? If it is just time then you would never be outputting these in a single scrape. Each scrape will return the "current" value, which would change over time.

[akshay sharma]

5:04 PM (0 minutes ago)

I understood your point

Now i've one metric and with multiple values updating like every minute, if you range over you will see all the values.

byte{ID="1",instance="localhost:8999",job="prometheus",queue="tx"}

16 @1608808879.845 17 @1608808909.845 17 @1608808939.845 18 @1608808969.845

I'll perform action on Above values.

But I can see each value is pushing two times, that's what I don't want. 

Here is the code snippet,

Registering handler()

label:       value1 = 30 + i

                value2 = 15 + i
                matric1 :=  prometheus.NewGauge(prometheus.GaugeOpts{
                Name: "byte",
                Help: "test",
                ConstLabels: prometheus.Labels{"ID":"1", "queue": "rx"},
                })
                matric1.SetToCurrentTime()
                matric1.Set(float64(value))
               
                matric2 :=  prometheus.NewGauge(prometheus.GaugeOpts{
                Name: "byte1",
                Help: "test1",
                ConstLabels: prometheus.Labels{"ID":"1", "queue": "tx"},
                })
                matric2.SetToCurrentTime()
                matric2.Set(float64(value2))
                prometheus.MustRegister(matric1)
                prometheus.MustRegister(matric2)
                i = i + 1
        time.Sleep(1*time.Minute)
        prometheus.Unregister(matric1)
        prometheus.Unregister(matric2)

  gotolabel

[Stuart Clark]

On 24/12/2020 11:37, akshay sharma wrote:

5:04 PM (0 minutes ago)

I understood your point

Now i've one metric and with multiple values updating like every minute, if you range over you will see all the values.

byte{ID="1",instance="localhost:8999",job="prometheus",queue="tx"}

16 @1608808879.845 17 @1608808909.845 17 @1608808939.845 18 @1608808969.845

I'll perform action on Above values.

But I can see each value is pushing two times, that's what I don't want. 

Here is the code snippet,

I'm not quite sure I understand what you mean by "each value is pushing two times"?

Prometheus will scrape your application at whichever scraping frequency you set in the Prometheus configuration. Once data is being ingested you will be able to run PromQL queries over it to produce graphs and trigger alerts.

[akshay sharma]

Thank you for the clarification. 

I was doing some operation on instant vectors as well as on range vectors using a PROMQL query.

I have one query while performing operation on an instant vector.

For ex:

node_value{ID="1",instance="xx.xxx.xx.xx:x",job="prometheus",node="node1"}

32 @1608811039.845 33 @1608811099.845 34 @1608811159.845 35 @1608811219.845 36 @1608811279.845 37 @1608811339.845 38 @1608811399.845 39 @1608811459.845 40 @1608811519.845 41 @1608811579.845

Now, I want to do a sum/any operation of 1st and last value of the above metrics.

How can we do that?, as I could see nothing specified in DOC.

or more any specific way we can fetch values(single value) like 1st, 2nd or 3rd value from above metrics.

Thanks,

Akshay