Modify the Y-axis in the metrics explorer chart monitoring

[Rakesh Sagitra]

Hello guys,
                   I am new here.  I was happy to discover this forum! :)

I'm struggling in modifying the y-axis of the metrics explorer chart. Please have a look at the attached screenshot. As you can see, 1221 is the response time, and data on Y-axis is displaying as per the latency.

I am trying to measure responseTime from an endpoint and is using a logger that provides that data point in the JSON payload. And from the graph shows, the axis is strangely distributed since I probably want to have the y-axis to show a range of the response time possibilities. So that he can notice any spikes and create the alert.

In short, I want to achieve the following:

• Y-axis should be display based on the API response time. As 1221 is displayed in the screenshot.

[Summit Tuladhar]

Hi Rakesh,

It looks like you are creating a logs-based metric and extracting a value out of logs and placing it as a label. Each label value creates a unique timeseries (separate line in the chart). It is not the value of the metric itself that is shown in the y-axis. 

To capture values from the logs and show it in the y-axis, you will need to use distribution logs-based metrics. Due to the unlimited number of log entries that you can ingest in a given period, you can only capture the statistics of the values as a distribution metric. See: https://cloud.google.com/logging/docs/logs-based-metrics/distribution-metrics

Regards,

Summit

--
© 2020 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/2e89aaed-7e59-47b0-b9c2-bfc25cee1fbfn%40googlegroups.com.

[Rakesh Sagitra]

Thank's for your response. I have tried to create the distribution metric also but didn't able to get the response time on the y-axis.I am getting different time series data rather than a response time

[Summit Tuladhar]

Can you attach a screenshot? 

Also, make sure you are not adding the latency as a metric label, as that will create a lot of timeseries.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/338742ea-3c67-4e1a-854b-89658e4cc0dcn%40googlegroups.com.

[Rakesh Sagitra]

PFA for the screenshot for metric label and explorer

[Summit Tuladhar]

It looks like you are adding a metric label called Response_time and adding the jsonPayload.responseTime. This creates a new timeseries for each responseTime value, which is not what you want. You just need to use the extract value to capture as a distribution without using any labels.

Since you cannot delete a label from an existing metric, please create a new distribution metric without any labels. Make sure you delete this bad metric as it will cause cardinality issues.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/216fea1c-b636-453a-9f9b-e571e0ad9be1n%40googlegroups.com.

[Rakesh Sagitra]

I have removed the label and created new metrics. But again I am getting the different time series rather than response time on the y-axis. I have attached the screenshot below.Please have a look over it

[Summit Tuladhar]

The chart is showing the 99th percentile of the distribution. A distribution metric buckets the captured values as per your bucketing configuration. It does not capture each unique value as a separate point because of the large number of unique values that can be present in logs and it is unbounded. 

To have points in regular intervals, the distribution metric aggregates all points in the interval and buckets the values to make it bounded. The percentiles are calculated based on the values in the buckets. This is useful for capturing a large number of values and measuring statistics about it instead of capturing each unique value as a data point. You can also visualize the chart as a heatmap.

Does this work for your use case of measuring response times?

There's also a feature request open to support gauge logs-based metrics to capture individual points, but that will have limitations on sampling intervals: https://issuetracker.google.com/issues/136239018

+Brian Hurley (FYI)

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/dbd0ddd6-b25e-4560-9613-95f6cc87aa5an%40googlegroups.com.

[Rakesh Sagitra]

Can you please guide me on exactly how can I follow this approach guided by you? 

I want to have a graph of responsTime over timeseries data on a certain endpoint. This endpoint has the data in its logs. But how do I make a way to visualize this. Does not have to be a graph, can be a bar chart, or something that allows us to see the spikes easily.

[Summit Tuladhar]

Logs-based metrics can only capture a distribution of the values from logs. You can visualize the distribution using a Heatmap chart type. If there has been an increase in response time, for example, you can visualize that in the heat map. You can also look at the various percentiles of the response times.

What you cannot do is capture individual values from logs as individual points with exact y-axis values. Metrics have ingestion limits on the number of points that can be ingested per timeseries per minute. Logs have no such limit. There could be thousands of data points in one second. That's why a distribution metric is required to measure the average, or certain percentiles over those values. Does that make sense?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/c6cc6c4b-98f6-4d61-885d-663e65a85fe5n%40googlegroups.com.