Monitoring Dataflow custom metrics using Stackdriver

[egb...@kramphub.com]

Hi,

I'm working on a Dataflow job that defines a number of metrics, such as the number of malformed documents received. All these metrics are of Beam's Counter type. These metrics are visible in the Dataflow UI, and they also end up in Stackdriver.

In Stackdriver, I'd like to create some alerts that warn me when the rate of malformed documents rises. However, they are automatically added there as custom metrics of kind GAUGE. This does not make sense as they are counters, not gauges. The Gauge metric of Beam is not supported by Dataflow. These counters should be mapped to a metric of kind CUMULATIVE. I tried to delete the automatically created metric descriptors and add new ones with the same name, type and value_type where the kind is set to CUMULATIVE.

In Stackdriver Monitoring then all of a sudden, meaningful graphs appear that show the rate of change of this counter, which is something that I could set an alert on. However, no new data comes in anymore, even though the metrics are updated in the Dataflow UI.

Once I switch back to GAUGE, new data starts coming in again. I found a reference online that suggests that for CUMULATIVE metrics a start_time should be added by the client, I'm guessing Dataflow doesn't do that? However, how come that data received previously when it was still a GAUGE metric is reinterpreted as being a CUMULATIVE value properly, while new data is ignored?

Apparently, Dataflow forces you to use Counter metrics while Stackdriver forces you to "cast" these Counter metrics to Gauge metrics, without doing any conversion.

Is there any way to get meaningful alerts on Counters from Dataflow in Stackdriver?

Thanks!

Regards,

Egbert

[Mary Koes]

+Keith Jordy +Marie Cosgrove-Davies +Sergei Sokolenko 

Adding a few folks who work on our Dataflow integration with Stackdriver...

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/dbc9b158-a38f-465b-b89d-c5cbfd12a1f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Egbert van der Wal]

Hi,

Is there anything to accomplish what I'm trying to do? Or should I resort to using log-based metrics instead for now?

Thanks!

Regards,

Egbert

[Brian Hatfield]

Hi there,

We've recently stumbled across this thread troubleshooting the same kind of issue - we want the 'rate' aligner to appear in Stackdriver, however, it's not available because the dataflow custom metrics (which are type Counter in dataflow) are being mapped to type Gauge in Stackdriver, which is not what we intended.

Is there any update here? How can we have our Counters be Counters while using the integrated Beam/Dataflow Metrics?

Thanks!

Brian

On Friday, April 26, 2019 at 8:33:46 AM UTC-4, Mary Koes wrote:

+Keith Jordy +Marie Cosgrove-Davies +Sergei Sokolenko 

Adding a few folks who work on our Dataflow integration with Stackdriver...

On Fri, Apr 26, 2019 at 4:45 AM <egb...@kramphub.com> wrote:

Hi,

I'm working on a Dataflow job that defines a number of metrics, such as the number of malformed documents received. All these metrics are of Beam's Counter type. These metrics are visible in the Dataflow UI, and they also end up in Stackdriver.

In Stackdriver, I'd like to create some alerts that warn me when the rate of malformed documents rises. However, they are automatically added there as custom metrics of kind GAUGE. This does not make sense as they are counters, not gauges. The Gauge metric of Beam is not supported by Dataflow. These counters should be mapped to a metric of kind CUMULATIVE. I tried to delete the automatically created metric descriptors and add new ones with the same name, type and value_type where the kind is set to CUMULATIVE.

In Stackdriver Monitoring then all of a sudden, meaningful graphs appear that show the rate of change of this counter, which is something that I could set an alert on. However, no new data comes in anymore, even though the metrics are updated in the Dataflow UI.

Once I switch back to GAUGE, new data starts coming in again. I found a reference online that suggests that for CUMULATIVE metrics a start_time should be added by the client, I'm guessing Dataflow doesn't do that? However, how come that data received previously when it was still a GAUGE metric is reinterpreted as being a CUMULATIVE value properly, while new data is ignored?

Apparently, Dataflow forces you to use Counter metrics while Stackdriver forces you to "cast" these Counter metrics to Gauge metrics, without doing any conversion.

Is there any way to get meaningful alerts on Counters from Dataflow in Stackdriver?

Thanks!

Regards,

Egbert

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[Rory Petty]

+Sergei Sokolenko 

On Thu, May 30, 2019 at 12:15 PM Brian Hatfield <br...@digits.com> wrote:

Hi there,

We've recently stumbled across this thread troubleshooting the same kind of issue - we want the 'rate' aligner to appear in Stackdriver, however, it's not available because the dataflow custom metrics (which are type Counter in dataflow) are being mapped to type Gauge in Stackdriver, which is not what we intended.

Is there any update here? How can we have our Counters be Counters while using the integrated Beam/Dataflow Metrics?

Thanks!

Brian

On Friday, April 26, 2019 at 8:33:46 AM UTC-4, Mary Koes wrote:

+Keith Jordy +Marie Cosgrove-Davies +Sergei Sokolenko 

Adding a few folks who work on our Dataflow integration with Stackdriver...

On Fri, Apr 26, 2019 at 4:45 AM <egb...@kramphub.com> wrote:

Hi,

I'm working on a Dataflow job that defines a number of metrics, such as the number of malformed documents received. All these metrics are of Beam's Counter type. These metrics are visible in the Dataflow UI, and they also end up in Stackdriver.

In Stackdriver, I'd like to create some alerts that warn me when the rate of malformed documents rises. However, they are automatically added there as custom metrics of kind GAUGE. This does not make sense as they are counters, not gauges. The Gauge metric of Beam is not supported by Dataflow. These counters should be mapped to a metric of kind CUMULATIVE. I tried to delete the automatically created metric descriptors and add new ones with the same name, type and value_type where the kind is set to CUMULATIVE.

In Stackdriver Monitoring then all of a sudden, meaningful graphs appear that show the rate of change of this counter, which is something that I could set an alert on. However, no new data comes in anymore, even though the metrics are updated in the Dataflow UI.

Once I switch back to GAUGE, new data starts coming in again. I found a reference online that suggests that for CUMULATIVE metrics a start_time should be added by the client, I'm guessing Dataflow doesn't do that? However, how come that data received previously when it was still a GAUGE metric is reinterpreted as being a CUMULATIVE value properly, while new data is ignored?

Apparently, Dataflow forces you to use Counter metrics while Stackdriver forces you to "cast" these Counter metrics to Gauge metrics, without doing any conversion.

Is there any way to get meaningful alerts on Counters from Dataflow in Stackdriver?

Thanks!

Regards,

Egbert

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/dbc9b158-a38f-465b-b89d-c5cbfd12a1f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/037085fe-8ca1-42a1-87fb-f304f7101a98%40googlegroups.com.