discussion on the four golden signals

[ji...@comware.com.au]

Does anyone have any experience with implementing the 4 golden signals in stackdriver and GKE. We have deployed istio on our cluster, which should provide enough custom metrics to derive at least 3 of the 4 golden signals. 

[Patrick Eaton]

Hi Jima

How you considered using the Stackdriver metrics for Istio that are currently in beta?

By using the Istio's Stackdriver adapter (https://istio.io/docs/reference/config/policy-and-telemetry/adapters/stackdriver/), you will get access to a number of metrics (https://cloud.google.com/monitoring/api/metrics_other#other-istio.io) including request_count and request_latency. From that, you can get the most important golden signals

cheers-

patrick, stackdriver engineer

On Tue, Feb 12, 2019 at 5:34 AM <ji...@comware.com.au> wrote:

Does anyone have any experience with implementing the 4 golden signals in stackdriver and GKE. We have deployed istio on our cluster, which should provide enough custom metrics to derive at least 3 of the 4 golden signals. 

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/7c2a042d-581c-44e5-9d2c-1e6015a034b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mary Koes]

Hey Jima,

It's great to see customers focusing on the golden signals.  +Ruxanda Danetiu is leading an effort to make this even easier within Stackdriver...

Best,

Mary

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CAOj3vG%2BuudcNe0jNSjeS4%2BG8sttMMF9Eod4VGr1OdZ_BxpQK2w%40mail.gmail.com.

[ji...@comware.com.au]

Hey Patrick,

We are using istio in our GKE cluster. We currently have a requirement to use 1.10 otherwise we would defer to using managed istio. I suspect that we can get three out of the four directly from istio metrics. How do you measure/alert on saturation?

On Wednesday, February 13, 2019 at 12:46:34 AM UTC+11, Patrick Eaton wrote:

Hi Jima

How you considered using the Stackdriver metrics for Istio that are currently in beta?

By using the Istio's Stackdriver adapter (https://istio.io/docs/reference/config/policy-and-telemetry/adapters/stackdriver/), you will get access to a number of metrics (https://cloud.google.com/monitoring/api/metrics_other#other-istio.io) including request_count and request_latency. From that, you can get the most important golden signals

cheers-

patrick, stackdriver engineer

On Tue, Feb 12, 2019 at 5:34 AM <ji...@comware.com.au> wrote:

Does anyone have any experience with implementing the 4 golden signals in stackdriver and GKE. We have deployed istio on our cluster, which should provide enough custom metrics to derive at least 3 of the 4 golden signals. 

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[ji...@comware.com.au]

Het Mary,

We are currently leading an effort with a large organization to use GKE for all our projects. We have selected stackdriver for monitoring, alerting and incident management so any help or direction would be appreciated. We are in the middle of our first project and have CI/CD in place on our GKE cluster with istio. The next step is to identify and automate alerting policies, which will initially be based on the four golden signals and uptime checks. Subsequently, we will be looking at creating custom dashboards. The long term goal is to use this on a number of other processes.

On Wednesday, February 13, 2019 at 12:59:58 AM UTC+11, Mary Koes wrote:

Hey Jima,

It's great to see customers focusing on the golden signals.  +Ruxanda Danetiu is leading an effort to make this even easier within Stackdriver...

Best,

Mary

On Tue, Feb 12, 2019 at 8:46 AM 'Patrick Eaton' via Google Stackdriver Discussion Forum <google-stackdriver-discu...@googlegroups.com> wrote:

Hi Jima

How you considered using the Stackdriver metrics for Istio that are currently in beta?

By using the Istio's Stackdriver adapter (https://istio.io/docs/reference/config/policy-and-telemetry/adapters/stackdriver/), you will get access to a number of metrics (https://cloud.google.com/monitoring/api/metrics_other#other-istio.io) including request_count and request_latency. From that, you can get the most important golden signals

cheers-

patrick, stackdriver engineer

On Tue, Feb 12, 2019 at 5:34 AM <ji...@comware.com.au> wrote:

Does anyone have any experience with implementing the 4 golden signals in stackdriver and GKE. We have deployed istio on our cluster, which should provide enough custom metrics to derive at least 3 of the 4 golden signals. 

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
To post to this group, send email to google-stackdriver-discussion@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/7c2a042d-581c-44e5-9d2c-1e6015a034b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
To post to this group, send email to google-stackdriver-discussion@googlegroups.com.

[Patrick Eaton]

Jima,

I don't know of a clear, general, consensus definition of saturation for Istio. What does saturation mean for your application?

--pre

On Tue, Feb 12, 2019 at 3:54 PM <ji...@comware.com.au> wrote:

Hey Patrick,

We are using istio in our GKE cluster. We currently have a requirement to use 1.10 otherwise we would defer to using managed istio. I suspect that we can get three out of the four directly from istio metrics. How do you measure/alert on saturation?

On Wednesday, February 13, 2019 at 12:46:34 AM UTC+11, Patrick Eaton wrote:

Hi Jima

How you considered using the Stackdriver metrics for Istio that are currently in beta?

By using the Istio's Stackdriver adapter (https://istio.io/docs/reference/config/policy-and-telemetry/adapters/stackdriver/), you will get access to a number of metrics (https://cloud.google.com/monitoring/api/metrics_other#other-istio.io) including request_count and request_latency. From that, you can get the most important golden signals

cheers-

patrick, stackdriver engineer

On Tue, Feb 12, 2019 at 5:34 AM <ji...@comware.com.au> wrote:

Does anyone have any experience with implementing the 4 golden signals in stackdriver and GKE. We have deployed istio on our cluster, which should provide enough custom metrics to derive at least 3 of the 4 golden signals. 

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/7c2a042d-581c-44e5-9d2c-1e6015a034b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To post to this group, send email to google-stackdr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/0fd43061-ac0d-4619-a206-938c9d225faf%40googlegroups.com.

[Jim Alateras]

i guess resource saturation (cpu, memory etc). We do have resource limits on our containers/pod so they will scale in that instance. Are there any guidelines you can provide

[Steven Thurgood]

Hi,

regarding alerting policies, you should consider defining SLOs (see Site Reliability Workbook, Chapter 2) for your service, and using these as the basis for your alerting (Chapter 5). Depending upon the nature of your service, the error and latency signals you already have may be sufficient. Doing this helps move towards symptom-based alerting, and away from cause-based which is liable to result in false positives.

The traffic and saturation signals are useful for responding to SLO-based alerts.

Regarding saturation, I don't have any GKE or Istio-specific knowledge, but in general you need some measure of what you have, and some other measure of what you can achieve: whether this be a measure of your QPS vs your load-tested maximum QPS, or whether it be the number of instances in your instance group vs. your configured maximum number of instances.

On Wednesday, 13 February 2019 21:58:56 UTC, Jim Alateras wrote:

i guess resource saturation (cpu, memory etc). We do have resource limits on our containers/pod so they will scale in that instance. Are there any guidelines you can provide

> On 14 Feb 2019, at 8:28 am, Patrick Eaton <pre...@google.com> wrote:
>
> Jima,
> I don't know of a clear, general, consensus definition of saturation for Istio. What does saturation mean for your application?
> --pre
>
> On Tue, Feb 12, 2019 at 3:54 PM <ji...@comware.com.au> wrote:
> Hey Patrick,
>
> We are using istio in our GKE cluster. We currently have a requirement to use 1.10 otherwise we would defer to using managed istio. I suspect that we can get three out of the four directly from istio metrics. How do you measure/alert on saturation?
>
> On Wednesday, February 13, 2019 at 12:46:34 AM UTC+11, Patrick Eaton wrote:
> Hi Jima
>
> How you considered using the Stackdriver metrics for Istio that are currently in beta?
> By using the Istio's Stackdriver adapter (https://istio.io/docs/reference/config/policy-and-telemetry/adapters/stackdriver/), you will get access to a number of metrics (https://cloud.google.com/monitoring/api/metrics_other#other-istio.io) including request_count and request_latency. From that, you can get the most important golden signals
>
> cheers-
> patrick, stackdriver engineer
>
>
> On Tue, Feb 12, 2019 at 5:34 AM <ji...@comware.com.au> wrote:
> Does anyone have any experience with implementing the 4 golden signals in stackdriver and GKE. We have deployed istio on our cluster, which should provide enough custom metrics to derive at least 3 of the 4 golden signals.
>
>
>
>
> --
> © 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
>  

> Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

> ---
> You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

> To post to this group, send email to google-stackdr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/7c2a042d-581c-44e5-9d2c-1e6015a034b6%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> © 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
>  

> Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

> ---
> You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[Joe Stanton]

Is there any further support from Google on this? When not using Istio, the Golden Signals (especially % error rate) are very poorly supported in StackDriver! Would be great to see this become easier.