Stackdriver monitoring alerts (Google DNS + Cloudflare sites = Downtime Today)

[Mpp Loop]

Today I experienced an interesting issue. Several Stackdriver monitoring alerts notified me that several of my sites were down, one a VM on GCE and another on a separate non-GCP host.

After investigating with my web host and searching it appeared it was not a host-based issue but a problem at the DNS level. Based on what I was seeing reported on Twitter it's apparent that folks who were using Google's public DNS were having (continue to have?) problems seeing websites load in their browser that are on Cloudflare for DNS.

Then I realized, I also had StatusCake monitoring setup for the same websites and I never received any downtime errors from that service...only from Stackdriver. 

My assumption is that since there were reported issues for folks on Google public DNS trying to reach Cloudflare sites that the Stackdriver monitoring on GCP must be on the same Google DNS setup, thus reporting sites being down when it was only specific to users/systems on Google Public DNS .

At the moment things are working fine but there was a good amount of time today where this issue was taking place, between 1135 to about 1pm EST.

Simply wanted to post this here in case anyone else had experienced some site issues and were not aware of the issue between Google Public DNS and Cloudflare.

-Mike

[Nikki Oyekunbi]

Hello,

  I am from the Stackdriver Uptime Monitoring team. There was an (we believe unrelated) issue with Uptime check alerts in Stackdriver incorrectly firing. You are correct that the issue is now resolved. Thanks for posting.

Thanks,

        Nikki

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/7f787283-cb64-4007-88e4-8277fb833ad9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joshua Fox]

This approach  to bundling  logs by request in Flexible Environment is not going to be supported. 

So, we are trying  Google Cloud Logging which is supposed to support that.

1. As a proof of concept, we are simply calling LoggingExample ten times inside  the doGet() of the basic sample HelloServlet,  thus:  LoggingExample.main("write", "mylogname", "WARNING",  message ). But the  logs are not bundled by request. See screenshot.

2. We tried the Logback integration, using  logger.info("LOGBACK " + message). The  WEB-INF/logback.xml is attached. But  again, there is  no bundling by request.

How do we get logs bundled by request?

[Igor Peshansky]

I'll let someone else from the logging team chime in because there may be more needed, but as far as I know, Cloud Logging uses the HttpRequest field of the log entry to group, and that logging example doesn't set HttpRequest.

        Igor
-- sent from a mobile device, please excuse tyops and omissns

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CAD%3DB7cOcN_nrrQ%2BxsDc4JEJpZgiC3HByZV%2BONoUaZX6gpaegSA%40mail.gmail.com.

[Summit Tuladhar]

Grouping and correlation of log entries is described in http://go/stackdriver-logging-style#grouping-and-correlation, which documents that to bundle by request:

• The "parent" log entry must have a nonempty httpRequest.requestUrl field
  
• The "child" log entries must be in a different log stream than the parent, in the same project, and match the parent's resource.type and resource.labels fields exactly
  
• The child entries' timestamps must be within the range [parent.timestamp−parent.httpRequest.latency−60s,parent.timestamp]. (That is, the timestamp of the parent is assumed to be the end of the request, and any grouped logs are expected to have been emitted with earlier timestamps.)
  
• The trace field in all entries must match exactly.
  

A recent blogpost also describes the same: https://medium.com/google-cloud/combining-correlated-log-lines-in-google-stackdriver-dd23284aeb29

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CAN%3DUupfUodGvrvnKrJF6sYVJWwK9qg_pTn6FFOFd8pMPJ9NZtg%40mail.gmail.com.

[Marc Unangst]

+jbrydebell and +andreweames who are looking at making this feature more usable

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CAF%3DXVJAsgqMuwqqZsS9VCZzpxrWhSRqzMJUH7hX-Dh682rW0Xw%40mail.gmail.com.

[Chris Foresman]

This is not at all what is descrived in the documentation. It says that all entries with the same ID will be grouped. I implemented this and naturally it doesn't work, which makes sense given the complicated correlation rules you describe. However, I don't even think I _could_ implement this, since I'm pretty sure all the related log entries I want grouped are coming from the same instance and so the same log stream.

Maybe better documentation of the Trace ID and how it is generated and accessed per request would get me closer to getting the log entry grouping working, but I'd still be stymied by the requirement that entries from from different "streams" then the parent.

[Igor Peshansky]

There are many ways to achieve different log streams. On a GCE instance, it could be as simple as writing to another file. On GKE, this would mean a log from another container.
I would guess that the requirement for separate streams is imposed to prevent RPCs to self. Not sure why that would be needed, but I'll let others who are more familiar with tracing answer this one.

        Igor
-- sent from a mobile device, please excuse tyops and omissns

On Mon, Jul 2, 2018, 9:38 AM Chris Foresman <fore...@gmail.com> wrote:

This is not at all what is descrived in the documentation. It says that all entries with the same ID will be grouped. I implemented this and naturally it doesn't work, which makes sense given the complicated correlation rules you describe. However, I don't even think I _could_ implement this, since I'm pretty sure all the related log entries I want grouped are coming from the same instance and so the same log stream.

Maybe better documentation of the Trace ID and how it is generated and accessed per request would get me closer to getting the log entry grouping working, but I'd still be stymied by the requirement that entries from from different "streams" then the parent.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/97df6e5c-c9bd-4eee-a0c5-95098b3a23d0%40googlegroups.com.

[Chris Foresman]

The omg I don't understand is that this totally works in App Engine. Every log entry related to the same request is group together, even though all the log entries come from the same instance of the app.

[Chris Foresman]

Sorry; I should know better than to type on my phone. What I meant to say is that the _thing_ I don't understand is how this works in App Engine. It sounds like I could potentially group log entries from other services related to the same request (I think..., if I could figure out how to generate and distribute trace IDs), but if a request to a single service generates more than one log entry (almost every request), those _cannot_ be grouped by Stackdriver? I mean, it seems really strange this simply isn't possible.

[Joshua Fox]

Hello Jay and Andrew, (I just got back from vacation.)

How can we get logs bundled by request in Flex Env Java? 

Google Cloud Logging API  and Logback are said to do this but don't,  as shown in a   PoC which made logs appear in Log Viewer, but  unbundled.  

Summit Tuladhar referenced a  URL  accessible only to Googlers, and also  a Medium blogpost -- but this has a  do-it-yourself nonproductized approach, rather than the    a fully functional, tested solution that we need. (Also, the article is aimed at   GCE only, in Go, with Python code provided but said to be non-working),

This documentation discusses correlation of logs by request. I tried the low-level do-it-yourself  approach briefly outlined  there. (See  test code attached.) But besides the fact that this does not work, it made me realize that I don't have access to the info that the Flex Env team has, such as  the low-level end-to-end access to  HttpRequest, trace-ID, etc.)

 Google libraries  should do all  this automatically, given that the  information  is certainly available to  them. 

In fact,  Google does  succeed in bundling logs  in Standard Env, the   Flex Env beta (MVM), and in an "enhanced" library (that apparently will be  unsupported), so it is not clear why we do not have it in Flex Env.

If it is not going to be part of the libraries, then can we please get documentation and a code sample, particularly for the Logback approach?

--

 

JOSHUA FOX

Director, Software Architecture | Freightos

T (Israel): +972-545691165 | T (US):  +1-3123400953  

Smooth shipping.