Unexplained traffic loss

[Nathan Skone]

We have over 300 machines distributed across the US calling an API once a minute, which means we should always have a base level of traffic of at least 5 or 6 requests per second.  However, today we are seeing pits in our requests per second graph that show the traffic dropping to zero, then spiking directly afterward.  We have not deployed any new code recently, or otherwise touched our application in any way.

This traffic pattern has doubled our costs compared to a normal day. A screenshot is attached.

Thanks,

Nathan Skone

[Brandon Wirtz]

If you had network issues between locations, your traffic could be less evenly distributed, and would as a result make your app “peakier”.

Set your max pending latency higher and your cost won’t fluctuate when network latency changes the instantaneous request rate.

 

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/NBgU5Omj9hQJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Nathan Skone]

Brandon,

I do not believe this is a network issue, since each of the 300+ clients connects from different networks across the US, with a few overseas. These peaks seem to be caused by the Google App Engine itself.

Thanks,

Nathan

On Thursday, May 17, 2012 3:34:58 PM UTC-7, Brandon Wirtz wrote:

If you had network issues between locations, your traffic could be less evenly distributed, and would as a result make your app “peakier”.

Set your max pending latency higher and your cost won’t fluctuate when network latency changes the instantaneous request rate.

 

 

From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Nathan Skone
Sent: Thursday, May 17, 2012 2:52 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Unexplained traffic loss

 

We have over 300 machines distributed across the US calling an API once a minute, which means we should always have a base level of traffic of at least 5 or 6 requests per second.  However, today we are seeing pits in our requests per second graph that show the traffic dropping to zero, then spiking directly afterward.  We have not deployed any new code recently, or otherwise touched our application in any way.

 

This traffic pattern has doubled our costs compared to a normal day. A screenshot is attached.

 

Thanks,

Nathan Skone

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/NBgU5Omj9hQJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Jesse]

We are seeing the same thing.

[Nathan Skone]

Another note about this issue is that the admin graphs listed zero instances during these times, even though I had the minimum idle slider set to 4. This means that the Google App Engine incorrectly shut down instances that I specified should be running.

Thanks,

Nathan Skone

[Naresh Talluri]

Even i can see this behavior which i never seen before.

my app-id : beta-sb.

Thanks,
Naresh T

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/1jKeQMTTMmUJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Dylan]

We are having a similar issue.
Requests timing out or taking more than 10 seconds to return a 200kb
json document from the cache.
Our instance graph also looks the same for the past 24hrs, sharp
spikes in instances.

Anyone know how to get in touch with Google?

[Robert Schuppenies]

Have you filed a production issue?:

http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

[Robert Kluin]

Did anyone file an issue on this? I have also seen this behavior
before and would like to hear about the cause / resolution.



Robert