Frontend instances latency issues

[Andrius]

Hi,

I am having a problem to find out why same requests to different front
ends take much longer to respond. On average 1 request will take
175ms but some times it can take 800ms - 5sec.

A test with ajax request executed every 1 second. 1-3 out of 10 ajax
requests request will take 800ms or even longer, but others will do
the response on average 175ms

QPS* Latency* Requests Errors Age Memory Availability
0.033 43.5 ms 294 0 11:35:20 42.3 MBytes Resident
0.000 0.0 ms 83 0 11:35:20 40.9 MBytes Resident
0.017 2319.0 ms 125 0 11:35:20 41.7 MBytes Resident
0.483 31.0 ms 2835 0 2:22:29 100.6 MBytes Dynamic
0.550 25.4 ms 2530 0 2:05:09 93.4 MBytes Dynamic
0.617 37.0 ms 2649 0 2:20:25 92.7 MBytes Dynamic

I have added profiler and found that execution/prcessing/cpu time is
exactly the same within different front ends but response time varies
a lot!
As you can see one of the frontends has latency on average 2319.0 ms!
Why is that? Are there any issues with internal GAE network latency,
maybe within datacenters?