Yep, that helps. Thank you. I also bet that if you ran a test connecting over a 7 day period you would see lots of times where that ~200-400ms becomes ~1,000 to ~2,000+ms. When I run this same test machine-2-machine using the internal IP addresses on GCE, the time is close enough to zero that I don't care. The net difference in our app is a single request (a client can make up to 3 for a complex ERP screen) takes ~400-900ms (open tls connection, all the back and forth) whereas the same thing tested GCE to GCE via intenal ip takes 10ms (with high consistency.). Multiply those numbers by 3x for our biggest customers (most complex apps) and you are looking at ~30ms vs ~2,000mm wait per click (which gets old.) They do tolerate this because they are coming from really nasty systems, but we want to provide the best possible cloud experience.
I really hope someone at Google is monitoring this thread. If we could allocate our GAE std setup to the default GCE network for our project and avoid all this socket overhead it would be a very, very, very good thing.