GCE Network Jitter on Premium Tier network?

[Nate Weibley]

I'm wondering if I can get any insight on this; I run some monitoring of my internet connection to keep tabs on how healthy it is an to keep my previous ISP honest. I was having major packet-loss issues with my cable internet provider and needed a way to prove to them beyond a doubt that the packet loss was occurring on their side of the network.

As part of accomplishing this I set up smokeping instances on VMs in AWS and GCE which perform 20 1118 byte ICMP pings every 30 seconds to my home internet connection. I have since moved to Chicago and now have Webpass internet service (which was bought by Google Fiber in 2016), which is already light years better than what I had on cable.

However, in doing so I have noticed that the GCE route to my connection has a ton of jitter or "smoke" in the smokeping plots, where the AWS route is pretty darn clean. Both routes are consistently about 11ms RTT on average. 

You can see the GCE route data here: https://smokeping.nate-online.com/smokeping/smokeping.cgi?target=Cox.Home

You can see the AWS route data here: https://smokeping.aws.nate-online.com/smokeping/smokeping.cgi?target=Cox.Home 

I'm wondering what is causing this jitter in Google's route since I would assume they have better peering with an ISP they own. Both instances are running on similar hardware (GCE is on f1-micro in us-central1-b, AWS is on t3.micro in us-east-2a) and as I stated on GCE I'm using the Premium network tier.

I tried moving zones in the region but that didn't help, and us-central1 is the region with the shortest RTT to my home connection. Any ideas? 

-Nate

[Jason]

Hi,

From looking at both link that you provided, this is what I can see:

Chart in the last hour: 0% packet loss from AWS and GCE

Chart in the last 3 hours: 15% max packet loss on AWS and 24.5% max packet loss on GCE

Chart in the last 24 hours: 100% max packet loss on AWS and 8.75% max packet loss on GCE

Even though there was a 9.5 percent max packet loss between GCE and AWS in the last 3 hours, you can see that the routes were alot better in the last 24 hours as AWS hit a max of 100% percent packet loss, where GCE was only 8.5 %. Keep in mind that this is also monitoring your ISP connection, so I cannot provide an exact answer to why there is some packet loss between the ISP and the GCE instance.

[Nate Weibley]

Hi Jason,

The segments of extended packet loss are do to me moving from Florida to Illinois :) What I really care about is the data after Tuesday July 16th; that represents when the new connection was active and I migrated my cloud instances to nearby regions. There are some brief blips of packet loss while I was sorting out an IPv6 prefix delegation issue with the new ISP, but other than that neither route has dropped a single ICMP packet since I turned on my service.

[Fady (Google Cloud Platform)]

Thank you for the information. We, Google Cloud Platform Support, do not have visibility on how the ISP routes traffic to Google Cloud Platform or vice versa. I assume if you are using a public IP on the GCE instance, traffic will be leaving GCE and routed through the Internet. As to explain, the ISP mentions Google Fiber, and it may not be necessarily routing traffic internally through Google Cloud platform. 

To better understand this behavior, you may try the MTR tool on both the server (instance) and the client (local machine) as to compare routes and jitter for each hop, and for both Cloud platforms. The following additional articles and tools may also help with this approach [1] [2] [3] [4] [5].  

Furthermore, you may reach the ISP and Google Fiber support, explaining your findings. Once reaching them, please post on this thread as we, and the community are also curious.