DNS Rate Limiting ICMP (8.8.8.8 and 8.8.4.4)

[bamm...@gmail.com]

Hi All,

We are a larger company with several DIA's.  We noticed over the past week that saw ping to these two DNS servers were losing pings out one provider, but not the other.  Calling the provider they indicated that Google is starting to deploy a rate-limit on ICMP, possibly for DDOS control.  Is this true and will the larger community notice this with ping?

Thanks, Brian

[Brian Mcmichael]

Same situation. We had to route those blocks out our ATT link as a temporary solution. 

Thanks

Brian McMichael

Network Architect

On Sep 9, 2016, at 3:54 PM, 1pint.g...@gmail.com wrote:

We are a FTTH company with 2 Internet feeds from level 3 and Cogent. We are seeing the same issues as you but I don't believe this is due to rate-limiting. We are only experiencing packet loss across the Level 3 connection and not the Cogent. We also have customers that have been reporting slow response to anything google such as google search, youtube, etc. Also we have a customer using Google VPN for work and they are unable to hold up a connection.This leads me to believe they have a problem between Level 3 and Google. We have a ticket open with Level 3 but cannot find anywhere to open a ticket with Google.

[1pint.g...@gmail.com]

We are a FTTH company with 2 Internet feeds from level 3 and Cogent. We are seeing the same issues as you but I don't believe this is due to rate-limiting. We are only experiencing packet loss across the Level 3 connection and not the Cogent. We also have customers that have been reporting slow response to anything google such as google search, youtube, etc. Also we have a customer using Google VPN for work and they are unable to hold up a connection.This leads me to believe they have a problem between Level 3 and Google. We have a ticket open with Level 3 but cannot find anywhere to open a ticket with Google.

On Friday, September 9, 2016 at 1:37:16 PM UTC-5, bamm...@gmail.com wrote:

[phillip....@level3.com]

This is what I provided to Google and as you see at the time I was also seeing the issue through Cogent's lookingglass

Please provide any additional information below.

Cogent
Test Router Location Hostname / IP Address

8.8.8.8
Go!
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=26.2 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=26.2 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=26.1 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 3 received, 40% packet loss, time 5005ms  <---- 40% packet loss
rtt min/avg/max/mdev = 26.152/26.196/26.231/0.189 ms

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1  multi-use.cogentco.com (66.250.250.81)  1.193 ms  1.231 ms
2  te0-6-0-16.ccr21.dfw01.atlas.cogentco.com (154.54.44.193)  0.795 ms  0.809 ms
3  be2764.ccr41.dfw03.atlas.cogentco.com (154.54.47.214)  0.695 ms  0.799 ms
4  tata.dfw03.atlas.cogentco.com (154.54.12.106)  0.590 ms  0.560 ms
5  72.14.219.124 (72.14.219.124)  26.321 ms 209.85.172.106 (209.85.172.106)  26.254 ms
6  108.170.240.65 (108.170.240.65)  26.099 ms 108.170.240.193 (108.170.240.193)  26.153 ms
7  72.14.238.45 (72.14.238.45)  26.139 ms 72.14.234.151 (72.14.234.151)  26.109 ms
8  google-public-dns-a.google.com (8.8.8.8)  26.218 ms  26.083 ms


Level 3
ping 8.8.8.8 count 50
PING 8.8.8.8 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=3 ttl=63 time=0.360ms.
64 bytes from 8.8.8.8: icmp_seq=4 ttl=63 time=0.360ms.
64 bytes from 8.8.8.8: icmp_seq=6 ttl=63 time=0.356ms.
Request timed out. icmp_seq=1.
64 bytes from 8.8.8.8: icmp_seq=7 ttl=63 time=0.348ms.
Request timed out. icmp_seq=2.
64 bytes from 8.8.8.8: icmp_seq=8 ttl=63 time=0.364ms.
64 bytes from 8.8.8.8: icmp_seq=9 ttl=63 time=0.340ms.
Request timed out. icmp_seq=5.
64 bytes from 8.8.8.8: icmp_seq=11 ttl=63 time=0.348ms.
64 bytes from 8.8.8.8: icmp_seq=12 ttl=63 time=0.340ms.
64 bytes from 8.8.8.8: icmp_seq=15 ttl=63 time=0.360ms.
Request timed out. icmp_seq=10.
64 bytes from 8.8.8.8: icmp_seq=17 ttl=63 time=0.348ms.
64 bytes from 8.8.8.8: icmp_seq=18 ttl=63 time=0.352ms.
Request timed out. icmp_seq=13.
Request timed out. icmp_seq=14.
64 bytes from 8.8.8.8: icmp_seq=20 ttl=63 time=0.352ms.
Request timed out. icmp_seq=16.
64 bytes from 8.8.8.8: icmp_seq=24 ttl=63 time=0.376ms.
Request timed out. icmp_seq=19.
64 bytes from 8.8.8.8: icmp_seq=26 ttl=63 time=0.348ms.
Request timed out. icmp_seq=21.
64 bytes from 8.8.8.8: icmp_seq=27 ttl=63 time=0.356ms.
Request timed out. icmp_seq=22.
Request timed out. icmp_seq=23.
Request timed out. icmp_seq=25.
64 bytes from 8.8.8.8: icmp_seq=32 ttl=63 time=0.344ms.
64 bytes from 8.8.8.8: icmp_seq=33 ttl=63 time=0.360ms.
Request timed out. icmp_seq=28.
64 bytes from 8.8.8.8: icmp_seq=34 ttl=63 time=0.352ms.
Request timed out. icmp_seq=29.
64 bytes from 8.8.8.8: icmp_seq=35 ttl=63 time=0.352ms.
Request timed out. icmp_seq=30.
Request timed out. icmp_seq=31.
64 bytes from 8.8.8.8: icmp_seq=40 ttl=63 time=0.356ms.
64 bytes from 8.8.8.8: icmp_seq=41 ttl=63 time=0.348ms.
Request timed out. icmp_seq=36.
Request timed out. icmp_seq=37.
Request timed out. icmp_seq=38.
^C
ping aborted by user

---- 8.8.8.8 PING Statistics ----
43 packets transmitted, 21 packets received, 51.16% packet loss
round-trip min = 0.340ms, avg = 0.353ms, max = 0.376ms, stddev = 0.000ms

traceroute 8.8.8.8
traceroute to 8.8.8.8, 30 hops max, 40 byte packets
  1  0.0.0.0  * * *
  2  0.0.0.0  * * *
  3  Google-level3-100G.Dallas1.Level3.net (4.68.72.138)    4.82 ms  4.81 ms  4.86 ms
  4  108.170.240.129 (108.170.240.129)    5.01 ms
  4  108.170.240.1 (108.170.240.1)    4.96 ms  5.00 ms
  5  72.14.234.107 (72.14.234.107)    5.06 ms
  5  72.14.234.149 (72.14.234.149)    5.07 ms
  5  64.233.175.101 (64.233.175.101)    5.28 ms
  6  google-public-dns-a.google.com (8.8.8.8)    4.96 ms  4.94 ms  4.98 ms

[Alex Dupuy]

TL;DR: At the risk of repeating myself: Google Public DNS is a Domain Name System service, not an ICMP network testing service.

If you want to measure the quality of your DNS service from Google Public DNS, you should use a dnsping tool (https://github.com/farrokhi/dnsdiag [Python] or https://sourceforge.net/projects/dnsping/ [C#]) to send real DNS queries and check for responses. Note that while traceroute -U sends UDP/53 packets, they are not DNS queries, and traceroute -U is not a substitute for dnsping.

If dnsping shows significant levels of unanswered queries (and especially if ping and traceroute do not show any drops), you should check whether your IP address is generating more than 100 queries per second (the default per-IP address QPS limit for Google Public DNS). If you are legitimately generating more than 100 QPS and need to increase your QPS limit, you can request an increase through the Google Public DNS issue tracker.

While Google does not block ICMP or random UDP to 8.8.8.8 or other Google Public DNS IP addresses, there are rate limits on ICMP error replies from Google networking equipment, and ICMP error replies may be de-prioritized within Google networks and could be more likely to be dropped.

There are sometimes network anomalies (I hesitate to call them denial of service attacks, since ICMP is a laughable DoS vector) that send large volumes of ICMP queries to Google IP addresses, exceeding the rather high rate limiting thresholds, and suppressing some ICMP replies. This can cause ICMP-based network monitoring to report "dropped" packets, even when no packets are being dropped (Google is merely choosing not to respond to all ICMP echo requests). For much of the past two weeks ICMP traffic to Google in the DFW area has been affected by such an anomaly.

For this reason, occasional ping and traceroute drops are not evidence by themselves of a problem with your connectivity to Google Public DNS, and do not reflect the quality of service that your DNS queries are receiving.  If there is a real networking problem (high latencies or large numbers of drops) traceroute may be useful to diagnose and understand the source or nature of the problem, but evidence of DNS failures is needed to demonstrate the existence of a problem.

Furthermore, if you are having a networking (not DNS-specific) issue with your connection to Google, any DNS engineers you may reach through this forum or the issue tracker are not able to do anything about it (we may forward reports to networking teams in the case of complete connectivity loss causing SERVFAIL for domains, but that is not guaranteed). You should contact the Google networking teams through the ISP portal at isp.google.com or by escalating through your upstream ISP if you do not have a peering or GGC relationship with Google yourself.

[mksh...@gmail.com]

Hello ,


How Can i check whether  IP address is generating more than 100 queries per second (the default per-IP address QPS limit for Google Public DNS) ?

regards
Khalid

[Alex Dupuy]

Khalid wrote:

How Can i check whether  IP address is generating more than 100 queries per second (the default per-IP address QPS limit for Google Public DNS) ?

If you aren't running an ISP or corporate network, it would be unusual to generate more than 100 queries per second, but if you run a dnsping (query google.com or something that will already be cached, for best results) and you aren't seeing any drops, then you aren't exceeding the 100QPS limit.

If dnsping and ping both show drops, there may be network issues.

If dnsping shows drops, and ping doesn't, you (or others spoofing your address) are probably hitting the 100QPS limit.

If you have logs on your firewall or router that you can search you can tell whether it is you or (very unlikely) spoofers.

On a small network you could use a network traffic monitor to check.  If you run your own DNS resolver, you can use its logs too.

If you are running an ISP or corporate network with static IP addresses, and you are seeing dnsping drops, you can file a request on the Google Public DNS issue tracker to get a QPS increase.