QUIC traffic killing my local network

[Daniel Raeder]

This happens almost daily. QUIC packets sized 1392 Bytes sent less than a milisecond apart and results in what is effectively a DOS. When I initially discovered the QUIC protocol in wireshark captures, I disabled pre-fetching in chrome on every device I have it. That solved the issue for a time. Then, I got notifications in chrome and android to try out "Data Saver (Beta)". The issue immediately came back. After removing "Data Saver (Beta)", with pre-fetching still disabled everywhere, the problem with QUIC traffic killing my local network is not going away. The only workarounds I have are:

1. Reboot my cable modem

2. Identify the chattiest Google IP and block it via hosts file on the talkative PC

3. Stop using Google services and Chrome altogether.

I hope someone here has better solutions than these.

[Robbie Shade]

Hi Daniel,

Sorry to hear that QUIC might be causing problems for you - we'd love to get some more information and try and help fix this. 

Can you expand on what you mean by "QUIC traffic killing my local network"? QUIC is a new protocol that Chrome uses to talk to Google servers - it simply replaces what used to be HTTP/TLS over TCP, so I would expect the number of packets flowing across your network to be roughly similar.

QUIC certainly shouldn't be resulting in degradation of your network connection - can you give some more detail about exactly what the problem is that you're seeing?

Best,

Robbie

PS. You can try turning QUIC off completely in Chrome, by going to chrome://flags and searching for QUIC. Disabling QUIC like this shouldn't be necessary (and we'd like to know if it is!), but this is the best way to disable it rather than blacklisting individual IPs or turning off Data Saver.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.
To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

[Ryan Hamilton]

Hi Daniel,

I'm sorry to hear about QUIC causing problems for you! Chrome will, by-default, use QUIC to communicate with Google. You can go to chrome://flags, search for Experimental QUIC protocol and set it to "Disabled" if this is causing problems for you. Or, as you say, you could block outbound UDP packets to port 443.

That being said, we (QUIC team) really don't want to be killing anyone's network. QUIC packets should be sent at roughly the same size/rate as TCP packets. If that is not happening for you, something is wrong and we'd definitely like to fix it! Can you elaborate on the problem that QUIC is causing? Which cable modem/router/ISP do you have? Can you collect a net-internals trace when this is happening:

https://dev.chromium.org/for-testers/providing-network-details

CHeers,

Ryan

--

[Daniel Raeder]

ISP: Time Warner

Cable Modem: uBee DVW3201B Docsis 3.0 

I attached a brief net-internals capture, but the issue is not presently occurring. 

Here is a graph of my traffic hierarchy at the time of an issue I had today. I would like to post the wireshark capture itself, but I don't want to put that in the public domain for security reasons (I hope the net-internals capture I attached doesn't reveal security sensitive data?).

Here is an IO graph showing the spikes in traffic activity at the time of the latest instance today just before rebooting my cable modem:

Here are some pings to google.com while the issue was happening:

>ping google.com






Pinging google.com [173.194.46.68] with 32 bytes of data:

Reply from 173.194.46.68: bytes=32 time=987ms TTL=54

Reply from 173.194.46.68: bytes=32 time=995ms TTL=54




Ping statistics for 173.194.46.68:

    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 987ms, Maximum = 995ms, Average = 991ms

Control-C

^C

While the modem was being rebooted:

>ping google.com






Pinging google.com [173.194.46.68] with 32 bytes of data:

Request timed out.

Request timed out.

Request timed out.

Request timed out.




Ping statistics for 173.194.46.68:

    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

After the reboot:

>ping google.com






Pinging google.com [173.194.46.64] with 32 bytes of data:

Reply from 173.194.46.64: bytes=32 time=11ms TTL=54

Reply from 173.194.46.64: bytes=32 time=11ms TTL=54

Reply from 173.194.46.64: bytes=32 time=11ms TTL=54

Reply from 173.194.46.64: bytes=32 time=10ms TTL=54




Ping statistics for 173.194.46.64:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 10ms, Maximum = 11ms, Average = 10ms

[Daniel Raeder]

I think this graph is interesting from a wireshark capture while the issue is not occurring. It actually shows more QUIC traffic, and less SSL traffic. Does QUIC and SSL work together? 

[Ryan Hamilton]

On Fri, Oct 30, 2015 at 11:50 AM, Daniel Raeder <dra...@gmail.com> wrote:

I think this graph is interesting from a wireshark capture while the issue is not occurring. It actually shows more QUIC traffic, and less SSL traffic. Does QUIC and SSL work together? 

​QUIC is effectively ​TLS over UDP. So when QUIC is being used, you would expect http:// requests to Google to be sent over QUIC instead of over TLS. 

It would be great if you could collect a net-internals trace when this happens again. It's a bit hard to see what might be going on from the high level metrics. Were you forced to reboot your router, or did it basically crash when this was happening? It sounds like you're describing a situation where Chrome suddenly sends a flood of QUIC traffic. But "normally" Chrome is able to speak QUIC just fine. Is that about right?

Cheers,

Ryan

[Daniel Raeder]

Yes sir, I think your assessment id correct. I will post net-internals next time it occurs ..

[Ryan Hamilton]

Thanks!

[Daniel Raeder]

Oh, to answer your other question, it really just bogs things down. Neither my modem nor router(s) crash in any way. At first, I was rebooting both the router and modem, but I found rebooting the modem only was effective to stop the problem.

[Ryan Hamilton]

Interesting. I'm extremely curious to see what your net-internals log will say. Were you doing anything like downloading a large file from Google or watching a YouTube video when this started?

[Chris Bentzel]

Also: Do you have issues with Chrome bogging your network when QUIC is not used?

You could experiment with this by going to about:flags in Chrome, and selecting "Disabled" under the "Experimental QUIC protocol" option.

This would help us understand if it is a QUIC specific issue, or something to do with how Chrome schedules resource fetches.

[Daniel Raeder]

I usually don't know that it is happening until I'm trying to access a customer remotely, or make call using Microsoft Lync. I will notice it because I see degraded network performance for those two activities. So, I can't tell what prompts the behavior initially, since neither of these actions have anything to do with Chrome. I'll then pop open wireshark, monitor for a minute or two and see excessive quic traffic in the results. Once, without any modem reboot, I simply edited the hosts file on my PC and sent the most talkative Google IP to a null host and the issue cleared. Because of this, combined with having previously disabled pre-fetching and seeing the issue disappear for quite some time, I correlate google traffic to the issue. I may be wrong in my findings.

Also, because it's intermittent (sometimes things are working fine for days), testing with and without QUIC enabled isn't going to be a quic process.. no pun intended :).  I'll leave it on for a while though and try to catch it in the act with net-internals.

[Daniel Raeder]

Another thing that's interesting to me is the QUIC traffic seems excessive with large packet sizes in BOTH directions when this happens. Upload and download.. 

[ianG]

On 30/10/2015 18:50 pm, Daniel Raeder wrote:
> I think this graph is interesting from a wireshark capture while the

> issue is not occurring. It actually shows /more/ QUIC traffic, and
> /less/ SSL traffic. Does QUIC and SSL work together?


As a lo-pri idle-time FAQ, something's been bothering me: How does QUIC
interact, or expect to interact, with TCP congestion control?



iang

[Ryan Hamilton]

​QUIC uses the TCP cubic congestion control algorithm, so it is expected to interact as another TCP flow would.

Cheers,

Ryan

[Fedor Kouranov]

Interesting indeed!

To step back a little: QUIC is HTTP over UDP, and seeing a lot of HTTP traffic is rather normal. You may be focusing on QUIC because it's new and unusual, but we don't yet see evidence that it's the source of your network's getting bogged down.

I'm staring at your Wireshark graphs, and what's most suspicious is that 80% of traffic is attributed to TCP, but not called out in the protocol breakdown. What is that traffic?

(Not trying to be defensive; if we are breaking anything for anyone our goal is to fix it ASAP. Just making sure we're diagnosing the problem correctly.)

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.
To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--

 // fnk

[Daniel Raeder]

Thanks for the response Fedor - I'm also not 100% convinced that it's QUIC that is causing my problems, but by blackholing the most talkative IP using QUIC, I was able to stop the issue without any reboot of my cable modem. And by disabling pre-fetching on every instance of Chrome I have, I solved the issue for months (until I added Data Saver (beta)! So there is at least correlative evidence that supports the notion. I'm hoping the issue happens again soon, so I can get the net-internals off to you experts for analysis to rule it in or out definitively. 

[Daniel Raeder]

By the way, I found that I still had Data Saver on one of my phones and disabled it last night. If it is indeed related to my problems, that may be why the issue persisted for a couple more days until I posted this thread. Maybe I should turn it back on.

[Jim Roskind]

Your comment about "disabling pre-fetching on every instance of Chrome" may indeed be separate from QUIC.... but may also suggest other issues are at play.

Historically, there were a number of routers (and/or cable modems) that responded poorly to pre-resolution (speculative DNS requests) as well as pre-connection (speculative TCP connections). 

Example: One class of router crashed (or hung?) when more than about 8 DNS resolutions were pending <yikes>, which drove us (as we evolved Chrome) to heavily restrict the number of simultaneously pending external requests (I think we stay under 8 today).

Example: Some routers have a "feature" (the specs call it a feature) which is labeled "DDOS Protection."  Sadly, the feature is designed to take *you* off the internet if you *you* send too many TCP open requests (re: transmit a SYN) without waiting for related SYNACK response.  Conceptually, they are trying to block virus-riddled home users from *performing* DDOS attacks on the internet.  Sadly, when Chrome sees you visit certain sites (and has records of what happened last time), it may quickly (speculative) open a list of sub-domains. This rapid opening may trigger the aforementioned feature. <sigh!!!>

Neither of those "features" relate much to QUIC.  In fact, QUIC (much like SPDY or HTTP/2) reduces the number of connections that are opened (reducing the impact of those features).  Today, such "value" from QUIC is generally limited to Google (where there are QUIC servers), but HTTP/2 is becoming more and more broadly deployed (providing similar reduction in load on routers and NAT tables (as well as DNS resolvers).

Now that you've posted details of your router, perhaps others on the internet will see a pattern relating to that device.

Jim

[Daniel Raeder]

Interesting. If others find this thread, here are some additional details about my modem:

uBee DVW3201B Docsis 3.0 Telephony EMTA Wireless cable modem U10C046
Boot Code Version : 10.2.1a
Software Version : 9.9.3106SIP
Hardware Version : 3.7.5
Configured in Bridge mode (Bridges entire device. No Routing, No NAT, No DHCP, No Firewall).

For my router, I have an ASUS RT-N66U running firmware version 3.0.0.4.376_3861. I have dropped packets logged, and found no correlation to the problem on the router side in my own diagnosis. But, DoS protection is enabled here. Maybe this is the issue.

[Fedor Kouranov]

Daniel: I did overlook some of the correlative evidence. It is still interesting to classify what that unaccounted TCP traffic does.

It is telling that you start noticing the issue when using remote access or calls. These are often implemented over UDP, so you may be running into some limitations on that front (on premise or at the ISP) when UDP traffic picks up.

--

 // fnk

[Daniel Raeder]

I ran continuous pings for a few weeks. The issue always precedes the phone call or RDP sessions, so something other than those is the cause. But I notice the issue due to its affect on those activities. When I checked the status of pings, I see 900+ ms response times beginning at least a few minutes before.

[ianG]

On 31/10/2015 23:45 pm, Ryan Hamilton wrote:
> On Sat, Oct 31, 2015 at 3:30 PM, ianG <ia...@iang.org

> <mailto:ia...@iang.org>>wrote:

Thanks - I wrote that when I was tired & had forgotten what congestion
control was, just had clue bat in use. Now I get it.

iang

[Daniel Raeder]

I had the issue occur again this morning. I hope this net-internals capture was long enough.

[Fedor Kouranov]

You have huge gaps between packets going out and coming back, up to nearly a second long. That's consistent with the sluggishness you're describing. Probably the packets are loitering in a deep buffer along the way.

Besides that, it doesn't seem this instance of Chrome is flooding anything...

--

 // fnk

[Daniel Raeder]

I saw the QUIC_ERROR: 25, which is a timeout. If this isn't Chrome, I'm back to square one on this elusive problem, and I'm also confused why blackholing the most talkative QUIC google IP stopped the problem at least once. I admit that could be entirely coincidental, but a very strange coincidence indeed.

[Fedor Kouranov]

Timeouts are not surprising if your network is bogged down that hard.

It may be Chrome for all we know, just not the instance you collected logs on. So far it appears that something is flooding your network. More specifically, you suspect too much traffic between the "chatty Google server" and the "talkative PC"? Can you try and track down the source of that traffic on the PC?

Also, how do you know it's QUIC packets? Just because they come from udp/80 or udp/443?

Also, I'm still curious about the unidentified TCP traffic on the wireshark.

[Daniel Raeder]

I've attached the capture filtered by TCP traffic. Lots of Chromecast activity...  

[Fedor Kouranov]

This capture is not during a slowdown, right? Looks healthy... Wireshark seems to try to be too smart with some of the packets and fails to categorize them properly...

[Christian Mueller]

It has already been mentioned but this all appears to point to a send buffer in the modem being way too large in combination with too much data being stuffed into the modem, either by QUIC or by something else. DSL (and probably cable) modems often have a surprisingly large send buffer and, once you managed to fill it up (e.g. file uploads), the modem will add delays in the second range to anything going out.

In SOHO environments, I typically solve this by limiting the traffic going out to the modem to something like 98% of what the modem can send but that works only if the modem has a fixed transmission rate (DSL in my case); cable modems share resources, thus this approach may not work (never had one myself).

In theory, I could imagine that the modem might even try and peek into the traffic to apply some flow control but that will, of course, fail for QUIC packets because the modem will hardly have heard about QUIC at this point. Which might explain why QUIC traffic has more effect on this than regular TCP traffic.

One way to test this would be to upload large test files to an external site while checking the ping time. One half of the tests would use TCP, the other half QUIC (UDP). You'd run 1, 2, 4, 8. ... uploads at the same time until you notice the ping time going up. Since mainly Google servers use QUIC these days, you might want to try uploading something to Google Drive with Chrome.

Thanks,

--Christian

[Daniel Raeder]

Thank you Christian for these excellent ideas. I will give them a try and see if I can forceably reproduce the issue as you recommend. That will greatly help me identify the problem... At this point, I no longer am pointing the finger at QUIC as the culprit. But this discussion has helped quite a bit and given me a number of ideas on things to look at. If I find it indeed does look like QUIC traffic again, I'll have some captures and net-internals uploaded here for further review.

--Christiaj

The information in this message may be confidential.  It is intended solely for

the addressee(s).  If you are not the intended recipient, any disclosure,

copying or distribution of the message, or any action or omission taken by you

in reliance on it, is prohibited and may be unlawful.  Please immediately

contact the sender if you have received this message in error.