QUIC Protocol Errors on Network

[Nathan Totten]

I am experiences QUIC protocol errors on every device connected to my network. I have tried troubleshooting the issue with both the manufacture of my networking equipment (Ubiquiti) and my ISP and neither have been able to find an issue. I am not an expert by an stretch when it comes to diagnosing these sorts of networking issues and I have exhausted all options that I can think of so I am hoping maybe this forum can help. 

The short of the issue is that every QUIC connection on my network seems to time out after 30 seconds with the error `net::ERR_QUIC_PROTOCOL_ERROR`. I have reproduced this on multiple Macbooks running Chrome 61 on macOS sierra and high sierra. I have taken one of these computers to a different network and not experienced the error. In addition, when I connect my computer to a VPN that is setup on AWS I do not experience the error. 

My assumption is that this is caused by a bug on my networking equipment or some misconfiguration with my ISP.

Networking Equipment is all Ubiquiti Unifi equipment (security gateway, switches, and APs). My ISP is Comporium (https://www.comporium.com/) and I am configured with a static IP address. They claim that they aren't blocking or otherwise manipulating traffic on UDP port 80 or 443, but of course that is just their support people so they may not have the full picture.

I am wondering what I can do to diagnose the issue. I sent a bunch of logs to Ubiquiti and worked through this with them and they couldn't find any issues. The easy answer is to just disable QUIC in Chrome, but then I lose the benefits.

Any help would be appreciated.

[Antoine Saverimoutou]

Hello Nathan,

I went through a first search of the reason of the "ERR_QUIC_PROTOCOL-ERROR" some time back when making some tests. First of all, you have no proxies or firewall in your network dropping QUIC packets or even your network equipment droping all on 443? Did you try lauching a tshark to see if packets are dropped? Did u try some troubleshooting via netInternals or post some results?

My problem was not the same as yours but maybe some of the following information I was able to collect may help you to see clearer.

When making tests via selenium on Chrome 61, I asked for QUIC protocol and QUIC_FORCED protocol. The QUIC_FORCED means  to connect via UDP 443.

When asking QUIC (basic), the websites not delevering QUIC made a fall back to http/2 or http/1 and everything was fine.

But when asking QUIC_FORCED, I experienced the "ERR_QUIC_PROTOCOL_ERROR" following the following use cases:

1. When trying to get a https website:

        - There is sometimes a fallback, the CHLO message sent by the client to the server is interpreted, TLS1.2 packets exchanged and connection made

        - Sometimes the CHLO is sent by the client but the server doesnot respond as shown in the figure below. At the 6th CHLO, Chrome sends a packet with Payload "No recent Activity" and timeout after 30secs

        - On other situations, I went in netInternals and saw a net_error = -356 (QUIC_PROTOCOL_ERROR) where the HTTP_TRANSACTION_READ_BODY which FAILED as well as the HTTP_CACHE_WRITE_INFO which FAILED leading to a net_error= -358 (QUIC_HANDSHAKE_FAILED)

2. When trying an unsecure website, i.e http, all goes smooth

Cheers

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.
To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

[Nathan Totten]

I have tried debugging with wireshark and netinternals, but I will admit I am not super familiar with debugging using these tools. What I was able to find is that there are a number of errors on QUIC traffic "ETHERNET FRAME CHECK SEQUENCE INCORRECT". Not sure what this means exactly, but the Frame check sequence is set to 0x000000 in every case. Maybe a router is corrupting these? Screenshot attached of wireshark capture. I'm happy to provide more info as well.

When running netinternals I am not seeing the errors you saw, but I do see instances of "quic_rst_stream_error".

{"params":{"quic_rst_stream_error":6,"stream_id":17},"phase":0,"source":{"id":976962,"type":10},"time":"177731344","type":245},

[Nathan Totten]

I ran some more debugging with net internals. You can see the output here: https://gist.github.com/ntotten/e0df5e30bce1a622706d8d82602fe70f

[Cherie Shi]

Nathan, could you provide a complete net internals so that we can take a close look from there? The one you uploaded on github is truncated. 

Instruction: https://dev.chromium.org/for-testers/providing-network-details

On Thu, Oct 5, 2017 at 7:56 AM, Nathan Totten <nat...@ntotten.com> wrote:

I ran some more debugging with net internals. You can see the output here: https://gist.github.com/ntotten/e0df5e30bce1a622706d8d82602fe70f

--

[Nathan Totten]

Attached is complete log. This was done in-private. I visited a youtube page and a video started playing. The QUIC error occurred several times.

Thanks!

On Fri, Oct 6, 2017 at 12:22 AM Cherie Shi <zho...@chromium.org> wrote:

Nathan, could you provide a complete net internals so that we can take a close look from there? The one you uploaded on github is truncated. 

Instruction: https://dev.chromium.org/for-testers/providing-network-details

On Thu, Oct 5, 2017 at 7:56 AM, Nathan Totten <nat...@ntotten.com> wrote:

I ran some more debugging with net internals. You can see the output here: https://gist.github.com/ntotten/e0df5e30bce1a622706d8d82602fe70f

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

[Nathan Totten]

I wanted to also note, that this morning I was able to plug my computer directly into the ISP connection and set to the static IP address. I was still able to reproduce this issue. I am guessing there is something blocking this from my ISP. Is that something you have seen before?

[Cherie Shi]

Thanks for the NetLog, Nathan. 

I just looked through your NetLog and noticed QUIC was working for a while in the beginning and then starts to encounter a huge amount of packet loss afterwards. It eventually got closed because too many RTOs. That smells like middlebox issue. Do you know if your network is controlled by any middlebox? Any does that have any limitation on UDP traffic? 

If you are still have issues, you might consider file a bug with us on https://crbug.com/wizard and we can move the discussions there. 

t=11735 [st=6270]    QUIC_SESSION_ACK_FRAME_RECEIVED
                     --> delta_time_largest_observed_us = "804096"
                     --> largest_observed = "30"
                     --> missing_packets = ["7","9","10","11","12","13","14","15","16","17","18","19","21","22","23","24","25","26","27","28"]
                     --> received_packet_times = [{"packet_number":30,"received":"264970058887"}]

On Fri, Oct 6, 2017 at 8:46 AM, Nathan Totten <nat...@ntotten.com> wrote:

I wanted to also note, that this morning I was able to plug my computer directly into the ISP connection and set to the static IP address. I was still able to reproduce this issue. I am guessing there is something blocking this from my ISP. Is that something you have seen before?

On Friday, October 6, 2017 at 9:35:14 AM UTC-4, Nathan Totten wrote:

Attached is complete log. This was done in-private. I visited a youtube page and a video started playing. The QUIC error occurred several times.

Thanks!

On Fri, Oct 6, 2017 at 12:22 AM Cherie Shi <zho...@chromium.org> wrote:

Nathan, could you provide a complete net internals so that we can take a close look from there? The one you uploaded on github is truncated. 

Instruction: https://dev.chromium.org/for-testers/providing-network-details

On Thu, Oct 5, 2017 at 7:56 AM, Nathan Totten <nat...@ntotten.com> wrote:

I ran some more debugging with net internals. You can see the output here: https://gist.github.com/ntotten/e0df5e30bce1a622706d8d82602fe70f

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

[Nathan Totten]

I am not sure, but I am guessing that Comporium has some middlebox on their network somewhere. Adding some of the people from Comporium who might be able to provide more information.

On Fri, Oct 6, 2017 at 2:37 PM Cherie Shi <zho...@chromium.org> wrote:

Thanks for the NetLog, Nathan. 

I just looked through your NetLog and noticed QUIC was working for a while in the beginning and then starts to encounter a huge amount of packet loss afterwards. It eventually got closed because too many RTOs. That smells like middlebox issue. Do you know if your network is controlled by any middlebox? Any does that have any limitation on UDP traffic? 

If you are still have issues, you might consider file a bug with us on https://crbug.com/wizard and we can move the discussions there. 

t=11735 [st=6270]    QUIC_SESSION_ACK_FRAME_RECEIVED
                     --> delta_time_largest_observed_us = "804096"
                     --> largest_observed = "30"
                     --> missing_packets = ["7","9","10","11","12","13","14","15","16","17","18","19","21","22","23","24","25","26","27","28"]
                     --> received_packet_times = [{"packet_number":30,"received":"264970058887"}]

On Fri, Oct 6, 2017 at 8:46 AM, Nathan Totten <nat...@ntotten.com> wrote:

I wanted to also note, that this morning I was able to plug my computer directly into the ISP connection and set to the static IP address. I was still able to reproduce this issue. I am guessing there is something blocking this from my ISP. Is that something you have seen before?

On Friday, October 6, 2017 at 9:35:14 AM UTC-4, Nathan Totten wrote:

Attached is complete log. This was done in-private. I visited a youtube page and a video started playing. The QUIC error occurred several times.

Thanks!

On Fri, Oct 6, 2017 at 12:22 AM Cherie Shi <zho...@chromium.org> wrote:

Nathan, could you provide a complete net internals so that we can take a close look from there? The one you uploaded on github is truncated. 

Instruction: https://dev.chromium.org/for-testers/providing-network-details

On Thu, Oct 5, 2017 at 7:56 AM, Nathan Totten <nat...@ntotten.com> wrote:

I ran some more debugging with net internals. You can see the output here: https://gist.github.com/ntotten/e0df5e30bce1a622706d8d82602fe70f

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

[Ian Swett]

A few years ago, there were some issues with Ubiquiti EdgeMAX routers blackholing QUIC traffic after a few seconds of normal connectivity.  

https://community.ubnt.com/t5/EdgeMAX/Problems-with-Google-Drive-quot-Trying-to-connect-quot-and-quot/m-p/1247106#M64734

It seemed like the problem was solved by newer firmware(1.8.0 and above).  I'm not sure if there were similar issues with the Unifi hardware, but it might be worth considering a firmware update if there's a newer one available and you can update it.

The wireshark log is interesting, since it seems to indicate the UDP checksum is set in one direction and is the default of 0 in the other direction.  But that shouldn't be an issue, since it's optional for IPv4.

Thanks for reporting this and providing so much debug info.

On Fri, Oct 6, 2017 at 3:22 PM, Nathan Totten <nat...@ntotten.com> wrote:

I am not sure, but I am guessing that Comporium has some middlebox on their network somewhere. Adding some of the people from Comporium who might be able to provide more information.

On Fri, Oct 6, 2017 at 2:37 PM Cherie Shi <zho...@chromium.org> wrote:

Thanks for the NetLog, Nathan. 

I just looked through your NetLog and noticed QUIC was working for a while in the beginning and then starts to encounter a huge amount of packet loss afterwards. It eventually got closed because too many RTOs. That smells like middlebox issue. Do you know if your network is controlled by any middlebox? Any does that have any limitation on UDP traffic? 

If you are still have issues, you might consider file a bug with us on https://crbug.com/wizard and we can move the discussions there. 

t=11735 [st=6270]    QUIC_SESSION_ACK_FRAME_RECEIVED
                     --> delta_time_largest_observed_us = "804096"
                     --> largest_observed = "30"
                     --> missing_packets = ["7","9","10","11","12","13","14","15","16","17","18","19","21","22","23","24","25","26","27","28"]
                     --> received_packet_times = [{"packet_number":30,"received":"264970058887"}]

On Fri, Oct 6, 2017 at 8:46 AM, Nathan Totten <nat...@ntotten.com> wrote:

I wanted to also note, that this morning I was able to plug my computer directly into the ISP connection and set to the static IP address. I was still able to reproduce this issue. I am guessing there is something blocking this from my ISP. Is that something you have seen before?

On Friday, October 6, 2017 at 9:35:14 AM UTC-4, Nathan Totten wrote:

Attached is complete log. This was done in-private. I visited a youtube page and a video started playing. The QUIC error occurred several times.

Thanks!

On Fri, Oct 6, 2017 at 12:22 AM Cherie Shi <zho...@chromium.org> wrote:

Nathan, could you provide a complete net internals so that we can take a close look from there? The one you uploaded on github is truncated. 

Instruction: https://dev.chromium.org/for-testers/providing-network-details

On Thu, Oct 5, 2017 at 7:56 AM, Nathan Totten <nat...@ntotten.com> wrote:

I ran some more debugging with net internals. You can see the output here: https://gist.github.com/ntotten/e0df5e30bce1a622706d8d82602fe70f

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

[Nathan Totten]

I thought maybe this was the case too, but I plugged my computer directly into the ISP and bypassed all of my Unifi equipment and saw the same error. At this point I am 99% certain it is an issue with my ISP.

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

[Ian Swett]

Thanks Nathan.

Can you grab one more netlog for me?  Only one connection failed in that one, and it'd be nice to see two failures to see how consistent the behavior is.

Thanks, Ian

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

[Ryan Hamilton]

Also, we're running an experiment to see if we can make Chrome handle situations like this more gracefully. Can you try running Chrome with the arguments specified here:

https://bugs.chromium.org/p/chromium/issues/detail?id=709912#c8

And see if that makes a difference?

Cheers,

Ryan

[Nathan Totten]

Ian -

Attached is a netlog that should contain 5 or 6 different instances of the error.

Best,
Nathan

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.

To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.