QUIC is much slower than HTTP/2 during experiments

[Suk Hwan Hong]

Hi, I am currently doing a project for my network class, and the project topic is a performance comparison between QUIC and HTTP/2.

We've set up 2 Ubuntu virtual boxes (server and client) connected to each other via virtual internal network.

Server is running Apache (serving HTTP/2) and a QUIC toy server (build from source), and client is running chromium to load the web page from the server.

I've created a simple test webpage with a single large image (~15MB) and replicated it to both Apache and QUIC servers.

However, when I tested the web page using chromium, the page load time is SIGNIFICANTLY slower with QUIC than HTTP/2.

QUIC takes at least ~15 seconds to load the page ("Load time" in developer console), whereas HTTP takes only ~3 seconds to load the page. (I've disabled the cache).

When I look at the "bytes received" graph in chrome://net-internals timeline, HTTP/2 reaches up to ~10MB/s, but QUIC only reachs up to 1MB/s, an astonishing ten-fold difference.

Is there any reason why this may be happening? (Am I missing something???)

Why is QUIC not sending the data as fast as HTTP/2?

(When I wiresharked the packets, QUIC is sending much larger number of UDP packets with smaller size (~1350 bytes), but HTTP/2 is sending much smaller number of TCP packets with larger size (~20000 bytes). But this is expected behavior as specified by the protocol....)

I would appreciate any insights into this issue.

Thanks.

[Amit Srivastava]

I had the exact same issue when I was using the toy client but when I switched to chrome browser I saw improvement in the performance.

I can share the commands that I use.

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.
To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

[Chetan Ahuja]

Hi Amit,

   Did you actually do a side-by-side comparison with HTTP(1.1/2) against QUIC. Mind sharing the results?

Thanks

Chetan

[Dev Sam]

Hi!

I am getting the same results.
Are we actually able to get good performance using the toy server?

Is it meaningful to use the toy server for performance comparison with HTTP/1.1/2?

Thanks

2016년 4월 13일 수요일 오전 11시 53분 14초 UTC+9, Suk Hwan Hong 님의 말:

[Ian Swett]

On Fri, Jun 17, 2016 at 3:52 AM, Dev Sam <sboos...@gmail.com> wrote:

Hi!

I am getting the same results.
Are we actually able to get good performance using the toy server?

If by good performance, you're thinking of high bandwidths, then no.  If you want to see how QUIC does at a few Mbps and with a few % loss, I would expect it to be more indicative.

--

[Dev Sam]

Thanks Ian.

Yes I was thinking of testing with high bandwidths.

But what is the reason for not being able to achieve high bandwidths?

2016년 6월 21일 화요일 오전 3시 25분 1초 UTC+9, Ian Swett 님의 말:

[Ian Swett]

I don't know the exact reasons, but I know we've spent almost no time time optimizing it and there are multiple internal and external reports indicating the test client and server are slower than Chrome against our production servers.  If you or anyone else would like to make optimizations, feel free to send us a Chromium CL.

[Povilas Balciunas]

Does your production server use the same QUIC code from chromium repo?

[Ian Swett]

Most of it is shared, but some code is adapted for Chromium, which runs on many platforms.

As an example, I don't believe the Chromium code uses recvmmsg by default, because MMSG_MORE is defined to 0, I believe for platform compatibility reasons.

https://cs.chromium.org/chromium/src/net/tools/quic/quic_packet_reader.cc?q=quic_packet_&sq=package:chromium&l=24

[Dev Sam]

Hi Ian,

According to our tests QUIC's throughput is does not reach higher than 150Mbps. 

If we set the link capacity below this, the capacity is reached. 

For example, 100 Mbps is measured on a 100Mbps link but only 150Mbps is measured for a 1Gbps link. 

It is not degrading gradually but instead it is clipped. 

You might not have done a lot of tests on high bandwidths but would you know any reason why throughput will be clipped at 150Mbps?

Thanks!

2016년 6월 22일 수요일 오후 8시 48분 16초 UTC+9, Ian Swett 님의 말:

[Ian Swett]

That does match some previous results, and I think it's due to implementation details of the client and server, but I don't know what the current bottleneck is.

It's possible to get higher bandwidths with Chrome and our production server in my experience. 

[Amit Srivastava]

This looks like an interesting task. I will look into it. And I might ask more questions as I read the code.

[Amit Srivastava]

Hi Chetan ,

I did not do a web page load time comparison. What I am doing is comparing QUIC to TCP as general purpose transport protocols. Like many  people

I got slower data rates with the toy client and server. So I used a chrome browser pointing at the toy server. My main concern was data rates at higher

RTT values which were partially addressed by switching to Chrome code.

Regards,

Amit Srivastava