how do you test QUIC performances ?

[Sébastien Noël]

Hi everyone

I'm currently trying to test QUIC performances in different conditions (bandwidth,
latency, packet reordering, ...) but i'm very surprised by the results,
and i'm wondering if i'm doing something wrong.

I'm using the code released by google on github
https://github.com/google/proto-quic
Initialy i was using the client from that same repository.

My first test was to simply download a 10MB file on a link with zero latency,
and variable bandwidth:
5mbits    17.9s
10mbits   9s
20mbits   4.5s
50mbits   2s
100mbits  1s

OK - that's mostly what i was expecting.

The next test was to download the same file with a fixed bandwidth (10mbits/s)
and add more latency on the path between the client and the server.
the results are bad :
10ms    21s
15ms    25.5s
20ms    29s
40ms    50.5s
60ms    72.5s

this can't be normal, but it's reproducible :/

if i re-run the same test using chromium from ubuntu16.04 :
10ms    13s
15ms    13s
20ms    14s
40ms    13.5s
60ms    13s
100ms   13s
200ms   13.5s
200ms   15s

That's not as bad as with the 'proto client' but a simple wget (HTTPS)
against the same machine is faster

I did test the proto client against GoogleDrive, numbers were also very hight.

The situation with chromium on GoogleDrive isn't very stable,
i'm getting differents numbers depending of the time of the day.

But in all case wget/https is simply faster than Chromium/QUIC (on Gdrive
or with the proto server)

- is it expected ?

- what tools do you use to test QUIC performances ?


Best regards

Sébastien

[Ian Swett]

I'm fairly sure you're hitting MaxCWND in most of these runs.  I suspect it's because the  

 

if i re-run the same test using chromium from ubuntu16.04 :
10ms    13s
15ms    13s
20ms    14s
40ms    13.5s
60ms    13s
100ms   13s
200ms   13.5s
200ms   15s

That's not as bad as with the 'proto client' but a simple wget (HTTPS)
against the same machine is faster

When the connection is extremely good(ie: localhost or on a fast local network), I would expect the kernelspace TCP implementation to beat QUIC's userspace implementation.  In particular, the UDP path in the kernel is not as optimized as TCP.  Also, for bulk transfers where packet loss is low or zero, TCP works very well.

 

I did test the proto client against GoogleDrive, numbers were also very hight.

The situation with chromium on GoogleDrive isn't very stable,
i'm getting differents numbers depending of the time of the day.

But in all case wget/https is simply faster than Chromium/QUIC (on Gdrive
or with the proto server)

- is it expected ?

- what tools do you use to test QUIC performances ?

We use measured application latency(ie: Search, Youtube) from actual user requests.  These tend to look quite different from single connection benchmarks.  

Best regards

Sébastien

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+...@chromium.org.
To post to this group, send email to proto...@chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

[Ian Swett]

Sorry, I hit send prematurely.

I'm fairly sure you're hitting MaxCWND in most of these runs.  I suspect it's because the client's receive buffer is too small.  This is the SRBF tag in the CHLO.  I don't know why it would be smaller in the proto-quic client, though.  Alternately, it could be flow control, but I don't think that's likely.

[Amit Srivastava]

I am also exploring QUIC performance using the toy client and server and I also have to say the results looked similar to what you mention.
I was advised to use the chrome browser and I I was able to get better results by pointing the the browser to the toy server.

I would like to ask the developers if the code used in production server and google and the chrome (chromium) browser is different from the code used to compile the toy client and the server?

Amit

[Alyssa (Rzeszutek) Wilk]

The toy server and client have maybe 90% overlap with the "real" server (GFE) and client (Chrome) but the differences are pretty important.  For example the toy client and server don't do a particularly good job of ceding, so generally focus on one request/response at a time rather than fair sharing.    Meanwhile as Ian pointed out earlier if you're playing on a single machine you're going to hit cwnd limitations and be comparing kernel looopback hacks to userspace QUIC.  We wouldn't expect that to be particularly flattering and it's not representative of the performance benefits we see.

The results we trust most are from real application latency (which is unfortunately hard for anyone without a google-scale network to reproduce) but we've done some multi-machine loadtesting, and some work with webpagetest as well, which might be easier to clone.

cheers,

Alyssa

[Amit Srivastava]

Hi Alyssa,

Thanks for replying. I have a few follow up questions for you.
1. Whats is GFE? 
2. When you say the toy code does not do a good job of ceding, do you mean the they do not cede bandwidth to competitive flows?

I have a multi machine testbed for my thesis on performance of QUIC as shown below. And I add latency on link L3 from the middlebox.
I run QUIC and TCP servers on  S1 and S2 boxes and access both using clients on C. As per my measurements QUIC uses less bandwidth than competing
TCP flows in other words QUIC's is not sharing bandwidth equally when latencies are high ~100ms.

        L1
S1 -------------\                                   L3
                  [switch]-----Middlebox ---------- [switch]-----C
S2 -------------/
        L2
 
3. Ian mentioned that QUIC hitting max cwnd value for results from chrome browser pointing at toy server. Does that mean that the toy server has a smaller cwnd than that deployed on production Google servers?

4. Is there a way to test the TCP friendliness of QUIC on my testbed and observe the performance benefits you see specially at very high bandwidth delay products such as 100 Mbps capacity links with 200ms latency?
 
Regards,
Amit

[Ian Swett]

On Fri, May 20, 2016 at 5:52 PM, amit srivastava <amit...@gmail.com> wrote:

Hi Alyssa,

Thanks for replying. I have a few follow up questions for you.
1. Whats is GFE? 

GFE stands for Google Front End, which is the Google web server.

 

2. When you say the toy code does not do a good job of ceding, do you mean the they do not cede bandwidth to competitive flows?

The QUIC code in the toy server doesn't cede to other flows on the same server well.  That shouldn't not effect how it cedes to other flows on the network.

 

I have a multi machine testbed for my thesis on performance of QUIC as shown below. And I add latency on link L3 from the middlebox.
I run QUIC and TCP servers on  S1 and S2 boxes and access both using clients on C. As per my measurements QUIC uses less bandwidth than competing
TCP flows in other words QUIC's is not sharing bandwidth equally when latencies are high ~100ms.

        L1
S1 -------------\                                   L3
                  [switch]-----Middlebox ---------- [switch]-----C
S2 -------------/
        L2
 
3. Ian mentioned that QUIC hitting max cwnd value for results from chrome browser pointing at toy server. Does that mean that the toy server has a smaller cwnd than that deployed on production Google servers?

In a CL that will land in Chromium this week, the max CWND has been increased to 2000 by default.  If that isn't large enough, you can just increase the default max in your local code.

 

4. Is there a way to test the TCP friendliness of QUIC on my testbed and observe the performance benefits you see specially at very high bandwidth delay products such as 100 Mbps capacity links with 200ms latency?

Curious, why would you be interested in performance under those characteristics? They're extremely uncommon for internet users.  But with a large enough CWND, it should be possible to test QUIC under those circumstances.

[Erik Klavon]

On May 23, 2016, at 05:50, 'Ian Swett' via QUIC Prototype Protocol Discussion group <proto...@chromium.org> wrote:

On Fri, May 20, 2016 at 5:52 PM, amit srivastava <amit...@gmail.com> wrote:

Hi Alyssa,

Thanks for replying. I have a few follow up questions for you.
1. Whats is GFE? 

GFE stands for Google Front End, which is the Google web server.

 

2. When you say the toy code does not do a good job of ceding, do you mean the they do not cede bandwidth to competitive flows?

The QUIC code in the toy server doesn't cede to other flows on the same server well.  That shouldn't not effect how it cedes to other flows on the network.

 

I have a multi machine testbed for my thesis on performance of QUIC as shown below. And I add latency on link L3 from the middlebox.
I run QUIC and TCP servers on  S1 and S2 boxes and access both using clients on C. As per my measurements QUIC uses less bandwidth than competing
TCP flows in other words QUIC's is not sharing bandwidth equally when latencies are high ~100ms.

        L1
S1 -------------\                                   L3
                  [switch]-----Middlebox ---------- [switch]-----C
S2 -------------/
        L2
 
3. Ian mentioned that QUIC hitting max cwnd value for results from chrome browser pointing at toy server. Does that mean that the toy server has a smaller cwnd than that deployed on production Google servers?

In a CL that will land in Chromium this week, the max CWND has been increased to 2000 by default.  If that isn't large enough, you can just increase the default max in your local code.

 

4. Is there a way to test the TCP friendliness of QUIC on my testbed and observe the performance benefits you see specially at very high bandwidth delay products such as 100 Mbps capacity links with 200ms latency?

Curious, why would you be interested in performance under those characteristics? They're extremely uncommon for internet users.

If you're a large org like Google or Microsoft or Facebook with globally distributed infrastructure, these cases are extremely uncommon. Not all use cases for a fundamental protocol like QUIC will be distributed in this manner.

Consider for example a company with a corporate data center on the east coast of the United States, and a business partner located in Asia. The business partner wishes to transfer large files to/from the example company's corporate datacenter. (This datacenter could be on prem or hosted in the cloud at an east coast facility.) A path latency of around 200 ms is reasonable for this case, and a path capacity of 100Mbps is common in some countries.

Maintainers of TCP stacks are beginning to fix their defaults so that it is possible to make use of all available capacity on high bandwidth delay product paths. Please consider these use cases when setting defaults for implementations of QUIC, especially in general purpose cases like browsers. It would be a shame to repeat past mistakes with a new protocol.

[Dev Sam]

I did test the proto client against GoogleDrive, numbers were also very hight.

Hi!
How are you able to test the proto client with Google Drive?
If I were to enter the Google Drive link as argument to quic_client, I will get a redirection response.

body: <HTML>
<HEAD>
<TITLE>Moved Temporarily</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Temporarily</H1>
The document has moved <A HREF="https://drive.google.com/file/d/0B2wDkFsY0DtbMHFlR3I3LVY3UVE/view?usp=drive_web">here</A>.
</BODY>
</HTML>

Request succeeded (redirect 302).

Thanks!

[S Ratan Kumar]

Hai Amit,

Good morning. We are interested in starting similar kind of work. Can you share your experience about the best QUIC client, QUIC server to use?

Thanks & Regards.

[Neil Craig]

Hello all

I don’t have a great deal of experience with QUIC other than basics but in answer to questions below:

GFE == Google Front End (AFAIK) - the public web servers used by Google for e.g. Search/Youtube and so on. Look at the “server" HTTP response header and you’ll see a value of “GFE”

Caddy (https://caddyserver.com/) is another “real” QUC-capable server (though QUIC support is flagged as experimental) - you can run this yourself which is nice.

Clients: Chrome supports QUIC, stable is on QUIC 35, canary is on 37 currently. The nice thing about Chrome is that you can enable the “protocol” column in the network tab of dev tools and check that QUIC is active (you’ll need to enable QUIC (see https://ma.ttias.be/enable-quic-protocol-google-chrome/) in Chrome first. You can also debug the QUIC comms in chrome://net-internals.

Hope that helps

Neil

[S Ratan Kumar]

Thank you very much. I want to create test bed locally . Is it possible to run caddy server on local host with QUIC support?

Kindly guide me.

Thanks & Regards.

Mr S Ratan Kumar  B Tech,M Tech,(Ph.D)
Assoc.Prof,CSE ANITS(A) www.anits.edu.in 

Coordinator-ANITS Incubation Center,

Coordinator-ANITS CSE Dept Autonomy,

SPOC- Infosys Campus Connect,

SPOC-ANITS ACM ICPC.

https://sites.google.com/site/sajjaratankumar/

Ph.No :9052492777

"People may hear your words,but they feel your attitude."

                              -John C Maxwell

--
You received this message because you are subscribed to the Google Groups "QUIC Prototype Protocol Discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to proto-quic+unsubscribe@chromium.org.

[Neil Craig]

I believe you can run Caddy locally with quic though i haven't done it myself. You'd need to provide your own cert and key (but they could be self-signed).

You'd need to use the tls config block from https://caddyserver.com/docs/tls in your Caddyfile, like so:

tls [cert key]

So you could have (untested) something like this in your Caddyfile:

site1.example.com:443 {

  root /path/to/html/files

  tls /path/to/cert /path/to/key

}

then create an entry in your /etc/hosts file for 127.0.0.1 site1.example.com and start caddy with caddy -quic.

I think that'd do the job, you might need to ensure that your ciphersuites are valid for quic.

Is that any help?

Cheers

Neil

On Monday, May 9, 2016 at 9:37:13 AM UTC+1, Sébastien Noël wrote:

[S Ratan Kumar]

Hai Neil Craig,

Good morning. Thank you very much for the information. I am facing trouble with certificates. I am not able to create self signed certificates. Kindly through some light on this issue.

Thanks & Regards.

Mr S Ratan Kumar  B Tech,M Tech,(Ph.D)
Assoc.Prof,CSE ANITS(A) www.anits.edu.in 

Coordinator-ANITS Incubation Center,

Coordinator-ANITS CSE Dept Autonomy,

SPOC- Infosys Campus Connect,

SPOC-ANITS ACM ICPC.

https://sites.google.com/site/sajjaratankumar/

Ph.No :9052492777

"People may hear your words,but they feel your attitude."

                              -John C Maxwell

[Neil Craig]

We're getting a bit off-topic here but assuming you're on a unix-like OS (i.e. not Windows), this might help:

https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl

If not, i'd suggest Googling as it's a very well-documented process.

Hope that helps!

On Monday, May 9, 2016 at 9:37:13 AM UTC+1, Sébastien Noël wrote:

[S Ratan Kumar]

Ok. Thank you very much. I will get back to you after setting up caddy on localhost.

Mr S Ratan Kumar  B Tech,M Tech,(Ph.D)
Assoc.Prof,CSE ANITS(A) www.anits.edu.in 

Coordinator-ANITS Incubation Center,

Coordinator-ANITS CSE Dept Autonomy,

SPOC- Infosys Campus Connect,

SPOC-ANITS ACM ICPC.

https://sites.google.com/site/sajjaratankumar/

Ph.No :9052492777

"People may hear your words,but they feel your attitude."

                              -John C Maxwell