Question about http/2 connection creation bahavior in chrome

[Johnson Li]

When visiting https://dropbox.com via HTTP/2, I noticed that there are two connection-id for the domain cfl.dropboxstatic.com. But It should be sufficient to keep only one connection for each domain when using HTTP/2. 

So I am wondering when and why chrom decides to create a new HTTP/2 connection even if there has already been an old one.

Thanks.

[Bence Béky]

This is related to https://crbug.com/621597. In short, Chromium opens
up a number of TLS connections at the same time, and only finds out
that HTTP/2 is negotiated after a few roundtrips.

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev

[Ben Maurer]

Hey

You guys are using crossorigin="anonymous" on your JS/CSS which seems to be loaded on that domain. Chrome is unable to share data between anonymous resources (eg your JS/CSS) and non-anonymous resources (eg images). We found a similar problem at FB -- we discussed it a bit on the net-dev mailing list. The worry here was that an evil server might try to correlate the cookies on an anonymous request and correlate them with a non-anonymous request.

We changed to using with-credentials for the cross-origin tag and it solved the problem. HTTP2 gives good compression of any cookies so it's better just to send the same headers for all requests.

I think it could be worth re-evaluating this behavior if at all possible. crossorigin=anonymous is a popular way to get line numbers for error messages. With the HTML spec requiring CORS for es6 modules more people will be using anonymous and run into this.

-b

[Matt Menke]

[+mkwst]  Are we sending cookies on requests for anonymous resources?  That should only happen if there are requests that disallow HTTP auth, but allow sending cookies (Or allow sending cookies, but not setting them).  Otherwise, only cookie-less requests should be included with anonymous requests.

--
You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+unsubscribe@chromium.org.
To post to this group, send email to net...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/aed17146-ff59-42f6-90fd-51c3a69be40e%40chromium.org.

[Ben Maurer]

https://www.w3.org/TR/cors/#terminology

The term user credentials for the purposes of this specification means cookies, HTTP authentication, and client-side SSL certificates that would be sent based on the user agent's previous interactions with the origin. Specifically it does not refer to proxy authentication or the Origin header.

[Matt Menke]

Right, I'm trying to figure out what you mean by "The worry here was that an evil server might try to correlate the cookies on an anonymous request and correlate them with a non-anonymous request."  I admittedly don't know much about anonymous requests, other than that Chrome's network stack considers requests anonymous (And directs them to another socket pool) if they have any one of the following three properties:  don't send cookies, don't set cookies, don't send auth data.

You sentence implies we are sending cookies with anonymous requests, which implies don't sent cookies is not sent, but one of the other two are.  Am I misunderstanding you?

--

You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+unsubscribe@chromium.org.
To post to this group, send email to net...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/43b00f5b-6567-47dd-9c51-258f38f1fb35%40chromium.org.

[Ben Maurer]

My understanding is that the anonymous requests do *NOT* have cookies sent nor is auth data sent. Chrome puts them over the anonymous socket pool.

I think it might be worth putting these requests over the normal sockets. 

To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.

To post to this group, send email to net...@chromium.org.

[Jawad Manzoor]

Hey.

I am doing research on HTTP/2 performance and I also noticed this behavior of parallel HTTP/2 connections. I quantified how often this happens using passive measurements and results are presented in this paper: http://ieeexplore.ieee.org/abstract/document/7818414/

Now I want to do some experiments by setting different number of parallel connections in HTTP/2 ( e.g, 2, 4 or 6). I was able to do it in Firefox and now I am trying with Chromium.

However, I am unable to locate the exact place in Chromium source code where the limit on HTTP/2 connections is implemented and further connections are prevented. I was hoping to get information from https://crbug.com/621597 but I do not have access to view it. I would appreciate any help on this.

--

[Bence Béky]

Forwarding some discussion (with Jawad's permission) in case other developers also find it of interest.

Forwarded conversation
Subject: Private message regarding: [chromium-dev] Question about http/2 connection creation bahavior in chrome
------------------------

From: Jawad Manzoor <jawadm...@gmail.com>
Date: Tue, Jun 6, 2017 at 6:50 PM
To: b...@chromium.org

Hey.

I am doing research on HTTP/2 performance and I also noticed this behavior of parallel HTTP/2 connections. I quantified how often this happens using passive measurements and results are presented in this paper: http://ieeexplore.ieee.org/abstract/document/7818414/

Now I want to do some experiments by setting different number of parallel connections in HTTP/2 ( e.g, 2, 4 or 6). I was able to do it in Firefox and now I am trying with Chromium.

However, I am unable to locate the exact place in Chromium source code where the limit on HTTP/2 connections is implemented and further connections are prevented. I was hoping to get information from https://crbug.com/621597 but I do not have access to view it. I would appreciate any help on this.

On Monday, October 24, 2016 at 9:00:37 PM UTC+2, Bence Béky wrote:

This is related to https://crbug.com/621597.  In short, Chromium opens 
up a number of TLS connections at the same time, and only finds out 
that HTTP/2 is negotiated after a few roundtrips. 

On Tue, Oct 11, 2016 at 9:34 PM, Johnson Li <johnso...@gmail.com> wrote: 
> 
> When visiting https://dropbox.com via HTTP/2, I noticed that there are two 
> connection-id for the domain cfl.dropboxstatic.com. But It should be 
> sufficient to keep only one connection for each domain when using HTTP/2. 
> 
> So I am wondering when and why chrom decides to create a new HTTP/2 
> connection even if there has already been an old one. 
> 
> Thanks. 
> 
> -- 
> -- 
> Chromium Developers mailing list: chromi...@chromium.org 
> View archives, change email options, or unsubscribe: 
> http://groups.google.com/a/chromium.org/group/chromium-dev 

----------
From: Bence Béky <b...@chromium.org>
Date: Tue, Jun 6, 2017 at 7:02 PM
To: Jawad Manzoor <jawadm...@gmail.com>

Hi Jawad,

To my best understanding, Chromium always uses only one HTTP/2 connection (for each user profile and for each privacy mode).  At first there are 6 TLS socket initiated to the same server as usual, but if they negotiate HTTP/2, then only the first one is actually used, the rest are just sitting in the socket pool unused.  There is currently work to change this to only open one connection if the server is known to support HTTP/2, and/or close idle HTTP/2 sockets more aggressively.

I imagine that you are trying to use multiple HTTP/2 connections in parallel, with each requests assigned to one of the multiple connections at random, or to the one with the lowers number of active connections, or something like that.  Like when six HTTP/1.1 connections are used parallel.  Is that right?  Unfortunately this is currently not supported by Chromium.

Cheers,

Bence

----------
From: Jawad Manzoor <jawadm...@gmail.com>
Date: Tue, Jun 6, 2017 at 7:25 PM
To: Bence Béky <b...@chromium.org>

I imagine that you are trying to use multiple HTTP/2 connections in parallel, with each requests assigned to one of the multiple connections at random, or to the one with the lowers number of active connections, or something like that.  Like when six HTTP/1.1 connections are used parallel.  Is that right?  Unfortunately this is currently not supported by Chromium.

Yes thats correct. I am trying to use multiple HTTP/2 connections in the same way as they are used in HTTP/1, just for some experiments.

I understand that its currently not supported in Chromium. I want to implement it myself and build it from source to test a few things related to my research. I discussed this with Mike Belshe a few months ago and he agreed that the single connection limit is too conservative and leads to poor performance in a number of different network conditions. That's why I want to test different number of connections.

Specifically, I want some pointers in the source code where the limit of a single HTTP/2 connection is implemented, so that I can change that limit to 2 or 4 connections. Could you point me to the relevant classes/functions in the source code?

Regards,

-

Jawad

----------
From: Bence Béky <b...@chromium.org>
Date: Tue, Jun 6, 2017 at 8:11 PM
To: Jawad Manzoor <jawadm...@gmail.com>

Hi Jawad,

Oh, sorry I didn't understand your request the first time.  I thought you were looking for a const int kMaxNumberOfConnections = 1; or similar statement.

An HTTP/2 connection is represented by a SpdySession object.  SpdySessionPool owns SpdySession objects, keyed by SpdySessionKey, at most one for each.  There's also some complicated alias mechanism for IP-based pooling.  HttpStreamFactoryImpl::Job tries to find an existing SpdySession to pool to, using SpdySessionPool::FindAvailableSession().  If not found, it requests a socket from the socket pool, which either creates a new socket or hands out an idle socket.  A socket is an open TLS connection with no HTTP/2 connection initialized on it, that is, no preface and initial SETTINGS sent (but HTTP/2 is already negotiated in the TLS handshake using ALPN).  Then HttpStreamFactoryImpl::Job hands this socket (held by the |connection_| member) to SpdySessionPool::CreateAvailableSessionFromSocket() which creates a new SpdySession on it, and makes SpdySessionPool own that SpdySession.

You will need to make sure that SpdySessionPool can handle multiple SpdySessions for each SpdySessionKey, make FindAvailableSession() pick one out of the multiple active ones for each request, and find a place to initiate creating new ones.  Sounds like a lot of work to me, let me know if you get stuck, I'll try to help.

BTW I do not know exactly why https://crbug.com/621597 is restricted, but it does not seem to have information relevant to your goal.

Also, if you are okay with it, I would like to move this discussion to the net...@chromium.org mailing list, so that other developers can also benefit from this discussion.

Cheers,

Bence

----------
From: Jawad Manzoor <jawadm...@gmail.com>
Date: Tue, Jun 6, 2017 at 10:16 PM
To: Bence Béky <b...@chromium.org>

Thanks a lot for the explanation Bence. I will let you know how it goes.

And yes you may take this discussion to the mailing list. It would be good to discuss it with other developers. 

Regards, 

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/d83a8fad-6882-4217-85c6-dd156baf2295%40chromium.org.

[Bence Béky]

Hi Jawad,

FYI https://crbug.com/718576 is the one about not opening 6 sockets, only 1, to a server that has supported HTTP/2 in the past.  You might need to undo some of that future work for your experiment.

Cheers,

Bence

On Wed, Jun 7, 2017 at 8:11 AM, Bence Béky <b...@chromium.org> wrote:

Forwarding some discussion (with Jawad's permission) in case other developers also find it of interest.

Forwarded conversation
Subject: Private message regarding: [chromium-dev] Question about http/2 connection creation bahavior in chrome
------------------------

From: Jawad Manzoor <jawadmanzoor@gmail.com>

Date: Tue, Jun 6, 2017 at 7:25 PM
To: Bence Béky <b...@chromium.org>

I imagine that you are trying to use multiple HTTP/2 connections in parallel, with each requests assigned to one of the multiple connections at random, or to the one with the lowers number of active connections, or something like that.  Like when six HTTP/1.1 connections are used parallel.  Is that right?  Unfortunately this is currently not supported by Chromium.

Yes thats correct. I am trying to use multiple HTTP/2 connections in the same way as they are used in HTTP/1, just for some experiments.

I understand that its currently not supported in Chromium. I want to implement it myself and build it from source to test a few things related to my research. I discussed this with Mike Belshe a few months ago and he agreed that the single connection limit is too conservative and leads to poor performance in a number of different network conditions. That's why I want to test different number of connections.

Specifically, I want some pointers in the source code where the limit of a single HTTP/2 connection is implemented, so that I can change that limit to 2 or 4 connections. Could you point me to the relevant classes/functions in the source code?

Regards,

-

Jawad

----------
From: Bence Béky <b...@chromium.org>
Date: Tue, Jun 6, 2017 at 8:11 PM
To: Jawad Manzoor <jawadm...@gmail.com>

Hi Jawad,

Oh, sorry I didn't understand your request the first time.  I thought you were looking for a const int kMaxNumberOfConnections = 1; or similar statement.

An HTTP/2 connection is represented by a SpdySession object.  SpdySessionPool owns SpdySession objects, keyed by SpdySessionKey, at most one for each.  There's also some complicated alias mechanism for IP-based pooling.  HttpStreamFactoryImpl::Job tries to find an existing SpdySession to pool to, using SpdySessionPool::FindAvailableSession().  If not found, it requests a socket from the socket pool, which either creates a new socket or hands out an idle socket.  A socket is an open TLS connection with no HTTP/2 connection initialized on it, that is, no preface and initial SETTINGS sent (but HTTP/2 is already negotiated in the TLS handshake using ALPN).  Then HttpStreamFactoryImpl::Job hands this socket (held by the |connection_| member) to SpdySessionPool::CreateAvailableSessionFromSocket() which creates a new SpdySession on it, and makes SpdySessionPool own that SpdySession.

You will need to make sure that SpdySessionPool can handle multiple SpdySessions for each SpdySessionKey, make FindAvailableSession() pick one out of the multiple active ones for each request, and find a place to initiate creating new ones.  Sounds like a lot of work to me, let me know if you get stuck, I'll try to help.

BTW I do not know exactly why https://crbug.com/621597 is restricted, but it does not seem to have information relevant to your goal.

Also, if you are okay with it, I would like to move this discussion to the net...@chromium.org mailing list, so that other developers can also benefit from this discussion.

Cheers,

Bence

----------

From: Jawad Manzoor <jawadmanzoor@gmail.com>

Date: Tue, Jun 6, 2017 at 10:16 PM
To: Bence Béky <b...@chromium.org>

Thanks a lot for the explanation Bence. I will let you know how it goes.

And yes you may take this discussion to the mailing list. It would be good to discuss it with other developers. 

Regards, 

[Bence Béky]

Hi,

Excellent!

1. In the same view, do you only see one connection to the server on a build without your change?  I could imagine two connections on a regular build per HostPortPair, and four connections with your modification, because SpdySessionKey also has a PrivacyMode field that can take two values.

If this is not why you have too many connections, then you might want to do some printf-style debugging, maybe logging some information every time FindAvailableSessions() is called.

2. This happens to me all the time.  Are the crashes not related to your change?  I vaguely remember that there is such a thing as LKGR, last known good revision, which of course changes very often, you might want to try patching one of those.  Or start a new thread on chromi...@chromium.org and ask how people tackle this problem.  Make sure to include the crash dump.

If, on the other hand, the crash is related to your change, then of course you have to fix that.

Let me know if you have any more quesions.  Cheers,

Bence

On Fri, Jun 9, 2017 at 12:13 PM, Jawad Manzoor <jawadm...@gmail.com> wrote:

Hi Bence,

I have made the changes as you suggested. AvailableSessionMap now stores std::set<base::WeakPtr<SpdySession>> for each SpdySessionKey. In FindAvailableSession() i get the set and check its size, if the size is less than 2 then i don't return the existing SpdySession, otherwise one of the sessions is randomly selected and returned.

I have also made changes to several other methods that extracted SpdySession from map. Now the set is first retrieved against SpdySessionKey and from this set a SpdySession is extracted.

You also mentioned that I need to find a place to initiate new SpdySessions. I haven't done this yet as I assume new SpdySession will automatically be created when i don't return an existing session to FindAvailableSession(), and instead return base::WeakPtr<SpdySession>(). Is this correct?

I built the ran the browser and now I am able to see multiple connections to the server. However, there are 2 problems.

1) I have restricted the number of sessions in the set to 2 but there are more than 2 connections to server. I have noticed up to 4 connections.

2) The browser is unstable and crashes after 10 to 15 seconds.

Did i do something wrong? Or do i have to make changes to some other classes also that use AvailableSessionMap, which i might have missed?

Regards,

On 8 June 2017 at 01:10, Bence Béky <b...@chromium.org> wrote:

That is correct, you do not need to worry about it for the time being.

B

On Wed, Jun 7, 2017 at 6:54 PM, Jawad Manzoor <jawadm...@gmail.com> wrote:

Hi Bence,

This bug has not been fixed yet in the latest source code right? So I don't have to worry about it at the moment.

Regards,

From: Jawad Manzoor <jawadm...@gmail.com>

Date: Tue, Jun 6, 2017 at 7:25 PM
To: Bence Béky <b...@chromium.org>

I imagine that you are trying to use multiple HTTP/2 connections in parallel, with each requests assigned to one of the multiple connections at random, or to the one with the lowers number of active connections, or something like that.  Like when six HTTP/1.1 connections are used parallel.  Is that right?  Unfortunately this is currently not supported by Chromium.

Yes thats correct. I am trying to use multiple HTTP/2 connections in the same way as they are used in HTTP/1, just for some experiments.

I understand that its currently not supported in Chromium. I want to implement it myself and build it from source to test a few things related to my research. I discussed this with Mike Belshe a few months ago and he agreed that the single connection limit is too conservative and leads to poor performance in a number of different network conditions. That's why I want to test different number of connections.

Specifically, I want some pointers in the source code where the limit of a single HTTP/2 connection is implemented, so that I can change that limit to 2 or 4 connections. Could you point me to the relevant classes/functions in the source code?

Regards,

-

Jawad

----------
From: Bence Béky <b...@chromium.org>
Date: Tue, Jun 6, 2017 at 8:11 PM
To: Jawad Manzoor <jawadm...@gmail.com>

Hi Jawad,

Oh, sorry I didn't understand your request the first time.  I thought you were looking for a const int kMaxNumberOfConnections = 1; or similar statement.

An HTTP/2 connection is represented by a SpdySession object.  SpdySessionPool owns SpdySession objects, keyed by SpdySessionKey, at most one for each.  There's also some complicated alias mechanism for IP-based pooling.  HttpStreamFactoryImpl::Job tries to find an existing SpdySession to pool to, using SpdySessionPool::FindAvailableSession().  If not found, it requests a socket from the socket pool, which either creates a new socket or hands out an idle socket.  A socket is an open TLS connection with no HTTP/2 connection initialized on it, that is, no preface and initial SETTINGS sent (but HTTP/2 is already negotiated in the TLS handshake using ALPN).  Then HttpStreamFactoryImpl::Job hands this socket (held by the |connection_| member) to SpdySessionPool::CreateAvailableSessionFromSocket() which creates a new SpdySession on it, and makes SpdySessionPool own that SpdySession.

You will need to make sure that SpdySessionPool can handle multiple SpdySessions for each SpdySessionKey, make FindAvailableSession() pick one out of the multiple active ones for each request, and find a place to initiate creating new ones.  Sounds like a lot of work to me, let me know if you get stuck, I'll try to help.

BTW I do not know exactly why https://crbug.com/621597 is restricted, but it does not seem to have information relevant to your goal.

Also, if you are okay with it, I would like to move this discussion to the net...@chromium.org mailing list, so that other developers can also benefit from this discussion.

Cheers,

Bence

----------

From: Jawad Manzoor <jawadm...@gmail.com>

Date: Tue, Jun 6, 2017 at 10:16 PM
To: Bence Béky <b...@chromium.org>

Thanks a lot for the explanation Bence. I will let you know how it goes.

And yes you may take this discussion to the mailing list. It would be good to discuss it with other developers. 

Regards, 

On Tue, Jun 6, 2017 at 6:47 PM, Jawad Manzoor <jawadm...@gmail.com> wrote:

Hey.

I am doing research on HTTP/2 performance and I also noticed this behavior of parallel HTTP/2 connections. I quantified how often this happens using passive measurements and results are presented in this paper: http://ieeexplore.ieee.org/abstract/document/7818414/

Now I want to do some experiments by setting different number of parallel connections in HTTP/2 ( e.g, 2, 4 or 6). I was able to do it in Firefox and now I am trying with Chromium.

However, I am unable to locate the exact place in Chromium source code where the limit on HTTP/2 connections is implemented and further connections are prevented. I was hoping to get information from https://crbug.com/621597 but I do not have access to view it. I would appreciate any help on this.

--

On Monday, October 24, 2016 at 9:00:37 PM UTC+2, Bence Béky wrote:

This is related to https://crbug.com/621597.  In short, Chromium opens
up a number of TLS connections at the same time, and only finds out
that HTTP/2 is negotiated after a few roundtrips.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/d83a8fad-6882-4217-85c6-dd156baf2295%40chromium.org.

--

Jawad 

--

Jawad 

[Jawad Manzoor]

Hello Bence,

I was travelling in the past weeks and couldn't work on the problem. Now I have finally managed to implement the desired solution with 2 default TCP connections for each HTTP2 session. I did the performance comparison between single connection and 2 connections using the following mininet setup. A cloned facebook public page(3.5MB size, 125 objects) is hosted on H2O web server and fetched using both the standard chromium browser and the custom one that I have built.  A WiFi network is emulated between client and server with 2% random packet loss injected using netem. I have exported the HAR files for both tests and the results are shown in the attached images.

With a single connection, almost 20s are spent in loading the page and a large amount of time is spent in receiving phase.

With 2 connections everything seems to be loaded at around 4.3s which is a great improvement. However, the HAR file shows that onLoad() event is called at 15.76s. I do not see anything loading during this time. What could be the reason for onLoad() event being called so late?

Regards,

--

Jawad

[Bence Béky]

Hi Jawad,

This is really amazing, thank you for performing this experiment!  I wonder exactly where this speedup comes from.  I mean, even with two connections, since each resource is assigned to one of the connections and does not migrate to the other one if there is a packet loss, therefore ultimately each lost packet needs to be retransmitted anyway.  Is it just that packet losses slow down one connection, and the other one can transmit at a higher speed in the meanwhile?

I'm cc-ing Helen to this thread for heads up, she has done a lot of work recently on reducing the number of HTTP/2 connections (because Chrome currently only uses one anyway).

As for the onLoad() event call, I do not know what holds it back for so long.  Maybe you can try recording a network event log simultaneously, to see what network activity happens between 4.3 s and 15.76 s.  Let me know if you need help reading these logs.

Best wishes,

Bence

[Jawad Manzoor]

Hi,

Since I am injecting random packet losses/non-congestion losses, it is quite probable that one of the two connections slows down while the other stays unaffected. 

I checked the duration of connection with tcpdump and verified that timeline of the HAR file(exported by chrome-har-capturer) is not correct. The actual page load time with 2 connections was around 11s and not 4.3s. However the improvement is still significant compared to a single connection, which takes around 20s.

Regards,

--

Jawad