Any SPDY3 client send WINDOW_UPDATE frame?

[gw...@litespeedtech.com]

Hi,

We are adding SPDY support to OpenLiteSpeed Web server 1.1, it will support both SPDY2/3.

When we test SPDY3 against Chrome 27.0.1453.93, chrome send SETTINGS_INITIAL_WINDOW_SIZE=10M, but never send any WINDOW_UPDATE frame when download from server, if the file size is larger than 10MB, download will stall by server when windows size reach 0.

Is there an option to turn on SPDY3 flow control in chrome?

Thanks,

George Wang

[Simone Bordet]

Hi,

On Fri, Jun 7, 2013 at 10:56 PM, <gw...@litespeedtech.com> wrote:
> Hi,
>
> We are adding SPDY support to OpenLiteSpeed Web server 1.1, it will support
> both SPDY2/3.
>
> When we test SPDY3 against Chrome 27.0.1453.93, chrome send
> SETTINGS_INITIAL_WINDOW_SIZE=10M, but never send any WINDOW_UPDATE frame
> when download from server, if the file size is larger than 10MB, download
> will stall by server when windows size reach 0.

Really ? Seems a nasty bug.

> Is there an option to turn on SPDY3 flow control in chrome?

Jetty's SPDYClient fully supports flow control, you can use it in testing.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.

[William Chan (陈智昌)]

+akalin

I should be very surprised if this were actually true for Chrome. If you have a server we can test at, we can verify this.

--

---
You received this message because you are subscribed to the Google Groups "spdy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spdy-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Fred Akalin]

On Fri, Jun 7, 2013 at 4:16 PM, William Chan (陈智昌) <will...@chromium.org> wrote:

+akalin

I should be very surprised if this were actually true for Chrome. If you have a server we can test at, we can verify this.

On Fri, Jun 7, 2013 at 1:59 PM, Simone Bordet <sbo...@intalio.com> wrote:

Hi,

On Fri, Jun 7, 2013 at 10:56 PM,  <gw...@litespeedtech.com> wrote:
> Hi,
>
> We are adding SPDY support to OpenLiteSpeed Web server 1.1, it will support
> both SPDY2/3.
>
> When we test SPDY3 against Chrome 27.0.1453.93, chrome send
> SETTINGS_INITIAL_WINDOW_SIZE=10M, but never send any WINDOW_UPDATE frame
> when download from server, if the file size is larger than 10MB, download
> will stall by server when windows size reach 0.

Really ? Seems a nasty bug.

I'd be very interested if you can repro this with your server. Can you capture a dump with chrome://net-internals/ and send it to me?

You can see for yourself that Chrome sends WINDOW_UPDATE frames by going to the Chrome download page ( https://www.google.com/intl/en/chrome/browser/ ), opening up chrome://net-internals/ , then starting the download. Chrome is >40 MB, so if we weren't sending update frames, it would stall.

If you look in the logs for the SPDY session for dl.google.com:443, you can see when the WINDOW_UPDATE frame is sent:

t=1370650090115 [st=  1486]    SPDY_SESSION_RECV_DATA
                               --> fin = false
                               --> size = 1393
                               --> stream_id = 1
t=1370650090115 [st=  1486]    SPDY_SESSION_SENT_WINDOW_UPDATE_FRAME
                               --> delta = 5243351
                               --> stream_id = 1
t=1370650090115 [st=  1486]    SPDY_SESSION_RECV_DATA
                               --> fin = false
                               --> size = 2786
                               --> stream_id = 1

...

t=1370650091409 [st=  2780]    SPDY_SESSION_RECV_DATA
                               --> fin = false
                               --> size = 1393
                               --> stream_id = 1
t=1370650091409 [st=  2780]    SPDY_SESSION_SENT_WINDOW_UPDATE_FRAME
                               --> delta = 5244273
                               --> stream_id = 1
t=1370650091410 [st=  2781]    SPDY_SESSION_RECV_DATA
                               --> fin = false
                               --> size = 1393
                               --> stream_id = 1

The code sends a WINDOW_UPDATE frame when there are >= window_size / 2 bytes pending, which is about 5MB, and the log above is consistent with that.

> Is there an option to turn on SPDY3 flow control in chrome?

Jetty's SPDYClient fully supports flow control, you can use it in testing.

I don't think that was what Simone was asking -- Chrome already has flow control turned on as part of SPDY3, since it's part of the protocol. No need to turn it on.

[gw...@litespeedtech.com]

Chrome indeed send back WINDOW_UPDATE frame at about 5MB mark. 

There is some confusion about SETTINGS_INITIAL_WINDOW_SIZE, chrome send SETTING frame with SETTINGS_INITIAL_WINDOW_SIZE=10M, our server send SETTINGS_INITIAL_WINDOW_SIZE=640K when connection establishes, probably at the same time.

If our server follow chrome setting, it will work, if use our setting, it will stall large download. So, what is the correct behavior? SPDY3 spec is not very clear on this. 

I thought the windows size should be set by the sender, if not set, 64K is used. Each stream should track two WINDOWs, one for sending, one for receiving. For client upload, 10M is used, for download 640K is used. but looks like it not the case, we have to follow setting set by client. What is the effect on client side for SETTINGS_INITIAL_WINDOW_SIZE=640K by the server?

Or, the stream created by client, uses client's setting, server pushed stream created, uses server's setting? 

BTW: Firefox and Opera all send WINDOW_UPDATE frame more aggressively. 

Thanks,

George

[Patrick McManus]

acsa

On Mon, Jun 10, 2013 at 3:12 PM, <gw...@litespeedtech.com> wrote:

There is some confusion about SETTINGS_INITIAL_WINDOW_SIZE, chrome send SETTING frame with SETTINGS_INITIAL_WINDOW_SIZE=10M,

That means the server is allowed to send 10MB of data before receiving any window updates.

 

our server send SETTINGS_INITIAL_WINDOW_SIZE=640K when connection establishes, probably at the same time.

That means the client is only allowed to send 640KB of data before receiving any window updates.

 

So, what is the correct behavior?

In the case of no window updates, the server should send up to 10MB of data before stalling on an empty flow control window. There is not a race condition here as the client SETTINGS frame is sent before the client SYN_STREAM that is being replied to with the data.

In the case of no window updates, the client should send up to 640KB of data before stalling on an empty flow control window. There is a race condition here because the DATA frames for the stream could be sent before the 640KB setting arrives. in that case the client would stall at 64KB and resume sending when the 640KB setting arrived.

Obviously any receiver (the client or the server) should generate window_updates when its buffering situation looks good and in a manner timely enough that the other end will receive the update before the window becomes empty. It appears chrome thinks that 5MB is enough buffering to make that happen (sounds like plenty to me too).

BTW: Firefox

Assuming normal buffering, Firefox will generate WINDOW_UPDATEs after 64KB of data is received (using a 256MB window). That's probably too aggressive in some ways, but

1] they're quite small and are piggybacked onto TCP acks that are sent more frequently anyhow

2] If a sending implementation is buggy and only uses the default 64KB window, this is will at least keep the session alive.

its not better or worse than chrome. just different.

hth.

-P

[gw...@litespeedtech.com]

In the case of no window updates, the server should send up to 10MB of data before stalling on an empty flow control window. There is not a race condition here as the client SETTINGS frame is sent before the client SYN_STREAM that is being replied to with the data.

In the case of no window updates, the client should send up to 640KB of data before stalling on an empty flow control window. There is a race condition here because the DATA frames for the stream could be sent before the 640KB setting arrives. in that case the client would stall at 64KB and resume sending when the 640KB setting arrived.

OK. clear now. The receiving side decides the window size.

 

Obviously any receiver (the client or the server) should generate window_updates when its buffering situation looks good and in a manner timely enough that the other end will receive the update before the window becomes empty. It appears chrome thinks that 5MB is enough buffering to make that happen (sounds like plenty to me too).

 

As long as both sides use the same window size, it will not cause problem. 

 

Assuming normal buffering, Firefox will generate WINDOW_UPDATEs after 64KB of data is received (using a 256MB window). That's probably too aggressive in some ways, but

1] they're quite small and are piggybacked onto TCP acks that are sent more frequently anyhow

2] If a sending implementation is buggy and only uses the default 64KB window, this is will at least keep the session alive.

its not better or worse than chrome. just different.

Agree. actually, prefer chrome's method. However, in case dealing with a buggy client, we will send out WINDOW_UPDATE once stall is detected. 

Thanks,

George

[Roberto Peon]

Hmm.

I wonder if we should provide a GOAWAY error code that means try again but not using SPDY/HTTP/2, etc.

I'd rather have that than hacks w.r.t.  WINDOW_UPDATE for buggy clients.

(Buggy clients should be told go away and use the alternative so that they get fixed!)

-=R

[gw...@litespeedtech.com]

I doubt a buggy client will care about the GOAWAY error code. :-)

[Fred Akalin]

Won't this be possible only if the client lets the server know it's stalled (like in SPDY/4)? Otherwise, how will the server know that the client has a smaller window than it should?

[gw...@litespeedtech.com]

server does not have to know if it is really stalled.

If not receive any data from client as expected after a timeout, send WINDOW_UPDATE to reset the window the initial size. optionally, mark this client as NOT_FLOWCONTROL_FRIENDLY and send WINDOW_UPDATE immediately when any data has been received in the future. Also, can use a PING frame to make sure the connection is still live, but I think it does not hurt to reset the window anyway.

George 

[Fred Akalin]

The server should not have to work around buggy clients like this.

[William Chan (陈智昌)]

Please do not workaround buggy clients. In the development of a new protocol, it's nice if people are strict about interoperability and break peers who misbehave. Let's not let people get away with sloppy implementations which we need to support long-term.

[Mike Belshe]

On Mon, Jun 10, 2013 at 8:49 PM, William Chan (陈智昌) <will...@chromium.org> wrote:

Please do not workaround buggy clients. In the development of a new protocol, it's nice if people are strict about interoperability and break peers who misbehave. Let's not let people get away with sloppy implementations which we need to support long-term.

+1.  If anything, proactively break them.

Mike

[gw...@litespeedtech.com]

I agree with the principle to let bad client break, so, we have not done any workaround yet.

Last week, we released OpenLiteSpeed 1.1 with SPDY 2/3 support, it should be the first high performance web server that speaks SPDY/3. 

We cut some corners on HEADER and CREDENTIAL frame processing, seems those are not used when serving HTTP traffic. Server push has not been utilized, may add it later if receive requests on this. :-)

George Wang