Thinking about flow control

644 views
Skip to first unread message

Roberto Peon

unread,
May 17, 2012, 4:44:36 PM5/17/12
to spdy...@googlegroups.com
All--

We've been having a number of private conversations about flow control.
The flow control that has been deployed in SPDY/3 seems to work, but it has some definite flaws and suboptimalities.
Basically, it only allows for per-stream flow control.

Per-stream flow control is certainly necessary; without it we'd have head-of-line blocking or infinite-buffering requirements at proxies, which is untenable.
On its, own, however, if we are still worried about infinite buffering and wish to increase the stream limit, per-stream flow control isn't enough. 

So, we probably need per-connection flow-control so that we can manage the total amount of buffering necessary, while still allowing for large enough per-stream flow-control windows to ensure that we can at least fill up the TCP connection.


We're hoping to get comments in over the next two weeks (at most), and then begin spec revisions and possibly experimentation with new flow control in (and for) SPDY/4.

So... comments? Ideas?
-=R

Adam Langley

unread,
May 17, 2012, 4:47:38 PM5/17/12
to spdy...@googlegroups.com
On Thu, May 17, 2012 at 4:44 PM, Roberto Peon <fe...@google.com> wrote:
> So, we probably need per-connection flow-control so that we can manage the
> total amount of buffering necessary

Why is TCP's per-connection flow control insufficient?


Cheers

AGL

Patrick McManus

unread,
May 17, 2012, 4:54:33 PM5/17/12
to spdy...@googlegroups.com
one reason is spdy flow control only covers data frames.. so a server
could protect itself from too much concurrent upload without impacting
its ability to serve get/etc..



Patrick McManus

unread,
May 17, 2012, 5:19:13 PM5/17/12
to spdy...@googlegroups.com
On Thu, 2012-05-17 at 13:44 -0700, Roberto Peon wrote:

>
>
> We're hoping to get comments in over the next two weeks (at most), and
> then begin spec revisions and possibly experimentation with new flow
> control in (and for) SPDY/4.
>
>
> So... comments? Ideas?
> -=R

I'm certainly in favor of doing something as the new flow control params
as used with the small windows currently on google.com definitely slow
down certain scenarios. If an aggregate session window allows the server
to safely use larger per-stream windows then that's a good fix.

I'd suggest that the new global window can both shrink and expand
(presumably via an explicit settings frame analogous to initial window),
and that per stream window_updates be changed to allow shrinking of the
window not just expanding it.

use case for that - its hard to know what the BDP really is, so if you
want fully streamed transfer its tempting to set the window extremely
large so spdy flow control is not the choke point.. if for some reason
the local data sink stops accepting data temporarily you can slam the
window shut with a negative delta and only actually end up buffering
~BDP of data instead of "extremely large".

-Patrick

Roberto Peon

unread,
May 17, 2012, 5:27:40 PM5/17/12
to spdy...@googlegroups.com
Agreed. In all cases where we overestimate, we may cause HOL blocking, but that is always true so long as we're on top of TCP.

One other thing that I was thinking about was that we could have default window sizes which differed based on the stream priority.
We'd react in hopefully one-RTT, so the question there is: Is it worth the additional complexity given the addition of the connection-based flow-control?

-=R
 

-Patrick


Tatsuhiro Tsujikawa

unread,
May 17, 2012, 9:35:41 PM5/17/12
to spdy...@googlegroups.com


2012/05/18 6:19 "Patrick McManus" <mcm...@ducksong.com>:

Do you mean use only per-connection flow control?
Then +1.
It achieves the control of total number of buffering, which is the very issue the flow control was introduced.

Patrick McManus

unread,
May 18, 2012, 7:54:30 AM5/18/12
to spdy...@googlegroups.com

> Do you mean use only per-connection flow control?
> Then +1.
> It achieves the control of total number of buffering, which is the
> very issue the flow control was introduced.
>

No - I'm suggesting we need both per-session and per-stream flow
control.

We need per stream flow control because different data sinks in the set
of multiplexed streams may consume data at a different rate, and you
want to be able to stop buffering one without impacting the other.

On the client side this could be a plugin vs a core rendering component
- or even a media component that has had the pause button pushed. A
proxy is obviously connected in diverse ways (with diverse bandwidth) on
the back side while multiplexing on the front, and an origin server
might be multiplexing 1 receiver that is writing to a big data warehouse
and one that is using its input for lookup in a light in memory hash.

If we add a session wide window - then we can do this stream control
with the windowing we have, or honestly we could do it with per stream
txon/txoff and that might lead to better quality implementations.


Simone Bordet

unread,
May 18, 2012, 8:12:08 AM5/18/12
to spdy...@googlegroups.com
Hi,

On Thu, May 17, 2012 at 10:44 PM, Roberto Peon <fe...@google.com> wrote:
> All--
>
> We've been having a number of private conversations about flow control.
> The flow control that has been deployed in SPDY/3 seems to work, but it has
> some definite flaws and suboptimalities.
> Basically, it only allows for per-stream flow control.
>
> Per-stream flow control is certainly necessary; without it we'd have
> head-of-line blocking or infinite-buffering requirements at proxies, which
> is untenable.

Yes.

> On its, own, however, if we are still worried about infinite buffering and
> wish to increase the stream limit, per-stream flow control isn't enough.

I am not sure I understand this paragraph. Can you expand ?

The current SETTINGS for the initial window size, when received,
applies to all active streams, so it's already more at the connection
level rather than at the stream level (i.e. you cannot update the
initial window size of only one stream).

Thanks,

Simon
--
http://cometd.org
http://intalio.com
http://bordet.blogspot.com
----
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

Peter Lepeska

unread,
May 18, 2012, 9:39:20 AM5/18/12
to spdy...@googlegroups.com
"Essentially, the proxy needs a way to advertise the amount of room it has in its outgoing buffers, which it can't easily do with TCP's flow-control."

Yes it can. When a proxy server's buffer grows to a certain threshold, it just stops posting buffers to the receive socket connected to the web server. When this happens the TCP stack buffer fills up (since no packets are being pulled off by the user mode process), and TCP starts sending smaller receive windows in its ACKs to the web server, which slows down its send rate. This seems fine to me.

I'm still not understanding what problem we are trying to solve by adding another layer of flow control to the TCP session.

Peter

On Thu, May 17, 2012 at 5:00 PM, Roberto Peon <fe...@google.com> wrote:
TCP's flow-control alone would allow HOL blocking to occur in cases where the proxy->server connections were full for only some streams because the userspace process of the server doesn't control advertisements of its ingress window.
Essentially, the proxy needs a way to advertise the amount of room it has in its outgoing buffers, which it can't easily do with TCP's flow-control.

-=R

Ryan Hamilton

unread,
May 18, 2012, 11:34:50 AM5/18/12
to spdy...@googlegroups.com
On Fri, May 18, 2012 at 6:39 AM, Peter Lepeska <bizzb...@gmail.com> wrote:
"Essentially, the proxy needs a way to advertise the amount of room it has in its outgoing buffers, which it can't easily do with TCP's flow-control."

Yes it can. When a proxy server's buffer grows to a certain threshold, it just stops posting buffers to the receive socket connected to the web server. When this happens the TCP stack buffer fills up (since no packets are being pulled off by the user mode process), and TCP starts sending smaller receive windows in its ACKs to the web server, which slows down its send rate. This seems fine to me.

I'm still not understanding what problem we are trying to solve by adding another layer of flow control to the TCP session.

Imagine a browser has a SPDY session open to a proxy.  In this session, imagine that there is a large POST upload in progress.  Further, imagine that the server that the proxy needs to relay the POST data to is slower than than client.  At this point the proxy needs to either buffer indefinitely (which clearly does not scale) or it needs to pause the upload.  If the only flow control knob at the proxy's disposal is at the TCP level, it can clearly pause the entire session.  However, this will prevent any other streams from making any  progress.  You might be surprised how often this situation comes up.

You can also spin the example around, as Patrick did earlier, and imagine that the server is using a stream to send streaming media to the client...  Same potential problem.

Cheers,

Ryan

Simone Bordet

unread,
May 18, 2012, 12:25:17 PM5/18/12
to spdy...@googlegroups.com
Hi,

On Fri, May 18, 2012 at 5:34 PM, Ryan Hamilton <r...@google.com> wrote:
>
>
> On Fri, May 18, 2012 at 6:39 AM, Peter Lepeska <bizzb...@gmail.com> wrote:
>>
>> "Essentially, the proxy needs a way to advertise the amount of room it has
>> in its outgoing buffers, which it can't easily do with TCP's flow-control."
>>
>> Yes it can. When a proxy server's buffer grows to a certain threshold, it
>> just stops posting buffers to the receive socket connected to the web
>> server. When this happens the TCP stack buffer fills up (since no packets
>> are being pulled off by the user mode process), and TCP starts sending
>> smaller receive windows in its ACKs to the web server, which slows down its
>> send rate. This seems fine to me.
>>
>> I'm still not understanding what problem we are trying to solve by adding
>> another layer of flow control to the TCP session.
>
>
> Imagine a browser has a SPDY session open to a proxy.  In this session,
> imagine that there is a large POST upload in progress.  Further, imagine
> that the server that the proxy needs to relay the POST data to is slower
> than than client.  At this point the proxy needs to either buffer
> indefinitely (which clearly does not scale) or it needs to pause the upload.
>  If the only flow control knob at the proxy's disposal is at the TCP level,
> it can clearly pause the entire session.  However, this will prevent any
> other streams from making any  progress.  You might be surprised how often
> this situation comes up.

I do not follow. The proxy should flow-control the browser.

> You can also spin the example around, as Patrick did earlier, and imagine
> that the server is using a stream to send streaming media to the client...
>  Same potential problem.

Same solution: the proxy flow-controls the server.

Back to upload example: browser sends a window-sized data frame, and
can't send more until it gets a window update.
Proxy receives the data frame, forwards to upstream server.
If the server is SPDY aware, the proxy will get a window update, and
upon receiving that, it sends the window update to the browser, which
will send another data frame.
If the server is not SPDY aware, the proxy writes to it and only when
the write is fully completed (to leverage TCP backpressure), then the
proxy sends a window update to the browser.

Similarly for server to client.

I feel I am missing something, but can't see it ?

Ryan Hamilton

unread,
May 18, 2012, 12:33:22 PM5/18/12
to spdy...@googlegroups.com
Perhaps I was confused.  I was responding to your question:

> I'm still not understanding what problem we are trying to solve by adding
> another layer of flow control to the TCP session.

To do this, I presented two situations where a browser talking to a proxy would want to stop a single stream in a SPDY session but allow other SPDY streams to progress.  TCP-only flow control does not provide this facility.  However, you replied to my message with this:

> If the server is SPDY aware, the proxy will get a window update, and
> upon receiving that, it sends the window update to the browser, which
> will send another data frame.
> If the server is not SPDY aware, the proxy writes to it and only when
> the write is fully completed (to leverage TCP backpressure), then the
> proxy sends a window update to the browser.

Both of these examples include WINDOW_UPDATE frames which are, of course, at the SPDY layer, not the TCP layer.  I think I don't understand the question you are asking?  Do you agree with the need for a particular stream in a SPDY session to be paused which others can proceed?

Cheers,

Ryan

 

Simone Bordet

unread,
May 18, 2012, 12:44:06 PM5/18/12
to spdy...@googlegroups.com
Hi,

On Fri, May 18, 2012 at 6:33 PM, Ryan Hamilton <r...@google.com> wrote:
> Perhaps I was confused.  I was responding to your question:
>
>> I'm still not understanding what problem we are trying to solve by adding
>> another layer of flow control to the TCP session.

Was not me, was Peter :)

> To do this, I presented two situations where a browser talking to a proxy
> would want to stop a single stream in a SPDY session but allow other SPDY
> streams to progress.  TCP-only flow control does not provide this facility.

Sure.

>  However, you replied to my message with this:
>
>> If the server is SPDY aware, the proxy will get a window update, and
>> upon receiving that, it sends the window update to the browser, which
>> will send another data frame.
>> If the server is not SPDY aware, the proxy writes to it and only when
>> the write is fully completed (to leverage TCP backpressure), then the
>> proxy sends a window update to the browser.
>
> Both of these examples include WINDOW_UPDATE frames which are, of course, at
> the SPDY layer, not the TCP layer.  I think I don't understand the question
> you are asking?  Do you agree with the need for a particular stream in a
> SPDY session to be paused which others can proceed?

I agree, and that's what the current flow-control is for.

I do not clearly see the case for a SPDY session level flow control,
in addition to SPDY stream level flow control.
See also my other email in reply to Roberto asking for more details.

We just want be part of the loop: we can't comment if we don't
understand the use case :)

Thanks,

Ryan Hamilton

unread,
May 18, 2012, 1:05:53 PM5/18/12
to spdy...@googlegroups.com
On Fri, May 18, 2012 at 9:44 AM, Simone Bordet <sbo...@intalio.com> wrote:
Hi,

On Fri, May 18, 2012 at 6:33 PM, Ryan Hamilton <r...@google.com> wrote:
> Perhaps I was confused.  I was responding to your question:
>
>> I'm still not understanding what problem we are trying to solve by adding
>> another layer of flow control to the TCP session.

Was not me, was Peter :)

Doh!  Well, that explains why I was confused :>  Sorry about that.  I'll let the other comment further on the session-level flow control debate.

Cheers,

Ryan

Patrick McManus

unread,
May 18, 2012, 1:09:32 PM5/18/12
to spdy...@googlegroups.com
On Fri, 2012-05-18 at 18:44 +0200, Simone Bordet wrote:

> I do not clearly see the case for a SPDY session level flow control,
> in addition to SPDY stream level flow control.

My understanding is that giving every stream a window equal to
session-buffers-available represents a scary overcommitment for the
server if every stream decided to utilize it at the same time. It also
creates an unwelcome incentive to minimize the number of parallel
streams.

Likewise, dividing the session buffers available into small shares for
each stream often results in some streams easily running out of window
space while other streams waste their allocations.

You'll see that google.com has been using small 12KB initial windows -
which are too small to allow full rate uploads for many common BDPs. The
flow control proposal arises out of the need to fix that bottleneck
while still informing the client of the server's available bufferspace
(which is presumably > 12KB).

Having 2 levels of flow control lets you separate the concerns: the
session value is about total buffers available, the per-stream value is
about letting different streams proceed at different rates (and that's
why I think it can be done with xon/xoff in the presence of a session
window).

-Patrick

Tatsuhiro Tsujikawa

unread,
May 18, 2012, 1:14:25 PM5/18/12
to spdy...@googlegroups.com
Thank you for answering.
I understand that stream sink has different consume rate.
Then we need a way to tell the other endpoint the initial window size
for each stream, no?
In the SPDY/3 spec, SETTINGS frame sets initial window size for all
streams in a session.
The one good point of introducing per-session flow control is that we
can use larger
initial window size for stream so that flow control does not slow down
the transfer.
If the endpoint know the particular stream sink is slow, then it has
to tell the other endpoint about it by shrinking window size for it.

Ryan Hamilton

unread,
May 18, 2012, 1:17:37 PM5/18/12
to spdy...@googlegroups.com
Right, that is what the WINDOW_UPDATE frame is for.  It changes the window size for a particular stream.

Cheers,

Ryan

Tatsuhiro Tsujikawa

unread,
May 18, 2012, 1:25:05 PM5/18/12
to spdy...@googlegroups.com
But WINDOW_UPDATE delta-window-size only allows positive value:

Delta-Window-Size: The additional number of bytes that the sender can
transmit in addition to existing remaining window size. The legal
range for this field is 1 to 2^31 - 1 (0x7fffffff) bytes.

My concern is that in SPDY/3, all streams have the same initial window
size, for example, 64KB.
We can increase window size by WINDOW_UPDATE (I'm not sure we can exceed initial
window size by sending WINDOW_UPDATE), but we cannot decrease window
size by that frame.

Tatsuhiro

> Cheers,
>
> Ryan
>

William Chan (陈智昌)

unread,
May 18, 2012, 1:26:39 PM5/18/12
to spdy...@googlegroups.com
There is no negative delta currently in the SPDY/3 spec:
"Delta-Window-Size: The additional number of bytes that the sender can transmit in addition to existing remaining window size. The legal range for this field is 1 to 2^31 - 1 (0x7fffffff) bytes."
 

-Patrick


Mike Belshe

unread,
May 18, 2012, 1:30:41 PM5/18/12
to spdy...@googlegroups.com
Is there data that accompanies this claim?

I'm mostly siding with Langley's question - that TCP's flow control should work for the overall layer.  Its simple and should suffice just fine.  It sounds like you're trying to flow control between multiple systems now.  (e.g. client -> proxy -> backend server).  This is a mistake - in the end, this will be a mistake - its over-optimizing for the proxy at the expense of the endpoints, because the only way to do this is to add round trips of latency onto the user.

I'll again caution against thinking flow control is easy.  It sounds easy, but its complicated with really subtle implications.

Mike

William Chan (陈智昌)

unread,
May 18, 2012, 1:32:32 PM5/18/12
to spdy...@googlegroups.com
Just to clarify, you don't have to *shrink* the window size per-se, but just not send more WINDOW_UPDATEs, or send ones with smaller deltas.

Patrick McManus

unread,
May 18, 2012, 1:37:49 PM5/18/12
to spdy...@googlegroups.com
On Fri, 2012-05-18 at 10:26 -0700, William Chan (陈智昌) wrote:
>
> I'd suggest that the new global window can both shrink and
> expand
> (presumably via an explicit settings frame analogous to
> initial window),
> and that per stream window_updates be changed to allow
> shrinking of the
> window not just expanding it.
>
> use case for that - its hard to know what the BDP really is,
> so if you
> want fully streamed transfer its tempting to set the window
> extremely
> large so spdy flow control is not the choke point.. if for
> some reason
> the local data sink stops accepting data temporarily you can
> slam the
> window shut with a negative delta and only actually end up
> buffering
> ~BDP of data instead of "extremely large".
>
>
> There is no negative delta currently in the SPDY/3 spec:
> "Delta-Window-Size: The additional number of bytes that the sender can
> transmit in addition to existing remaining window size. The legal
> range for this field is 1 to 2^31 - 1 (0x7fffffff) bytes."
>

right! that's why I said.. "window_update be changed to allow
shrinking".. :)

But I'm now thinking that, assuming a new per-session window, the
per-stream windows can be replaced with simpler per-stream xon/xoff
notifications. You don't have to worry about sizing those to network
conditions, so it will be more robust.





Mike Belshe

unread,
May 18, 2012, 1:38:40 PM5/18/12
to spdy...@googlegroups.com
On Fri, May 18, 2012 at 10:09 AM, Patrick McManus <mcm...@ducksong.com> wrote:
On Fri, 2012-05-18 at 18:44 +0200, Simone Bordet wrote:

> I do not clearly see the case for a SPDY session level flow control,
> in addition to SPDY stream level flow control.

My understanding is that giving every stream a window equal to
session-buffers-available represents a scary overcommitment for the
server if every stream decided to utilize it at the same time. It also
creates an unwelcome incentive to minimize the number of parallel
streams.

Likewise, dividing the session buffers available into small shares for
each stream often results in some streams easily running out of window
space while other streams waste their allocations.

You'll see that google.com has been using small 12KB initial windows -
which are too small to allow full rate uploads for many common BDPs. The
flow control proposal arises out of the need to fix that bottleneck
while still informing the client of the server's available bufferspace
(which is presumably > 12KB).

The 64KB default is supposed to make flow control so rare that it almost never gets in the way while also providing a reasonable backstop.

One thing I hate about SPDY flow control is it gives too much control to the proxy (similar to how we don't want proxies to be able to turn off compression at the user's expense).   I hope Chrome and Firefox implement a minimum window size clamp of 32KB or something to prevent these shenanigans.

Mike

William Chan (陈智昌)

unread,
May 18, 2012, 1:39:34 PM5/18/12
to spdy...@googlegroups.com
Does this work? How do you know they've received the xoff notification? Do we need sequence numbers in streams? How do you detect a broken peer so you can send them an error?
 

Patrick McManus

unread,
May 18, 2012, 1:53:47 PM5/18/12
to spdy...@googlegroups.com
On Fri, 2012-05-18 at 10:38 -0700, Mike Belshe wrote:

>
> The 64KB default is supposed to make flow control so rare that it
> almost never gets in the way while also providing a reasonable
> backstop.
>

it's hard to square that rarity at 64KB with the entire presence of the
TCP window scaling option which is needed past the same threshold.

just 8mbit/sec at 100ms of latency needs 100KB of BDP. That's hardly a
corner case, right?




Patrick McManus

unread,
May 18, 2012, 2:01:18 PM5/18/12
to spdy...@googlegroups.com
On Fri, 2012-05-18 at 10:39 -0700, William Chan (陈智昌) wrote:

>
> Does this work? How do you know they've received the xoff
> notification?

you can't verify it, other than at some extreme point as the total
session window provides the real backstop. but when do you really care?

do you really have a partitioned set of resources available to some
streams but not others (in which case I agree, we need per stream
windows), or do you just want to pushback and stop the sending asap (in
which case xoff ought to be sufficient).

For the most part the resource in question is ram and ram is mostly
fungible.



William Chan (陈智昌)

unread,
May 18, 2012, 2:01:41 PM5/18/12
to spdy...@googlegroups.com
One thing that Roberto neglected to mention from our private conversations is also trying to improve latency, and to what degree flow control is useful here. Let me call your attention to a few points:

* SPDY/3 is faster than SPDY/2 for *some* webpages - Google has data on some of our properties to support this. We don't have data (yet) on cases where it is hurting our page load times. But don't take that to mean that that could not happen.
  - It turns out that you may incur bufferbloat by returning responses too quickly. SPDY/3 (as a side-effect) helps fight the problem due to per-stream flow control windows that restrain streams. Then if you get a high priority resource request later (let's say there's a script at the bottom of a document behind a gazillion large images), it won't be stuck behind all the other data in the bloated buffers.
  - Obviously, if the flow control windows are too small, then we're not filling the pipe as quickly as we could, and thus flow control can make certain pages *slower*.
* We don't have data on what percent of the web would see improvements with the current 64k window size, and what percent would have no impact, and what percent would get slower. It'd be interesting to see.
* Note that the original goal for flow control was to better manage buffers (rather than rely on TCP flow control, which would lead to HoL blocking), not prevent this bufferbloat.
* Note that just because there is space in the peer's per-stream rwin, it does not mean that a sender *has* to use that space. It theoretically could self-throttle if it expects that there is bufferbloat (maybe examine inter-packet arrival times?).
* But if the receiver believes the sender has a naive implementation, perhaps it's better to force the sender not to cause bufferbloat by constraining it with smaller window sizes via SETTINGS for per-stream window sizes. Or, perhaps better would be to do so via theoretical SPDY/4 per-session flow control window sizes.
* Also note that individual streams *can* have different rwins. You control them individually via WINDOW_UPDATE deltas. Why would you want to do this? How about long-lived streams vs short-lived streams (the current common case). Long-lived streams like big downloads/uploads or tunneling (SPDY over SPDY for example) may require different window sizes.
* These considerations all become more complicated given networks with higher variance in relevant characteristics, like mobile networks.

Anyway, I just wanted to point out that the implementation choices are complicated. It'd be great to hear discussion from folks (and more data if possible) about what good implementations *should* do.

William Chan (陈智昌)

unread,
May 18, 2012, 2:11:06 PM5/18/12
to spdy...@googlegroups.com
On Fri, May 18, 2012 at 11:01 AM, Patrick McManus <mcm...@ducksong.com> wrote:
On Fri, 2012-05-18 at 10:39 -0700, William Chan (陈智昌) wrote:

>
> Does this work? How do you know they've received the xoff
> notification?

you can't verify it, other than at some extreme point as the total
session window provides the real backstop. but when do you really care?

do you really have a partitioned set of resources available to some
streams but not others (in which case I agree, we need per stream
windows), or do you just want to pushback and stop the sending asap (in
which case xoff ought to be sufficient).

I'm not exactly clear on what you mean by partitioned set of resources available to some streams but not others, but if I understand correctly, then yes, I suspect that this can happen, in particular with proxies. If you don't agree, then perhaps I need more clarity on what you mean.

Patrick McManus

unread,
May 18, 2012, 2:18:47 PM5/18/12
to spdy...@googlegroups.com
On Fri, 2012-05-18 at 11:11 -0700, William Chan (陈智昌) wrote:

>
> I'm not exactly clear on what you mean by partitioned set of resources
> available to some streams but not others, but if I understand
> correctly, then yes, I suspect that this can happen, in particular
> with proxies. If you don't agree, then perhaps I need more clarity on
> what you mean.
>

>

you have a session window of N for M streams.

do you really care if 1 stream consumes more than 1/Mth of that, as long
as the session doesn't exceed N? (or even 1/2 or whatever..) I do
understand the need to give feedback to individual streams that they
shouldn't send because that data won't be acked right away (thus xoff).

from my pov the peer has been given a quota of N, how they want to split
that up can be up to them.

I don't hate per stream windows, its just a knob that's hard to set
right so if it can be removed that's a good thing. Maybe it can't be
removed :)

Peter Lepeska

unread,
May 18, 2012, 2:35:09 PM5/18/12
to spdy...@googlegroups.com
Hi Ryan,

I understand why SPDY needs to do per stream flow control. That part makes sense to me and is well illustrated by your example. But I thought this thread was about adding another layer of flow control to the TCP session itself. That part seems redundant with what TCP is already doing.

Thanks,

Peter

Simone Bordet

unread,
May 19, 2012, 1:29:43 PM5/19/12
to spdy...@googlegroups.com
Hi,

On Fri, May 18, 2012 at 8:18 PM, Patrick McManus <mcm...@ducksong.com> wrote:
> you have a session window of N for M streams.
>
> do you really care if 1 stream consumes more than 1/Mth of that, as long
> as the session doesn't exceed N? (or even 1/2 or whatever..) I do
> understand the need to give feedback to individual streams that they
> shouldn't send because that data won't be acked right away (thus xoff).
>
> from my pov the peer has been given a quota of N, how they want to split
> that up can be up to them.
>
> I don't hate per stream windows, its just a knob that's hard to set
> right so if it can be removed that's a good thing. Maybe it can't be
> removed :)

What do you suggest to avoid that one stream that takes 90% of N does
not starve other streams (e.g. I am streaming a movie, but it's
boring, so I want to do a little browsing meanwhile) ?
Also, you seem to suggest a mechanism to increase the per-stream
window (that starts at a default < N, but eventually may increase up
to N), right ?
While for decreasing the per-stream window you suggest negative window updates.

I am worried about the fact that there may be the need for some
inter-stream communication to adjust the per-stream window size (e.g.
stream1 takes most of N, but as soon as another transfer is requested,
stream1 needs to be shrunk in order to give some room to stream2), but
perhaps I am not understanding it right.

Patrick McManus

unread,
May 19, 2012, 9:30:33 PM5/19/12
to spdy...@googlegroups.com, Simone Bordet
On 5/19/2012 1:29 PM, Simone Bordet wrote:
> Hi,
>
> On Fri, May 18, 2012 at 8:18 PM, Patrick McManus<mcm...@ducksong.com> wrote:
>> you have a session window of N for M streams.
>>
>> do you really care if 1 stream consumes more than 1/Mth of that, as long
>> as the session doesn't exceed N? (or even 1/2 or whatever..) I do
>> understand the need to give feedback to individual streams that they
>> shouldn't send because that data won't be acked right away (thus xoff).
>>
>> from my pov the peer has been given a quota of N, how they want to split
>> that up can be up to them.
>>
>> I don't hate per stream windows, its just a knob that's hard to set
>> right so if it can be removed that's a good thing. Maybe it can't be
>> removed :)
> What do you suggest to avoid that one stream that takes 90% of N does
> not starve other streams (e.g. I am streaming a movie, but it's
> boring, so I want to do a little browsing meanwhile) ?

If you send tx-off when you pause the movie then you'll end up buffering
BDP. (or less if it isn't sending at line rate).

If, instead of txon/txoff, you had a per-stream-window smaller than BDP
then the stream can never flow at line rate, almost by definition. And
you would of course buffer less for that one stream. I don't think the
risk of accidentally constraining the transfer rate is worth that, but
it would be the argument in favor of per stream windows.

If, instead of txon/txoff, you had a per-stream window greater than BDP
the stream would buffer more than BDP before it was stopped. This could
be mitigated down to BDP if we had negative deltas.

BDP is very hard to know - so its hard to make the per stream window
size choice accurately as an implementor and that lets people
accidentally slow things down without a lot of insight into why. I've
definitely already seen this once with spdy/3 and I suspect the more
chances there are to select window sizes the more often they will be
selected too small.

Simone Bordet

unread,
May 21, 2012, 11:14:56 AM5/21/12
to spdy...@googlegroups.com
Hi,

On Sun, May 20, 2012 at 3:30 AM, Patrick McManus <mcm...@ducksong.com> wrote:
> If you send tx-off when you pause the movie then you'll end up buffering
> BDP. (or less if it isn't sending at line rate).

But I am not pausing the movie.
I am watching the movie, but meanwhile I start a big download, for example.
How do you transfer part of the session window taken by the movie
stream to the download stream ?

> If, instead of txon/txoff, you had a per-stream-window smaller than BDP then
> the stream can never flow at line rate, almost by definition. And you would
> of course buffer less for that one stream. I don't think the risk of
> accidentally constraining the transfer rate is worth that, but it would be
> the argument in favor of per stream windows.
>
> If, instead of txon/txoff, you had a per-stream window greater than BDP the
> stream would buffer more than BDP before it was stopped. This could be
> mitigated down to BDP if we had negative deltas.
>
> BDP is very hard to know - so its hard to make the per stream window size
> choice accurately as an implementor and that lets people accidentally slow
> things down without a lot of insight into why.  I've definitely already seen
> this once with spdy/3 and I suspect the more chances there are to select
> window sizes the more often they will be selected too small.

Not sure you offered a solution ? :)

Just to play the devil's advocate here: are we not reinventing TCP on
top of SPDY ?
With all the problems it carries (e.g. bufferbloat), and additional
ones (how to share bandwidth among streams) ?

Can we get a clearer explanation of what Google found in its measurement ?

Thanks !

Patrick McManus

unread,
May 21, 2012, 11:44:56 AM5/21/12
to spdy...@googlegroups.com
On Mon, 2012-05-21 at 17:14 +0200, Simone Bordet wrote:

> But I am not pausing the movie.
> I am watching the movie, but meanwhile I start a big download, for example.
> How do you transfer part of the session window taken by the movie
> stream to the download stream ?
>

I don't think that's a great use of flow control. imo flow control is
about not overrunning the consumer's buffers. Your scenario is using it
to try and manage bandwidth sharing (you don't need more buffers to
support the different streams as the aggregate bandwidth hasn't
changed). I don't think it should manage stream rates - but if the group
thinks it should then I agree txon/txoff ain't going to fill the bill. I
think priorities are a better approach to bandwidth issues and spdy
needs to make them dynamically manageable.

> Just to play the devil's advocate here: are we not reinventing TCP on
> top of SPDY ?

to some degree (hopefully a minor one) the multiplexing demands this.
Definitely a price to be paid for the benefit of being able to actually
improve the web today. (instead of doing a sctp like approach)

> Can we get a clearer explanation of what Google found in its measurement ?
>

google can share their own data points, but I can attest that a gmail
attachment over spdy/3 against a 12KB per-stream window can be 3x slower
with latencies in the high (but not insane) range of 200ms. and 12kb is
what we've seen deployed to this point - roberto suggested that a
per-session window would allow a greater value.

and you can see Chris Strom running 64KB download windows down to zero
at
http://japhr.blogspot.com/2012/05/firefox-spdy2-vs-spdy3-in-graphs.html ..

it seems to me this will be a very hard knob to set correctly. So if it
can go away - it should.



Peter Lepeska

unread,
May 21, 2012, 12:50:47 PM5/21/12
to spdy...@googlegroups.com
I realize that this email thread is primarily about adding another layer of flow control to SPDY that operates at the SPDY session level but I'm still trying to understand if even the per stream flow control is necessary.

So it seems like the main problem per stream flow control is meant to solve is head of line blocking. To be specific, the problem is when you have a low and a high priority stream active and the low priority stream is transferring a lot of data from a fast content server and the high priority stream is only sending intermittently. In this case, there can be a full 64KB of low priority data in flight across the TCP connection when the high priority data needs to be sent over the TCP connection. If the first packet of the 64KB gets dropped, then the high priority data has to wait at least until the low priority block gets re-transmitted. 

Per stream flow control mitigates (but does not solve) this by preventing the low priority data from ever using the full TCP connection so that the number of low priority bytes in front of the high priority data is smaller. The drawback is that the low priority stream never gets to use the full connection, even when there is no high priority data to send. Is this description correct? And is this the primary use case for per stream flow control?

Has it been considered to use multiple SPDY sessions, one TCP connection per priority level at any given time, so that two streams with different priorities are never competing for bandwidth over the same TCP connection?

Not asking anyone in particular. Just trying to think through the problem. My apologies if I'm taking the discussion backwards a bit.

Thanks,

Peter

Eric Ceres

unread,
May 21, 2012, 1:06:18 PM5/21/12
to spdy...@googlegroups.com, spdy...@googlegroups.com
Hi,

To add more clarity here is an example. A large file is being uploaded. Let's say the server is not able to store the file fast enough. Then with a standard http transmission the receive buffer fills up and the upload waits for more buffer space.

With speedy this means the server stops reading the buffer then all requests will be blocked. This leaves few options:
1. Handle infinite buffer (not really doable)
2. Have someway to block further requests on the stream by notify all reads blocked/unblocked (on/off switch which could trigger new connection)
3. Have someway to block buffering for a stream (moves buffer/blocking from TCP to protocol handling level) 

Eric Ceres

Roberto Peon

unread,
May 21, 2012, 1:17:55 PM5/21/12
to spdy...@googlegroups.com
Nah, this is nowhere near as complicated as TCP. We're only solving a flow-control issue here.
 
With all the problems it carries (e.g. bufferbloat), and additional
ones (how to share bandwidth among streams) ?

If this is done right, we won't have bufferbloat (or, if we do, it would still be better than the bufferbloat which would be happening with many TCP connections).

-=R

Roberto Peon

unread,
May 21, 2012, 1:23:58 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 9:50 AM, Peter Lepeska <bizzb...@gmail.com> wrote:
I realize that this email thread is primarily about adding another layer of flow control to SPDY that operates at the SPDY session level but I'm still trying to understand if even the per stream flow control is necessary.

So it seems like the main problem per stream flow control is meant to solve is head of line blocking. To be specific, the problem is when you have a low and a high priority stream active and the low priority stream is transferring a lot of data from a fast content server and the high priority stream is only sending intermittently. In this case, there can be a full 64KB of low priority data in flight across the TCP connection when the high priority data needs to be sent over the TCP connection. If the first packet of the 64KB gets dropped, then the high priority data has to wait at least until the low priority block gets re-transmitted. 

Well, per-stream flow control is necessary to prevent HOL blocking through proxies, since the proxies will demux your multiplexed session, and then remux onto other connections (any of which will have different path-restrictions or buffer requirements). Most loadbalancers are proxies.

 

Per stream flow control mitigates (but does not solve) this by preventing the low priority data from ever using the full TCP connection so that the number of low priority bytes in front of the high priority data is smaller. The drawback is that the low priority stream never gets to use the full connection, even when there is no high priority data to send. Is this description correct? And is this the primary use case for per stream flow control?

Has it been considered to use multiple SPDY sessions, one TCP connection per priority level at any given time, so that two streams with different priorities are never competing for bandwidth over the same TCP connection?

It has been considered and, at least so far, rejected as impractical. Basically, each TCP connection likely goes to a different machine for any decent scale deployment (most of which use loadbalancers). You end up losing too much latency, you add significant cost, and you end up with much more potential buffer bloat.

 

Not asking anyone in particular. Just trying to think through the problem. My apologies if I'm taking the discussion backwards a bit.

Its a good thought, and if we find that the problems with multiplexing are insurmountable it will likely be tried. We're not there yet, though!
-=R

Peter Lepeska

unread,
May 21, 2012, 1:25:07 PM5/21/12
to spdy...@googlegroups.com
That example helps a lot. So per stream flow control in this example is about managing buffering on the SPDY server and the SPDY client for uploads and downloads respectively. If that is the case, then it should only kick in when the network is faster than the receiving end point, which in this case is the content server receiving the uploaded file, right?

So then in summary there are two use cases for per stream flow control -- 1) mitigating HOL blocking and 2) managing buffering by providing feedback on receiver rate of data acceptance. Any others?

The problem with #1 is that it slows down low priority connections even when there is no high priority data to send. For #2, there has to be some type of feedback because as you said infinite buffering is not an option but at least the per stream flow control only has to kick in when the receiving end point is slow.

Anyone please correct anything incorrect about the above understanding,

Peter

Costin Manolache

unread,
May 21, 2012, 1:25:16 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 10:06 AM, Eric Ceres <eric...@gmail.com> wrote:
Hi,

To add more clarity here is an example. A large file is being uploaded. Let's say the server is not able to store the file fast enough. Then with a standard http transmission the receive buffer fills up and the upload waits for more buffer space.

With speedy this means the server stops reading the buffer then all requests will be blocked. This leaves few options:
1. Handle infinite buffer (not really doable)
2. Have someway to block further requests on the stream by notify all reads blocked/unblocked (on/off switch which could trigger new connection)
3. Have someway to block buffering for a stream (moves buffer/blocking from TCP to protocol handling level) 

So the goal is to allow the sender to keep sending high-priority and 'fast' streams, based on some info about the receiver buffers and how much of each stream is buffered. 

If SPDY had session-level flow control, the sender would know how much 
total buffer space is available - and slow down some streams, stop sending lower priority, etc - but still be able to send higher priority streams until receiver buffer is completely filled. At that point - it can't send anything else.

With the current per-stream, the problem is that the receiver ( or proxy ) needs a buffer that is the sum of all streams windows, for the worse-case when all streams are stalled.

With extra session-level you can have a smaller buffer in the receiver, and if the client may adapt the sending. But eventually you can still get to a worse-case state where the receiver buffer is all full and you can't send anything new stream. In which case if the client knows it's happened, it can either open a new connection or cancel some of the streams.

Costin

Peter Lepeska

unread,
May 21, 2012, 1:59:15 PM5/21/12
to spdy...@googlegroups.com
" Well, per-stream flow control is necessary to prevent HOL blocking through proxies, since the proxies will demux your multiplexed session, and then remux onto other connections (any of which will have different path-restrictions or buffer requirements). Most loadbalancers are proxies. "

I'm having trouble understanding how proxies exacerbate the problem of HOL blocking. Can you provide a little more detail on this example? Is it b/c a proxy adds its own buffering and therefore increases the amount of low priority data, over and above a single TCP send window, that can be ahead of high priority data on the stream?

Thanks,

Peter

Greg Wilkins

unread,
May 21, 2012, 2:20:42 PM5/21/12
to spdy...@googlegroups.com
On 18 May 2012 19:09, Patrick McManus <mcm...@ducksong.com> wrote:
Having 2 levels of flow control lets you separate the concerns: the
session value is about total buffers available, the per-stream value is
about letting different streams proceed at different rates (and that's
why I think it can be done with xon/xoff in the presence of a session
window).

Patrick,

Can you confirm deny if the following paraphrasing captures what you mean by xon/xoff.

Your hypothesis is that we can achieve near maximum TCP/IP throughput, but without HOL channel blocking by having two levels of flow control.  The outer flow control is applied to the session as a whole and once some threshold is reached, then channels will start being given Xoff messages to stop them sending more data.     So which channels would received the Xoff?  The next one to send a data packet or the ones with the most recent usage?

When would a Xon be sent? after a time delay? once all queued data is sent? once queued data is reduced below some threshold?

Do you think it would be possible to use TCP itself as the outer flow control?  Ie let all channels send as much data as they like until the next frame would block the TCP/IP stream.  At this point Xoff's would be issued to the busiest channels and data would be queued for the non-busy channels... until such point as they become busy and are Xoffed or until the buffers are full and all channels are Xoffed.    Xons would be issued as the buffers are emptied (to the least busy channels first). I think such an algorithm would represent a small amount of HOL blocking as it would allow a busy channel to run the TCP/IP connection until it was actually flow controlled, but it would give the non busy channels priority once the flow resumes.     If something like this is workable, then it has the advantage of not needing any window sizes at all - just TCP flow control and channel xon/xoff.

cheers




William Chan (陈智昌)

unread,
May 21, 2012, 2:50:45 PM5/21/12
to spdy...@googlegroups.com
If a proxy receives data for a stream X, but its backend cannot consume the data anymore, then without per-stream flow control or infinite buffering, the proxy must stop reading the TCP connection. Thus, stopping the reading of data for stream X requires stopping the reading of the entire TCP connection, so stream X causes HoL blocking of all other stream data behind it in the TCP connection.

Patrick McManus

unread,
May 21, 2012, 3:11:07 PM5/21/12
to spdy...@googlegroups.com
On Mon, 2012-05-21 at 20:20 +0200, Greg Wilkins wrote:
>
>
> On 18 May 2012 19:09, Patrick McManus <mcm...@ducksong.com> wrote:
> Having 2 levels of flow control lets you separate the
> concerns: the
> session value is about total buffers available, the per-stream
> value is
> about letting different streams proceed at different rates
> (and that's
> why I think it can be done with xon/xoff in the presence of a
> session
> window).
>
> Patrick,
>
> Can you confirm deny if the following paraphrasing captures what you
> mean by xon/xoff.
>
> Your hypothesis is that we can achieve near maximum TCP/IP throughput,
> but without HOL channel blocking by having two levels of flow control.
> The outer flow control is applied to the session as a whole and once
> some threshold is reached, then channels will start being given Xoff
> messages to stop them sending more data.

pretty much

> So which channels would received the Xoff? The next one to send a
> data packet or the ones with the most recent usage?
>

not specified by the protocol - the receiver can implement it any way
they want. I would expect that the receiver would xoff a stream who's
data sink had built a stream specific buffer too large. (i.e. was not
consuming at line rate). I'm not naive, I understand that this is a sort
of windowing in itself but for configuration it separates out the
necessary BDP component for making the stream run at line rate, which I
hope we all agree is the goal under normal circumstances, and that's the
knob people are going to intuitively buffer too low and not future
proof.

When someone asks themselves "how many buffers do I allocate to the data
base process" are they asking themselves contextually "how many buffers
do I allocate for the DB and at what speed and latency?" I'm going to
say they don't do that and that's an impossibly hard question to ask an
admin anyhow. Much better to say "how many buffers on top of those
required by the network for full data rate" will you allocate to this
stream?

The full session buffer limit can backstop you if you really don't have
the resources to run in a high BDP environment.

> When would a Xon be sent? after a time delay? once all queued data is
> sent? once queued data is reduced below some threshold?
>

from the pov of the protocol - it isn't specified. I would expect some
watermark approach.

> Do you think it would be possible to use TCP itself as the outer flow
> control?

Maybe; but probly not.

I think its a little tricky for a server (or the server side of a proxy)
to do things like 1] exhaust tcp rwin, 2] xoff all senders, 3] update
initial window via settings to prevent new senders and 4] reopen tcp
rwin a bit to process GETs and pings and goaways etc..

The client in that situation is going to have data buffered in its tcp
layer that it put there before it realized rwin was empty (every body
runs with some send buffers) that is going to go out when the window
reopens.. it could be anything.. and there is of course some state
(compression, sequence numbers, etc) to the data stream so its a hard
bell to unring. Better to just not generate it when it can't be given to
the tcp stack.

That's the advantage of a spdy centric session window - it just covers
the goodput and lets the control stuff flow as freely as tcp allows.


Costin Manolache

unread,
May 21, 2012, 3:26:24 PM5/21/12
to spdy...@googlegroups.com
Can the decision be made on the sender side instead ?

The receiver would send back info about total buffer size and any stream that has more is using more than x% of the buffer (or some other measure).  

So instead of 2 flow controls - one for stream, one per channel, you would just send more detailed info about the buffer and top stream status, and let the client chose what to send next and what to suspend. I think client or backend are better to decide if they have enough info.

Costin

Roberto Peon

unread,
May 21, 2012, 3:34:28 PM5/21/12
to spdy...@googlegroups.com
Detail about the buffer is what flow control exposes, though, right?
The client always chooses what to send and what not to send anyway. A receiver never has control over the sender, they can merely give them policy and hope the sender acts upon it.

We'll always need per-stream flow control. No avoiding that. Anything which accurately signals the number of bytes per stream which it is safe to send, and which provides a constant and predictable overall memory usage on the receiver for all streams will do.

I'm having a hard time reconciling your suggestion with this. If we do the per-stream flow control, and can't do 2 levels of flow control, what data would we send which wouldn't BE flow control that would allow the server to be assured (with a well behaved client) to use no more buffer than it wishes to use?

-=R

Costin Manolache

unread,
May 21, 2012, 4:57:01 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 12:34 PM, Roberto Peon <fe...@google.com> wrote:
Detail about the buffer is what flow control exposes, though, right?
The client always chooses what to send and what not to send anyway. A receiver never has control over the sender, they can merely give them policy and hope the sender acts upon it.

Right now the proxy controls things by setting window - I assume if a client doesn't obey flow control the server will close the connection, it's not quite 'client decide what to send'.

Yes, the details (inputs) would be the same for any solution - max buffer of the receiver, current window for each stream, how much of the total and per-stream buffer is used.

My suggestion is to send all this data to client - instead of sending info about how much each stream is allowed to send, send data about how much of each stream is buffered and the total buffer size.

For example: the proxy has a 1M buffer per connection. Client can choose to send 900K of one stream, than wait for the flow control frame, and keep sending high-priority or small requests. If the large upload doesn't make any progress in 10 min, it can chose to cancel it, or not send any more bytes until it makes progress. 

Benefits:

- A smart client will have more choices on what it sends - in particular it may chose to drop some streams. I think that's the main difference - if few  large uploads gets stale for a long time, there is nothing in the current per-stream / per-channel flow control that would help, the only help is to drop the stale upload.

- a dumb client doesn't need to do anything - can just ignore the flow control and the TCP flow control will do its job.

- the server doesn't have to send data for all streams - only for the streams that use large buffers or are stalled ( fewer bytes ).

- the initial per stream window is close to the total buffer size of the proxy ( instead of small per stream window to account for the worse case of all streams getting buffered )

- the code may be simpler on the server, and for equivalent functionality it may be simple on the client.




We'll always need per-stream flow control. No avoiding that. Anything which accurately signals the number of bytes per stream which it is safe to send, and which provides a constant and predictable overall memory usage on the receiver for all streams will do.

Yes, you need to know the buffering status per-stream - maybe not for all streams, and it doesn't have to be 'per stream flow control'. You just send info about the buffer usage on the proxy, client can make any choice as long as it respects total buffer size.


 

I'm having a hard time reconciling your suggestion with this. If we do the per-stream flow control, and can't do 2 levels of flow control, what data would we send which wouldn't BE flow control that would allow the server to be assured (with a well behaved client) to use no more buffer than it wishes to use?

Server will send total buffer size (once), than periodically will send stream status info. For example any time it would send a 'window update' now, or only if the buffer use is > 50%. 

The data is about the same with the current flow control - how many bytes can be sent by the client safely (now) versus how many bytes are currently buffered (in my proposal). The frequency can be the same - or you can send flow info only when you need to (now window updates are required for each stream, or client will be forced to stop sending). The decision on when to allow sending is similar - except it's done by client using the info from server, instead of server. 

Costin

Costin Manolache

unread,
May 21, 2012, 5:06:05 PM5/21/12
to spdy...@googlegroups.com
If I'm too confusing, my proposal is:

- remove the current per stream window

- server will declare it's total buffer  ( with a reasonable default - 64k or 1M), and use a per-connection window.

- server will send flow control packets with a list of stream IDs and how much bytes are buffered for each stream. There are few options on which streams to include in the flow control packet, and when to send this packet.

- client will not send more than [total buffer] until it receives a flow control packet with a stream-based window update. When it receives a flow control packet - it can chose what stream to send based on free space, priorities, etc - as long as total in-flight bytes fit total buffer.

- client should also detect if some streams are 'stalled' - no progress for X minutes, too much data buffered - and may time-out them so it can send other streams.

It's still a combination of connection and stream flow control.

Costin

William Chan (陈智昌)

unread,
May 21, 2012, 5:09:15 PM5/21/12
to spdy...@googlegroups.com
This doesn't make sense to me. You say "remove the current per stream window" and then you say "it's still a combination of connection and stream flow control".

On Mon, May 21, 2012 at 2:06 PM, Costin Manolache <cos...@gmail.com> wrote:
If I'm too confusing, my proposal is:

- remove the current per stream window

- server will declare it's total buffer  ( with a reasonable default - 64k or 1M), and use a per-connection window.

- server will send flow control packets with a list of stream IDs and how much bytes are buffered for each stream. There are few options on which streams to include in the flow control packet, and when to send this packet.

This sounds like per stream window updates. Why do you say "remove the current per stream window"?
 

- client will not send more than [total buffer] until it receives a flow control packet with a stream-based window update. When it receives a flow control packet - it can chose what stream to send based on free space, priorities, etc - as long as total in-flight bytes fit total buffer.

With a per-stream window and per-session window, a sender could still do this. When space opens up in the per-session window, the sender is obviously allowed to choose which stream to send data over, assuming that per-stream window has space.

Costin Manolache

unread,
May 21, 2012, 5:41:13 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 2:09 PM, William Chan (陈智昌) <will...@chromium.org> wrote:
This doesn't make sense to me. You say "remove the current per stream window" and then you say "it's still a combination of connection and stream flow control".

It replaces the current definition of per-stream window - right now each stream window/flow is sent individually, all streams have the same initial window, and the information sent back is how much it's allowed to sent for the stream.

It still has per-stream flow control - in the sense that you send back how much is buffered for each stream ( all or or only streams that have >x bytes or % buffered ). 
 


On Mon, May 21, 2012 at 2:06 PM, Costin Manolache <cos...@gmail.com> wrote:
If I'm too confusing, my proposal is:

- remove the current per stream window

- server will declare it's total buffer  ( with a reasonable default - 64k or 1M), and use a per-connection window.

- server will send flow control packets with a list of stream IDs and how much bytes are buffered for each stream. There are few options on which streams to include in the flow control packet, and when to send this packet.

This sounds like per stream window updates. Why do you say "remove the current per stream window"?

The window updates will be per session. 

The 'per stream window' is replaced by info about how much is currently buffered for the stream, combined with info about total session buffer available. 


 
 

- client will not send more than [total buffer] until it receives a flow control packet with a stream-based window update. When it receives a flow control packet - it can chose what stream to send based on free space, priorities, etc - as long as total in-flight bytes fit total buffer.

With a per-stream window and per-session window, a sender could still do this. When space opens up in the per-session window, the sender is obviously allowed to choose which stream to send data over, assuming that per-stream window has space.

Yes, but I guess it's a more direct decision when you know total buffer and 
how much each stream has buffered. And when you start new streams - they won't be limited by a small per-stream window, but by per-session window.

You can probably infer from the stream window how much of that stream is buffered and if the stream is stale.


Costin 

William Chan (陈智昌)

unread,
May 21, 2012, 5:58:14 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 2:41 PM, Costin Manolache <cos...@gmail.com> wrote:
On Mon, May 21, 2012 at 2:09 PM, William Chan (陈智昌) <will...@chromium.org> wrote:
This doesn't make sense to me. You say "remove the current per stream window" and then you say "it's still a combination of connection and stream flow control".

It replaces the current definition of per-stream window - right now each stream window/flow is sent individually, all streams have the same initial window, and the information sent back is how much it's allowed to sent for the stream.

It still has per-stream flow control - in the sense that you send back how much is buffered for each stream ( all or or only streams that have >x bytes or % buffered ). 
 


On Mon, May 21, 2012 at 2:06 PM, Costin Manolache <cos...@gmail.com> wrote:
If I'm too confusing, my proposal is:

- remove the current per stream window

- server will declare it's total buffer  ( with a reasonable default - 64k or 1M), and use a per-connection window.

- server will send flow control packets with a list of stream IDs and how much bytes are buffered for each stream. There are few options on which streams to include in the flow control packet, and when to send this packet.

This sounds like per stream window updates. Why do you say "remove the current per stream window"?

The window updates will be per session. 

The 'per stream window' is replaced by info about how much is currently buffered for the stream, combined with info about total session buffer available. 


 
 

- client will not send more than [total buffer] until it receives a flow control packet with a stream-based window update. When it receives a flow control packet - it can chose what stream to send based on free space, priorities, etc - as long as total in-flight bytes fit total buffer.

With a per-stream window and per-session window, a sender could still do this. When space opens up in the per-session window, the sender is obviously allowed to choose which stream to send data over, assuming that per-stream window has space.

Yes, but I guess it's a more direct decision when you know total buffer and 
how much each stream has buffered. And when you start new streams - they won't be limited by a small per-stream window, but by per-session window.

Don't you know how much each stream has buffered via per-stream flow control windows? And the existence of a per-session window obviates the need for small initial per-stream windows. You can make them larger now. 64k is a reasonable default still (if you disagree, let's fix it then), and the SETTINGS frame lets you adjust it to a more appropriate value within one RTT for the server rwin, and zero RTTs for the client rwin.

I guess I don't understand how communicating a percentage of stream buffer size is better than a per-stream flow control window size. The latter approach lets you dynamically adjust the window size on a per-stream basis in a more elegant matter IMO.

Roberto Peon

unread,
May 21, 2012, 6:01:29 PM5/21/12
to spdy...@googlegroups.com
I think they both achieve the same goal. The server can lie about the amount of buffer it has available to accomplish any of the goals that it would have done. The question thus resolves to: which is easier to implement.
I'd guess that the current window-based way is easier and cheaper since the window-updates are delta-encodings for the state changes.

Did you think this would allow something else Costin? I readily admit I could be missing something :)

-=R

Costin Manolache

unread,
May 21, 2012, 7:17:09 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 2:58 PM, William Chan (陈智昌) <will...@chromium.org> wrote:
On Mon, May 21, 2012 at 2:41 PM, Costin Manolache <cos...@gmail.com> wrote:
On Mon, May 21, 2012 at 2:09 PM, William Chan (陈智昌) <will...@chromium.org> wrote:
This doesn't make sense to me. You say "remove the current per stream window" and then you say "it's still a combination of connection and stream flow control".

It replaces the current definition of per-stream window - right now each stream window/flow is sent individually, all streams have the same initial window, and the information sent back is how much it's allowed to sent for the stream.

It still has per-stream flow control - in the sense that you send back how much is buffered for each stream ( all or or only streams that have >x bytes or % buffered ). 
 


On Mon, May 21, 2012 at 2:06 PM, Costin Manolache <cos...@gmail.com> wrote:
If I'm too confusing, my proposal is:

- remove the current per stream window

- server will declare it's total buffer  ( with a reasonable default - 64k or 1M), and use a per-connection window.

- server will send flow control packets with a list of stream IDs and how much bytes are buffered for each stream. There are few options on which streams to include in the flow control packet, and when to send this packet.

This sounds like per stream window updates. Why do you say "remove the current per stream window"?

The window updates will be per session. 

The 'per stream window' is replaced by info about how much is currently buffered for the stream, combined with info about total session buffer available. 


 
 

- client will not send more than [total buffer] until it receives a flow control packet with a stream-based window update. When it receives a flow control packet - it can chose what stream to send based on free space, priorities, etc - as long as total in-flight bytes fit total buffer.

With a per-stream window and per-session window, a sender could still do this. When space opens up in the per-session window, the sender is obviously allowed to choose which stream to send data over, assuming that per-stream window has space.

Yes, but I guess it's a more direct decision when you know total buffer and 
how much each stream has buffered. And when you start new streams - they won't be limited by a small per-stream window, but by per-session window.

Don't you know how much each stream has buffered via per-stream flow control windows? And the existence of a per-session window obviates the need for small initial per-stream windows. You can make them larger now. 64k is a reasonable default still (if you disagree, let's fix it then), and the SETTINGS frame lets you adjust it to a more appropriate value within one RTT for the server rwin, and zero RTTs for the client rwin.

Would 64K be the default for both session and stream windows ?

I guess my proposal is almost equivalent - if you know the stream window and client window you can calculate how much is buffered. After you fill the session window you need to wait for session and stream window updates in both cases.


I guess I don't understand how communicating a percentage of stream buffer size is better than a per-stream flow control window size. The latter approach lets you dynamically adjust the window size on a per-stream basis in a more elegant matter IMO.

I think it's more direct to tell the client the relevant information - how big is the proxy buffer and how much of each stream is buffered. 

The 'percentage' was an optimization - you don't need to send flow control for streams that go trough / have very low buffers, or if the proxy buffer has plenty of space. The per-stream flow will only kick in when it's needed.
You can still do delta-encoding for stream and session. 


What would be the meaning of 'per stream window' - it's no longer amount you are allowed to send for that stream - you need to consider the session window and all other streams windows. It's this computation that I think would be cleaner if you work in reverse, with how much is buffered instead of the 'window' which is no longer a direct indication of how much to send.

But if you can find a good explanation of the stream window and how it interact with the session window it would be less confusing.

Costin

Costin Manolache

unread,
May 21, 2012, 7:25:35 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 3:01 PM, Roberto Peon <fe...@google.com> wrote:
I think they both achieve the same goal. The server can lie about the amount of buffer it has available to accomplish any of the goals that it would have done. The question thus resolves to: which is easier to implement.

Agreed.
 
I'd guess that the current window-based way is easier and cheaper since the window-updates are delta-encodings for the state changes.

I think a 'dumb client' would be simpler with my proposal - it would just ignore all flow control packets. The TCP flow control would ensure the 'per session' buffer is used. 

In both cases a smart client needs to keep track of how much they can send on each stream - based on session window and stream window, which in both cases is a proxy for how fast that stream is and how much is buffered. 

In the current flow,the information is received as one packet per stream, when the stream is making progress, plus one new packet for session.
In my proposal - the almost same information will be sent in a single packet, which will only be sent if the session buffer is to big. 

In the good case - all streams floating around - my proposal wouldn't send any stream flow control packet, only the session info.

Costin

William Chan (陈智昌)

unread,
May 21, 2012, 7:35:13 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 4:17 PM, Costin Manolache <cos...@gmail.com> wrote:


On Mon, May 21, 2012 at 2:58 PM, William Chan (陈智昌) <will...@chromium.org> wrote:
On Mon, May 21, 2012 at 2:41 PM, Costin Manolache <cos...@gmail.com> wrote:
On Mon, May 21, 2012 at 2:09 PM, William Chan (陈智昌) <will...@chromium.org> wrote:
This doesn't make sense to me. You say "remove the current per stream window" and then you say "it's still a combination of connection and stream flow control".

It replaces the current definition of per-stream window - right now each stream window/flow is sent individually, all streams have the same initial window, and the information sent back is how much it's allowed to sent for the stream.

It still has per-stream flow control - in the sense that you send back how much is buffered for each stream ( all or or only streams that have >x bytes or % buffered ). 
 


On Mon, May 21, 2012 at 2:06 PM, Costin Manolache <cos...@gmail.com> wrote:
If I'm too confusing, my proposal is:

- remove the current per stream window

- server will declare it's total buffer  ( with a reasonable default - 64k or 1M), and use a per-connection window.

- server will send flow control packets with a list of stream IDs and how much bytes are buffered for each stream. There are few options on which streams to include in the flow control packet, and when to send this packet.

This sounds like per stream window updates. Why do you say "remove the current per stream window"?

The window updates will be per session. 

The 'per stream window' is replaced by info about how much is currently buffered for the stream, combined with info about total session buffer available. 


 
 

- client will not send more than [total buffer] until it receives a flow control packet with a stream-based window update. When it receives a flow control packet - it can chose what stream to send based on free space, priorities, etc - as long as total in-flight bytes fit total buffer.

With a per-stream window and per-session window, a sender could still do this. When space opens up in the per-session window, the sender is obviously allowed to choose which stream to send data over, assuming that per-stream window has space.

Yes, but I guess it's a more direct decision when you know total buffer and 
how much each stream has buffered. And when you start new streams - they won't be limited by a small per-stream window, but by per-session window.

Don't you know how much each stream has buffered via per-stream flow control windows? And the existence of a per-session window obviates the need for small initial per-stream windows. You can make them larger now. 64k is a reasonable default still (if you disagree, let's fix it then), and the SETTINGS frame lets you adjust it to a more appropriate value within one RTT for the server rwin, and zero RTTs for the client rwin.

Would 64K be the default for both session and stream windows ?

I suspect we can make the session window larger than 64K. I'm open to suggestions here.
 

I guess my proposal is almost equivalent - if you know the stream window and client window you can calculate how much is buffered. After you fill the session window you need to wait for session and stream window updates in both cases.


I guess I don't understand how communicating a percentage of stream buffer size is better than a per-stream flow control window size. The latter approach lets you dynamically adjust the window size on a per-stream basis in a more elegant matter IMO.

I think it's more direct to tell the client the relevant information - how big is the proxy buffer and how much of each stream is buffered. 

The 'percentage' was an optimization - you don't need to send flow control for streams that go trough / have very low buffers, or if the proxy buffer has plenty of space. The per-stream flow will only kick in when it's needed.
You can still do delta-encoding for stream and session. 


What would be the meaning of 'per stream window' - it's no longer amount you are allowed to send for that stream - you need to consider the session window and all other streams windows. It's this computation that I think would be cleaner if you work in reverse, with how much is buffered instead of the 'window' which is no longer a direct indication of how much to send.

But if you can find a good explanation of the stream window and how it interact with the session window it would be less confusing.

I guess I'm not sure what's confusing about the interaction with the stream window and the session window. When writing stream data in a naive implementation, the amount you'd write is amount_to_write = min(stream_data_length, stream_window_size, session_window_size). Then stream_window_size -= amount_to_write and session_window_size -= amount_to_write. A more advanced implementation may examine the remaining session window size to see how much space is left and the stream priority, and based on info like that, may opt to write less (or zero) stream data if the session window is small and/or the stream priority is low.

If we wanted to allow dynamically resizing the session window size apart from per stream windows, then we'd have to introduce a separate SESSION_WINDOW_UPDATE or something. Or add a new field to the WINDOW_UPDATE frame for the amount to adjust the session window.

Hasan Khalil

unread,
May 21, 2012, 7:47:58 PM5/21/12
to spdy...@googlegroups.com
On Mon, May 21, 2012 at 7:35 PM, William Chan (陈智昌) <will...@chromium.org> wrote:
If we wanted to allow dynamically resizing the session window size apart from per stream windows, then we'd have to introduce a separate SESSION_WINDOW_UPDATE or something. Or add a new field to the WINDOW_UPDATE frame for the amount to adjust the session window.

Or neither: send a WINDOW_UPDATE for stream 0.

    -Hasan

Costin Manolache

unread,
May 21, 2012, 10:03:24 PM5/21/12