Re: Need advice about how chunked streaming technologies work with cache settings

[Nick Hurley]

----- Original Message -----
> 1/ What is the default setting in Firefox for the Cache? Is it RAM
> then Disk. Is this flow described anywhere?
>
>
> One of our cache experts, Nick, has told me that the resources are
> cached in both RAM and disk by default.

To clarify, both RAM and disk caches are *enabled* by default. As things stand right now, most resources will be cached in the disk cache, and not in RAM (IIRC, RAM is reserved for TLS resources served with cache-control no-store).

> Nick, Michal, is there a flow described somewhere for the cache that
> Alan could look at? Do you know how plugins like Flash use the
> cache? Is there a difference?

I don't believe there is any documentation of the flow for the entire cache, no. I am also not aware of how Flash and other plugins make use of the cache.

> 2/ How is garbage collection instigated - the latest Firefox has an
> option suggesting that, by default, there is an automatic garbage
> collection which must use some rules... perhaps about disk space? For
> example: in Google Chrome, the default cache can take up huge amounts
> of disk space growing without some sort of size limit it seems, plus
> you can only change the cache settings on the CLI of Chrome (no
> 'consumer' level configuration in the 'preferences' dialogs).
>
>
> I'll let Nick and Michal comment on garbage collection for regular
> cache objects. We also have a media cache, which the guys on
> dev-media have a better understanding since they wrote it, but I
> don't think it's used for Flash, only for built-in decoders.

The regular cache is limited to 1GB (or less, depending on free disk space). Once that limit is reached, space is freed up as needed to store new entries in an LRU fashion. Memory cache is limited to 32MB, I think? It uses a size-aware LRU eviction algorithm. There is some configuration for disk cache size in the regular preferences pane in Firefox.

[Alan Ogilvie]

Just realised that it would be useful if I supplied you with a real-world
example:

http://www.bbc.co.uk/bbcone/watchlive/

This is 'UK Only', restricted based on the requesting IP address.

[Alan Ogilvie]

Thank you all for your response so far - clearly this is going to be a
somewhat complicated answer (I didn't believe that it wouldn't) ;-)

3/ - yes, my (and somewhat 'our') hypothesis is that because chunk requests
are made through the host browser, they will be restricted as any other
internet content is - for example, requesting a web page, then all the
assets from the HTML response, is clearly managed by limiting the number of
concurrent 'threads' (I may obviously be using the wrong words here) to
some sort of rule for 'same domain'. If we had more than one player in the
browser instance (maybe on multiple tabs/windows) all requesting chunks
from the same domain, but different 'channels' (ie. different streams),
then they will likely 'fight' each other for spare slots in the concurrent
'threads'. We're not sure how much of an 'edge case' that is - though I can
believe some of our viewers/listeners having a couple of windows open
during big events (like the Olympics in London coming soon) with HDS
playback in Flash.

I have supplied an example of HDS through Flash which is live on our site
at the moment. This may help you see how it's working in the real world.
I've already poked around to see what happens to the chunks.

In RTMP communication, or where NetConnect (or similar) is being used
directly inside the Flash Player container - Flash handles that
communication. But for HDS it throws the HTTP request up to the host
browser. This is what sees to happen.

On 18 May 2012 02:05, Steve Workman <swor...@mozilla.com> wrote:

> Hi Alan,
>
> (I'm cross posting to dev-media to get some insight on the media cache -
> media folks, this email is with respect to Flash/HDS, and I don't think it
> uses the media cache - please confirm.
>
> And Patrick, I'm bouncing it back to you for the resource-sharing question
> about parallel chunk requests :) I'm not 100% on the details, so I'm going
> to spend some time on that, but in the meantime, you might be able to give
> Alan a quicker response than me.
>
> And Nick and Michal, please take a look over the disk cache question).
>
> On May 17, 2012, at 6:31 AM, Patrick McManus wrote:
>
> Maybe steve can give a little insight here. He's been working on a DASH
> implementation.
>
>
> As Patrick wrote, I'm working on DASH/WebM for Firefox, so I'll comment as
> best I can from that context, but there may be differences with how Flash
> works. Also, for some of these things I'm learning as well, so forgive me
> for forwarding you to other folks for specific details.
>
> On May 13, 2012, at 5:20 PM, AlanO wrote:
>
> I'm working with the HDS streaming technologies, within the Adobe
> Flash world, and I'm trying to compile a set if 'FAQs' about each main
> browser available to consumers and how it will react to the 'chunks'
> being delivered via the HTTP stack.
>
> My current understanding on Firefox is that, although the Adobe Flash
> plugin handles presentation of the video, the chunks are requested via
> the host browser's network stack. Resulting in an HTTP request for a
> chunk (aka 'fragment').
>
> A couple of things have struck me about this:

>
> 1/ What is the default setting in Firefox for the Cache? Is it RAM
> then Disk. Is this flow described anywhere?
>
>
> One of our cache experts, Nick, has told me that the resources are cached
> in both RAM and disk by default.
>

> Nick, Michal, is there a flow described somewhere for the cache that Alan
> could look at? Do you know how plugins like Flash use the cache? Is there a
> difference?
>
>

> 2/ How is garbage collection instigated - the latest Firefox has an
> option suggesting that, by default, there is an automatic garbage
> collection which must use some rules... perhaps about disk space? For
> example: in Google Chrome, the default cache can take up huge amounts
> of disk space growing without some sort of size limit it seems, plus
> you can only change the cache settings on the CLI of Chrome (no
> 'consumer' level configuration in the 'preferences' dialogs).
>
>
> I'll let Nick and Michal comment on garbage collection for regular cache
> objects. We also have a media cache, which the guys on dev-media have a
> better understanding since they wrote it, but I don't think it's used for
> Flash, only for built-in decoders.
>
>

> 3/ What are the methods employed when multiple chunks are requested -
> as with loading a web page, there is some 'pipeline' going on in the
> network stack. Perhaps limiting the total number of working threads
> for 'same domain' requests - this could impact on HDS chunks. Consider
> where more than one HDS stream is present in a users browser (e.g.
> multiple tabs/windows with an HDS stream) - if chunks are served from
> the same FQDN, would there be a 'fight' due to slots within the worker
> threads in the network stack?
>
>
> Chunks as a concept is something on my todo-list-to-think-about for DASH,
> albeit to examine the effect of different chunk sizes on TCP window size
> and thus bandwidth usage, which is, of course, a different issue.
>
> So, to make sure I understand your question, you're asking if there could
> be a fight for resources if there were two or more media presentations
> grabbing files/chunks from the same FQDN. Assuming that's right, I'm going
> to look into it and get back to you, because I'm not 100% sure. I have a
> notion that it depends on what protocol you're using (HTTP, maybe with
> pipelining, or SPDY) and the number of parallel TCP connections available.
> As far as the number of threads or other resources we have in Firefox, I'm
> not sure of those details.
>
> In the meantime, Patrick may actually have a better idea about the inner
> workings - sorry to bounce it back, Patrick :)
>
>
> Although I mention HDS here, this would likely go for any other
> chunked streaming methods (MPEG-DASH, HLS) which may have a client
> implemented inside a browser host.
>
>
> For sure, so it's useful discussion to have all round.
>
>
> I hope that someone in your team could assist me with these queries.
>
> Alan
> _______________________________________________
> dev-tech-network mailing list
> dev-tech...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-network
>
>
>

[Robert O'Callahan]

> I'll let Nick and Michal comment on garbage collection for regular

> > cache objects. We also have a media cache, which the guys on
> > dev-media have a better understanding since they wrote it, but I
> > don't think it's used for Flash, only for built-in decoders.
>

Correct. Our media cache does not affect Flash in any way.

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]

[Steve Workman]

On May 21, 2012, at 3:26 AM, Alan Ogilvie wrote:

> On this one - remember that in the case of HDS in Flash, the requests for fragments are being bubbled up to the host browser HTTP stack - so would this change your answer?
>
Nope. Our HTTP stack and ram/disk cache are completely different entities to the Media Cache. I only mentioned it to get confirmation from one of the media guys. So, it wouldn't change the answer. Flash should not be using the Media Cache.

> Further - why would we want fragments going into the media cache (i.e. what's the benefits of doing so)?
>
For Flash, we don't. For media using built-in decoders it is a store for buffering data; the decoding thread and the network/socket thread work asynchronously as consumer and producer respectively. But don't worry about it for Flash :)

[Steve Workman]

(Note: lists.mozilla.org seems to be having some issues this week according to a few colleagues, so these posts might not all appear. Feel free to email directly if you're concerned).

On May 18, 2012, at 4:17 AM, Alan Ogilvie wrote:

> Thank you all for your response so far - clearly this is going to be a
> somewhat complicated answer (I didn't believe that it wouldn't) ;-)

:) Yup, but we can work it out and de-complexify (new word maybe) it all.

>
> 3/ - yes, my (and somewhat 'our') hypothesis is that because chunk requests
> are made through the host browser, they will be restricted as any other
> internet content is - for example, requesting a web page, then all the
> assets from the HTML response, is clearly managed by limiting the number of
> concurrent 'threads' (I may obviously be using the wrong words here) to
> some sort of rule for 'same domain'. If we had more than one player in the
> browser instance (maybe on multiple tabs/windows) all requesting chunks
> from the same domain, but different 'channels' (ie. different streams),
> then they will likely 'fight' each other for spare slots in the concurrent
> 'threads'. We're not sure how much of an 'edge case' that is - though I can
> believe some of our viewers/listeners having a couple of windows open
> during big events (like the Olympics in London coming soon) with HDS
> playback in Flash.

That's a good example.

So, I've done so more poking around today, and this is what I can see in our networking code as far as resources for connections are concerned.

At a high level, we have 4 maximums for the number of connections (you can look at these in "about:config" - if you're unfamiliar with that, just type about:config in the address bar and tell Firefox that you'll be careful):

network.http.max-connections = 256
network.http.max-connections-per-server = 15
network.http.max-persistent-connections-per-proxy = 8
network.http.max-persistent-connections-per-server = 6

I'm guessing that proxy will matter for those folk watching the games from their workplaces, but there may be other scenarios. And of course, we're not limited to the Olympic games here.

These values are configurable, but for the vast majority of users, they won't touch them.

Once we get an HTTP request, we create a transaction and determine what to do next based on the following logic in nsHttpConnectionMgr::ProcessNewTransaction (currently found at http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpConnectionMgr.cpp#1641). Note that the transaction is created by the content thread and then dispatched to the socket thread for the following:

// 0 - If this should use spdy then dispatch it post haste.
// 1 - If there is connection pressure then see if we can pipeline this on
// a connection of a matching type instead of using a new conn
// 2 - If there is an idle connection, use it!
// 3 - if class == reval or script and there is an open conn of that type
// then pipeline onto shortest pipeline of that class if limits allow
// 4 - If we aren't up against our connection limit,
// then open a new one
// 5 - Try a pipeline if we haven't already - this will be unusual because
// it implies a low connection pressure situation where
// MakeNewConnection() failed.. that is possible, but unlikely, due to
// global limits
// 6 - no connection is available - queue it

So, deciding what to do with the transaction depends on what protocol it's using (SPDY, HTTP pipelining or non-pipelining), how many connections are free and the type of connection. If the connection is SPDY, the transaction is added as a stream if there is room (max of 100 streams per connection I think). Otherwise the transaction is queued until a slot opens up in the SPDY stream. So there are multiple layers in the decision-making to determine what is a relevant resource (num of sockets, num of streams, etc).

Some points:
"under pressure" means most of the allowed parallel connections are already used up.
"idle connections" are persistent connections, i.e. sockets that are kept open thereby avoiding the DNS and TCP RTT for subsequent requests.

Once we figure out what type of connection, the transaction is dispatched.

Some things I'm still exploring:
- We have one socket thread on which the previous decision about how to add a transaction is processed. I'm not sure if we have more socket threads for reading and writing or how we multiplex all of that - I'll find that out.
- There may be a max for the number of requests which can be pipelined.

>
> I have supplied an example of HDS through Flash which is live on our site
> at the moment. This may help you see how it's working in the real world.
> I've already poked around to see what happens to the chunks.

What have you observed if anything?

Steve.

[Steve Workman]

I'll let Nick and Michal comment on garbage collection for regular cache objects. We also have a media cache, which the guys on dev-media have a better understanding since they wrote it, but I don't think it's used for Flash, only for built-in decoders.
>

[Alan Ogilvie]

On this one - remember that in the case of HDS in Flash, the requests for
fragments are being bubbled up to the host browser HTTP stack - so would
this change your answer?

Further - why would we want fragments going into the media cache (i.e.
what's the benefits of doing so)?

On 21 May 2012 06:11, Robert O'Callahan <rob...@ocallahan.org> wrote:

> > I'll let Nick and Michal comment on garbage collection for regular
>
>> > cache objects. We also have a media cache, which the guys on
>> > dev-media have a better understanding since they wrote it, but I
>> > don't think it's used for Flash, only for built-in decoders.
>>
>

[Alan Ogilvie]

That's interesting - as we look towards MPEG-DASH profiles... which would
seem to be something that might be 'built-in' at some time in the future.

On 21 May 2012 17:03, Steve Workman <swor...@mozilla.com> wrote:

>
> On May 21, 2012, at 3:26 AM, Alan Ogilvie wrote:
>

> On this one - remember that in the case of HDS in Flash, the requests for
> fragments are being bubbled up to the host browser HTTP stack - so would
> this change your answer?
>

> Nope. Our HTTP stack and ram/disk cache are completely different entities
> to the Media Cache. I only mentioned it to get confirmation from one of the
> media guys. So, it wouldn't change the answer. Flash should not be using
> the Media Cache.
>

> Further - why would we want fragments going into the media cache (i.e.
> what's the benefits of doing so)?
>

> For Flash, we don't. For media using built-in decoders it is a store for
> buffering data; the decoding thread and the network/socket thread work
> asynchronously as consumer and producer respectively. But don't worry about
> it for Flash :)
>
>

[Alan Ogilvie]

Thank you for this so far - this is really useful.

Is the config setting -
'network.http.max-persistent-connections-per-server' - is that as per the
proxy setting? (or is there some good documentation on what these values
stand for)

Here's my understanding/guess at what these actually mean: (some of which
from some digging on google -
http://kb.mozillazine.org/Network.http.max-persistent-connections-per-proxy)

*) network.http.max-connections - total number of HTTP connections, of any
type, across any one instance of Firefox (that includes all tabs/windows
spawned by that instance) - i.e. a funky web page with 'real-time' polling
of http in one tab, would necessarily be in contention with another tab
using HDS. (some people have silly numbers of tabs open at once, I know I
do!)

*) network.http.max-connections-per-server - any one server, identified by
the FQDN prefix (e.g. stats.bbc.co.uk is considered different from
www.bbc.co.uk), can open HTTP connections up to this maximum (assuming the
max-connections hasn't been reached)

*) network.http.max-persistent-connections-per-proxy - if a proxy is
configured within Firefox (may include a system proxy configuration) then
this value controls the number of HTTP connections of type 'keep-alive'.
This value is still limited by max-connections-per-server and, in turn,
max-connections. [not sure if this is also limited by
max-persistent-connections-per-server, though the numbers you gave seem to
suggest this isn't - maybe it's possible to have
max-persistent-connections-per-proxy >
max-persistent-connections-per-server)

*) network.http.max-persistent-connections-per-server - this value controls
the number of HTTP connections of type 'keep-alive'. (When Firefox is not
configured with a proxy). This value is still limited by
max-connections-per-server and, in turn, max-connections.


Other things that come to mind...

*) Fragment URLs may be reused by other copies of the Flash 'HDS' client -
under browser based caching rules. Though not clear on how this works -
likely the headers on the fragments saying how long they live (not sure,
haven't sniffed those yet)

*) For the 'http.max...' configuration settings and current 'pipeline'
usage - it would be nice to see a couple of things live in Firefox: a) the
actual usage of the different HTTP slots for a particular tab/window b) the
total for the entire browser - this sounds like something difficult to see,
but given the general 'add-on' world of Firefox... is that something that
could be exposed? (not for public consumption perhaps)

*) Background Tabs - [theory] seem to throttle bandwidth, likely to reduce
impact on foreground tabs, resulting in Flash HDS players that employ an
adaptive bitrate algorithm to fall back to the lowest bitrate available.
Haven't tried this with windows in the background (without focus) - wonder
if that is also doing some intelligent throttling. (and when I say
'throttle' - perhaps it's simply that foreground tabs/windows get more
priority to the available http connections or that the transfers of the
HTTP requests are slower to avoid resource hogging)

*) I wonder if pipelining happens on the HDS fragment requests - is there a
'smoking gun' that makes this easy to spot. I'm assuming we'd prefer
pipelining, right?

*) Seems like these http.max... settings vary between Firefox versions, do
they vary on platform? (what's the 'size' of the 'problem', should I be
more concerned)

*) I am trying to speak to our contacts in Adobe to see what they think
about browser cache implications for HDS, and I suspect this will have an
impact on their MPEG-DASH thinking too. Do you know if you have had any
contact with them around this?

*) additional use of SPDY could speed up these sort of fragment requests -
though support seems limited. I'd not considered that before.

[Jason Duell]

[moving followups to dev.tech.network, as this seems to be more
network-focused at this point.]

On May 21, 12:17 pm, Alan Ogilvie <alan.ogil...@gmail.com> wrote:
> *) For the 'http.max...' configuration settings and current 'pipeline'
> usage - it would be nice to see a couple of things live in Firefox: a) the
> actual usage of the different HTTP slots for a particular tab/window b) the
> total for the entire browser - this sounds like something difficult to see,
> but given the general 'add-on' world of Firefox... is that something that
> could be exposed? (not for public consumption perhaps)

It certainly ought to be possible to write an addon to do this. I
don't know if anyone has.

> *) Background Tabs - [theory] seem to throttle bandwidth, likely to reduce
> impact on foreground tabs, resulting in Flash HDS players that employ an
> adaptive bitrate algorithm to fall back to the lowest bitrate available.

It's true that we prioritize the foreground tab's channels. But the
only notion of "priority" we have right now (this may need to change
at some point) is that higher-priority requests sit at the head of the
wait queue if there are more requests than available HTTP
connections. Once a request has been issued on the wire, we treat it
the same as any other, so you get whatever rate TCP winds up giving
you in the presence of multiple TCP streams being open (i.e. a bit of
a crapshoot). The API I talked about on dev.tech.network (that allows
you to set a channel to download with a 'smooth' constant rate) may
help with this. For now channel loading is 'bursty', but so long as
there's close to enough aggregate bandwidth for all the channels being
loaded, and you have decent amount of buffering, you'll be ok. You
may be right that this sort of thundering herd approach might make a
channel that loses the TCP bandwidth sweepstakes decide to pick a
lower-quality video than it needs to. We'll probably need smarter
logic to deal with this at some point.

> *) I wonder if pipelining happens on the HDS fragment requests - is there a
> 'smoking gun' that makes this easy to spot. I'm assuming we'd prefer
> pipelining, right?

Http pipelining mainly reduces latency, which is not really much of an
issue in bandwidth-dominated apps like streaming. You probably
*don't* want to pipeline stream requests, because that essentially
ties the 2nd and further requests to a particular connection, when
they could otherwise grab any available connection that becomes
available. Worse, if other, small resources needed for a page are
pipelined behind a stream, it will be a *long* time before they are
satisfied (I believe Patrick wrote some logic that separates different
content types onto different connections precisely to avoid this
issue, known as "head of line blocking").

> *) Seems like these http.max... settings vary between Firefox versions, do
> they vary on platform? (what's the 'size' of the 'problem', should I be
> more concerned)

The only cases that might cause you grief are for users using proxies
(8 max connections will be an issue if you want to show a number of
streams that approaches--or, obviously, exceeds--8. Mobile will also
be an issue until we increase the number of max-http connections there
(it's currently only 6, vs 256 on desktop). I can't imagine people
streaming more than a few videos on a mobile phone today, but on
android tablets it seems more likely. The likely solution here is to
increase the limits if we see trouble happening, and then see how
badly that breaks existing proxy servers and mobile phones (not much
fun, but I don't see a better fix).

> *) I am trying to speak to our contacts in Adobe to see what they think about browser cache implications for HDS, and I suspect this will have an impact on their MPEG-DASH thinking too. Do you know if you have had any contact with them around this?

none that I've heard of.

> *) additional use of SPDY could speed up these sort of fragment requests -
> though support seems limited. I'd not considered that before.

What sort of frequent requests? SPDY, like pipelining, is mainly
geared to avoiding the latency of lots of little requests, so it's
unlikely to make a difference with the large chunks involved in
streaming.

cheers,

Jason
Mozilla