cache effectiveness testing

[Taras Glek]

Hi,
As part of the Snappy effort, we asked QA to compare cache effectiveness
of Firefox against Opera (which some people claim has a superior cache).
We have some preliminary data now.


The testplan is described at
https://wiki.mozilla.org/Performance/Snappy/Testing:BFCache_Sprint#Reference_Test_Case

I put up a snapshot of the results at
http://people.mozilla.com/~tglek/snappy/cache1.html

I was suspecting that our cache hit-rates may be worse, but the data
proved that theory wrong.

Data indicates that either writing to our cache is expensive or we are
not http-pipelining/etc properly. My theory is that disk cache is to
blame, so I asked QA to rerun this comparing to Firefox to Firefox with
disk cache disabled.

There is also something horrible going on with huffingtonpost.

Taras

[Taras Glek]

No-disk-cache followup appears to confirm that writing cache absolutely
annihilates pageload perf.

http://people.mozilla.com/~tglek/snappy/cache2.html

I wonder if this is a symptom of
https://bugzilla.mozilla.org/show_bug.cgi?id=695399

Taras

[Taras Glek]

Sorry for the spam, I meant
https://bugzilla.mozilla.org/show_bug.cgi?id=717761.

Taras

[Brian Smith]

Taras Glek wrote:
> On 1/17/2012 2:55 PM, Taras Glek wrote:
> >> The testplan is described at
> >> https://wiki.mozilla.org/Performance/Snappy/Testing:BFCache_Sprint#Reference_Test_Case

Thank you for this.

I would suggest the following changes to the test methodology for future tests:

1. Document how the disk cache is cleared.
2. Have the user clear the disk cache and then reboot the machine and then run the test.
3. Document which anti-virus is installed and what it's configuration is. In particular, is it set to scan when files are written? (I think that the baseline should be Microsoft Security Essentials in its default configuration, which should have such scanning enabled by default.)
4. Reboot between switches in browsers. (maybe not too important)
5. Add Chrome and/or IE.

It would be great to have comparisons where MSE is enabled in its default configuration and where MSE is disabled.

> > No-disk-cache followup appears to confirm that writing cache
> > absolutely annihilates pageload perf.
> >
> > http://people.mozilla.com/~tglek/snappy/cache2.html

Are the "Firefox, no-cache" numbers comparable to the first column (Run 1, Firefox) or the third column (Run 2, Firefox)? That is, were the "no-cache" numbers generated with an empty cache or with the cache in the state it would be in after Run 2?

Here is the test procedure as far as I understand it:

Run the test for all three sites in Firefox
* This is "Firefox, Run 1"
Quit Firefox and start Opera
Run the test for all three sites in Opera
* This is "Opera, Run 1"
Quit Opera and start Firefox
Run the test for all three sites in Firefox
* This is "Firefox, Run 2"
Quit Firefox and start Opera
Run the test for all three sites in Opera
Quit Opera

Thanks again,
Brian

[Anthony Hughes]

Thanks for the additional details.

"Firefox, no cache" was tacked on at the end. We ran the original 4 testruns in order, then ran the "no cache" scenario on a new profile after the fact.

I hope that answers your question. Let me know if not.

[Anthony Hughes]

> It would be great to have comparisons where MSE is enabled in its default configuration and where MSE is disabled.

I assume we could have done this but we decided to keep this as simple as possible due to time/resource constraints and to remove any invariabilities in developing a baseline.

[Brian Smith]

Anthony Hughes wrote:
> "Firefox, no cache" was tacked on at the end. We ran the original 4
> testruns in order, then ran the "no cache" scenario on a new profile
> after the fact.
>
> I hope that answers your question. Let me know if not.

Thanks for the replies Anthony.

One big variable is whether or not the cache has to create the initial cache files during the test run. When you create a new profile, there is additional work that the cache has to do, that is probably done partially during startup and partially during the processing of the first request. It is good to measure this, but it is a variable that has to be controlled and accounted for.

Similarly, even when the cache exists, there is a bunch of work that has to be done the first time it is read from and/or the first time it is written to in a session. The cost of this one-time(-ish) work is reflected in the amazon.com scores, I guess. Good to keep in mind when looking at the numbers.

- Brian

[Jason Duell]

It's great to be getting these numbers--thanks!

Just to be clear--so "Run1" is from a new cache/profile. Is "Run2"
using a populated cache, or from an existing cache you've cleared? I'm
guessing the former from the test description page. In that case it's
definitely worth noting that no-cache is about as fast (or faster) then
the cache-hit case for many pages (the bigger the page--ex the home
pages--the more likely caching seems to help). We'd want to break down
the page content to see what's up with that (loading non-cachable or
small items that need validation is probably always going to be slower
for a disk cache, given the lookup overhead, so that's excusable.
Otherwise we have perf work to do).

I'm sure you've sunk lots of time into this already, but it'd be great
to see Chrome numbers--they seem to be outperforming us the most in the
tomshardware Browser contests.

Jason

On 01/17/2012 04:50 PM, Anthony Hughes wrote:
> Thanks for the additional details.
>

> "Firefox, no cache" was tacked on at the end. We ran the original 4 testruns in order, then ran the "no cache" scenario on a new profile after the fact.
>
> I hope that answers your question. Let me know if not.
>
>

> _______________________________________________
> dev-tech-network mailing list
> dev-tech...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-network

[Taras Glek]

On 1/18/2012 3:26 PM, Jason Duell wrote:
> It's great to be getting these numbers--thanks!
>
> Just to be clear--so "Run1" is from a new cache/profile. Is "Run2"
> using a populated cache, or from an existing cache you've cleared?
> I'm guessing the former from the test description page. In that case
> it's definitely worth noting that no-cache is about as fast (or
> faster) then the cache-hit case for many pages (the bigger the
> page--ex the home pages--the more likely caching seems to help).
> We'd want to break down the page content to see what's up with that
> (loading non-cachable or small items that need validation is probably
> always going to be slower for a disk cache, given the lookup overhead,
> so that's excusable. Otherwise we have perf work to do).
>
> I'm sure you've sunk lots of time into this already, but it'd be great
> to see Chrome numbers--they seem to be outperforming us the most in
> the tomshardware Browser contests.

I think this test data highlights significant deficiencies in our cache
performance that we should address asap. Chrome data would be amusing to
look at, but I don't think it's important to get to fix our cache in the
near term.

[Nick Hurley]

On Wed, Jan 18, 2012 at 4:08 PM, Taras Glek <tg...@mozilla.com> wrote:

> I think this test data highlights significant deficiencies in our cache
> performance that we should address asap. Chrome data would be amusing to
> look at, but I don't think it's important to get to fix our cache in the
> near term.
>

I think the Chrome numbers would actually be even more useful, given how
close in design Chrome's cache is to ours. Given that data, it might help
narrow down whether the deficiencies are because of the actual design of
the cache or whether they're particular to our specific implementation.

[RIUM+]

Since we're just trying to isolate the impact of caching & nothing
else, it may be advisable to perform an initial run through the sites
in a Private Browsing session, to prime the network connection. This
would remove variability related to caching DNS requests & other
network issues. That can easily add a few hundred milliseconds of
delay to whichever software package you test first (in this case
Firefox run 1).

[Patrick McManus]

assuming you want to measure the benefit/savings of response caching you
don't want to bypass dns and connection setups - while variable, they
are a big part of the caching savings and the reason you want to cache!

you also need to measure/model it across a variety of RTTs - the benefit
varies according to the network - and decide what ranges are important
to you. Which makes sense becuase the value of caching a slow object is
higher than caching a fast object. (and I personally think that should
be part of the cache-replacement function.) Cache benchmarks taken over
unshaped lans and localhost are just silly - its all overhead without
any upside.

If you're just trying to measure overhead, that's another story.

On 1/20/2012 11:29 PM, RIUM+ wrote:
> Since we're just trying to isolate the impact of caching& nothing

> else, it may be advisable to perform an initial run through the sites
> in a Private Browsing session, to prime the network connection. This

> would remove variability related to caching DNS requests& other

> network issues. That can easily add a few hundred milliseconds of
> delay to whichever software package you test first (in this case
> Firefox run 1).