Bug: Many cache hits not indicated in Net view

72 views
Skip to first unread message

Kyle Scholz

unread,
Jul 20, 2007, 8:35:59 PM7/20/07
to Firebug
Currently in Net view, only 304 responses are marked as cache hits.
Many cache hits never trigger an HTTP request and they're currently
displayed in dark gray.

Below is a diff on net.js that uses the cache lookup in
getFileSizeFromCache() to determine if the file was served from the
cache. A file *should* have been served from the cache if the entry
was available at the start of the entry was not updated as a result of
the request. I'm uncertain of the behavior when the caching policy
changes on a file that's currently in the cache -- That's a tricky
edge case.

$ diff net.orig.js net.js
1114,1115c1114
< if (file.size == -1)
< getFileSizeFromCache(file, this);
---
> getFileSizeFromCache(file, this);
1477c1476,1483
< file.size = descriptor.dataSize;
---
> if(file.size == -1)
> {
> file.size = descriptor.dataSize;
> }
> if(descriptor.lastModified && descriptor.lastFetched &&
> descriptor.lastModified < Math.floor(file.startTime/1000)) {
> file.fromCache = true;
> }

John J Barton

unread,
Jul 21, 2007, 1:58:53 PM7/21/07
to Firebug
Thanks Kyle...do you have a small test case that illustrates the issue
you are describing? If you can post it or put it in the issues site I
think it would be easier to understand the problem you are solving.
John.

Kyle Scholz

unread,
Jul 23, 2007, 8:33:45 PM7/23/07
to Firebug
Sure, assuming you're using default settings for your cache, this
should work:

Suppose you visit http://www.cnn.com/. If you're visiting for the
first time this hour, Firefox should request "main.js". With Firebug,
you can see that the "Expires" header in the response for this file
indicates that the newly created cache entry for this item should not
expire for an hour. Now click on a section heading like "World".
Firefox should NOT request main.js again. You can verify that it does
not with the LiveHTTPHeaders extension or Ethereal. Instead, the file
will delivered from the cache without issuing an HTTP request. Firebug
currently displays the loadtime for main.js in dark gray, indicating
that it was not served from the cache.

In net.js, you'll notice that "file.fromCache" is only set to true
when an HTTP request is made and 304 response is received.

I think the the Net tab would be even more informative if 304
responses were styled differently from non-request-issuing cache hits,
since this can be critical to performance tuning. I hope this helps.
I'm happy to put in some more effort on this if it seems useful.

-- Kyle

On Jul 21, 10:58 am, John J Barton <johnjbar...@johnjbarton.com>
wrote:

John J Barton

unread,
Jul 24, 2007, 11:57:26 AM7/24/07
to Firebug
Sorry Kyle, I don't understand enough about the net tab to help. I see
a bunch of stuff happen on cnn.com; I don't know what part is correct
and what part is not.
John.

On Jul 23, 5:33 pm, Kyle Scholz <kyle.sch...@gmail.com> wrote:
> Sure, assuming you're using default settings for your cache, this
> should work:
>

> Suppose you visithttp://www.cnn.com/. If you're visiting for the

Kyle Scholz

unread,
Jul 24, 2007, 2:41:07 PM7/24/07
to Firebug
According to Firebug documentation, light gray timeline entries
indicate that a file was retrieved from the cache. A file may be
retrieved from the cache for two reasons:

1) The file exists in cache and Firefox issues an HTTP request with an
If-Modified-Since header. The remote web server responds with a 304
status code and an empty body.

2) The file was placed in cache with a Cache-Control policy that
enables Firefox to retrieve the file from cache without making a
request to the remote web server -- as long as the cache entry has not
expired. In the CNN example, main.js is returned with the header
"Cache-Control: max-age=3600".

Firebug presently displays timeline entries for case 1 in light gray.
The second case always shows in dark gray, implying that the file was
just fetched from the remote server when it was not.

-- Kyle

John J Barton

unread,
Jul 24, 2007, 11:57:51 PM7/24/07
to Firebug
Ok, very nicely described...now your patch does what in relation to
these two cases?

And: why did you ask for different styling for 304? As I understand
it:
#1 304: light gray
#2 non-request-issuing cache hits & regular requests: dark gray.
So the confusion comes from two cases of #2 styling?

Kyle Scholz

unread,
Jul 27, 2007, 2:50:24 PM7/27/07
to Firebug
Hi John, Thanks for responding. With my patch, 304s and non-request-
issuing cache hits are displayed in light gray.

It would be useful to distinguish between 304s and non-request-issuing
cache hits, just so we can tell when a request was issued. Either
using different colors or indicating the response code for 304s would
suffice.

I've also added a Cache tab for results in Net that shows full cache
details for any cached file. I can pass this along if you're
interested.

-- Kyle

Kyle Scholz

unread,
Jul 27, 2007, 5:10:08 PM7/27/07
to Firebug
Here's a diff for the cache features I mentioned:
- Don't hide response codes for 304s
- Add a tab for each response in the cache, that shows insertion time,
access time, fetch count, etc.

-- Kyle

371c371
< if (file.status && file.status != 200 && file.status != 304)
---
> if (file.status && file.status != 200)


1114,1115c1114
< if (file.size == -1)
< getFileSizeFromCache(file, this);
---

> getCacheEntry(file, this);
1397c1396,1397
< endTime: null
---
> endTime: null,
> cacheEntry: []
1459c1459
< function getFileSizeFromCache(file, netProgress)
---
> function getCacheEntry(file, netProgress)
1476a1477,1478
> if(file.size == -1)
> {
1477a1480,1501


> }
> if(descriptor.lastModified && descriptor.lastFetched &&
> descriptor.lastModified < Math.floor(file.startTime/1000)) {
> file.fromCache = true;
> }

> file.cacheEntry = [
> { name: "Last Modified",
> value: getDateFromSeconds(descriptor.lastModified)
> },
> { name: "Last Fetched",
> value: getDateFromSeconds(descriptor.lastFetched)
> },
> { name: "Data Size",
> value: descriptor.dataSize
> },
> { name: "Fetch Count",
> value: descriptor.fetchCount
> },
> { name: "Device",
> value: descriptor.deviceID
> }
> ];
1488a1513,1518
> function getDateFromSeconds(s) {
> var d = new Date();
> d.setTime(s*1000);
> return d;
> }
>
1668a1699,1703
> ),
> A({class: "netInfoCacheTab netInfoTab", onclick: "$onClickTab",
> view: "Cache",
> $collapsed: "$file|hideCache"},
> "Cache" // todo: Localization
1694a1730,1734
> ),
> DIV({class: "netInfoCacheText netInfoText"},
> TABLE({class: "netInfoCacheTable", cellpadding: 0, cellspacing: 0},
> TBODY()
> )
1720a1761,1765
> hideCache: function(file)
> {
> return !file.cacheEntry || file.category=="image";
> },
>
1832a1878,1888
>
> if (hasClass(tab, "netInfoCacheTab") && file.loaded && !netInfoBox.cachePresented)
> {
> netInfoBox.cachePresented = true;
>
> var responseTextBox = getChildByClass(netInfoBox, "netInfoCacheText");
> if(file.cacheEntry) {
> this.insertHeaderRows(netInfoBox, file.cacheEntry, "Cache");

Mark Kahn

unread,
Jul 27, 2007, 8:44:14 PM7/27/07
to fir...@googlegroups.com
John -

There are two different cases where an item is fetched from the cache:

case #1: Firefox sends an HTTP request saying "I have a copy of this file that was modified at 5PM.  Do you have anything new?"  If the server does, it sends the new page.  If not, it responds with a 304 HTTP code and firefox uses its copy from cache.  There is still client--server communication here, just not nearly as much as downloading a full page.

case #2: Firefox has a cached file, usually an image, a .css file, etc that has a cache timeout set.  Firefox will not even bother sending a request to the server to see if this file is modified, it will just load it from its cache and display it.

Kyle is saying that before his patch, in case #2, the net tab treated case #2 as if it were NOT loading from the cache and instead actually being fetched from the server.

I agree with him that the two cache cases should be distinguished since they are different - one requires client--server communication and the other does not.

-Mark


On 7/27/07, Kyle Scholz <kyle....@gmail.com> wrote:

Here's a diff for the cache features I mentioned:
- Don't hide response codes for 304s
- Add a tab for each response in the cache, that shows insertion time,
access time, fetch count, etc.

-- Kyle

371c371
<         if (file.status && file.status != 200 && file.status != 304)
---
>         if (file.status && file.status != 200)
1114,1115c1114
<             if ( file.size == -1)
> > > > > > > >                         if( descriptor.lastModified && descriptor.lastFetched &&

John J Barton

unread,
Jul 29, 2007, 1:04:21 PM7/29/07
to Firebug
Ok all we need is a simple test case that illustrates all of (and
only) these cases so I can test the changes.

Kyle it would be really helpful if you sent diffs against branches/
explore. Otherwise I need more information about the patch since my
line numbers aren't the same as those on trunk.

John.

On Jul 27, 5:44 pm, "Mark Kahn" <cwol...@gmail.com> wrote:
> John -
>
> There are two different cases where an item is fetched from the cache:
>
> case #1: Firefox sends an HTTP request saying "I have a copy of this file
> that was modified at 5PM. Do you have anything new?" If the server does,
> it sends the new page. If not, it responds with a 304 HTTP code and firefox
> uses its copy from cache. There is still client--server communication here,
> just not nearly as much as downloading a full page.
>
> case #2: Firefox has a cached file, usually an image, a .css file, etc that
> has a cache timeout set. Firefox will not even bother sending a request to
> the server to see if this file is modified, it will just load it from its
> cache and display it.
>
> Kyle is saying that before his patch, in case #2, the net tab treated case
> #2 as if it were NOT loading from the cache and instead actually being
> fetched from the server.
>
> I agree with him that the two cache cases should be distinguished since they
> are different - one requires client--server communication and the other does
> not.
>
> -Mark
>

> On 7/27/07, Kyle Scholz <kyle.sch...@gmail.com> wrote:
>
>
>
> > Here's a diff for the cache features I mentioned:
> > - Don't hide response codes for 304s
> > - Add a tab for each response in the cache, that shows insertion time,
> > access time, fetch count, etc.
>
> > -- Kyle
>
> > 371c371
> > < if (file.status && file.status != 200 && file.status != 304)
> > ---
> > > if (file.status && file.status != 200)
> > 1114,1115c1114

> > < if (file.size == -1)
> > < getFileSizeFromCache(file, this);
> > ---

> ...
>
> read more »

Kyle Scholz

unread,
Aug 1, 2007, 3:04:45 AM8/1/07
to Firebug
Hi John, I wrote up a simple test on my web server. This will be
easiest to follow if you have LiveHTTPHeaders installed:

http://kylescholz.com/projects/firebug/test-cache.html

Using the Firebug Net tab, you can see that two additional files were
request, a.cgi and b.cgi.

- a.cgi has "Cache-Control: private" and "Expires: Sun, 27 Jul 2008
00:00:00 GMT". Firefox should not request this file again until July
27th 2008 or when it is forced with "refresh".

- b.cgi has "Cache-Control: private" and "Expires: Sun, 01 Aug 2001
00:00:00 GMT". Firefox will issue a new request for this file the next
time it is needed.

Open LiveHTTPHeaders and put it in capture mode. Click the link for
"Test Page 2". The same two files are referenced on this page, but
Firefox will have only requested b.cgi. You can now verify this with
LiveHTTPHeaders. My server issued a 304 response to the request for
b.cgi because the file has not been modified.

Now, on the Net tab in Firebug, a.cgi is displayed in dark gray and
b.cgi is displayed in light gray. Both files were retrieved from the
cache, but a request was issued for b.cgi.

With the change I provided, both entries are displayed in light gray,
the entry for b.cgi indicates that the cache lookup resulted from a
304 response (the status code is displayed next to the file name), and
both entries now have a "Cache" tab that shows the attributes of the
corresponding cache entry.

Please let me know if there are questions. Thanks,

-- Kyle

On Jul 29, 10:04 am, John J Barton <johnjbar...@johnjbarton.com>
wrote:


> Ok all we need is a simple test case that illustrates all of (and
> only) these cases so I can test the changes.
>
> Kyle it would be really helpful if you sent diffs against branches/
> explore. Otherwise I need more information about the patch since my
> line numbers aren't the same as those on trunk.
>
> John.
>
> On Jul 27, 5:44 pm, "Mark Kahn" <cwol...@gmail.com> wrote:
>
> > John -
>
> > There are two different cases where an item is fetched from thecache:
>
> > case #1: Firefox sends an HTTP request saying "I have a copy of this file
> > that was modified at 5PM. Do you have anything new?" If the server does,
> > it sends the new page. If not, it responds with a 304 HTTP code and firefox

> > uses its copy fromcache. There is still client--server communication here,


> > just not nearly as much as downloading a full page.
>
> > case #2: Firefox has a cached file, usually an image, a .css file, etc that

> > has acachetimeout set. Firefox will not even bother sending a request to


> > the server to see if this file is modified, it will just load it from its

> >cacheand display it.


>
> > Kyle is saying that before his patch, in case #2, the net tab treated case

> > #2 as if it were NOT loading from thecacheand instead actually being
> > fetched from the server.
>
> > I agree with him that the twocachecases should be distinguished since they


> > are different - one requires client--server communication and the other does
> > not.
>
> > -Mark
>
> > On 7/27/07, Kyle Scholz <kyle.sch...@gmail.com> wrote:
>

> > > Here's a diff for thecachefeatures I mentioned:


> > > - Don't hide response codes for 304s

> > > - Add a tab for each response in thecache, that shows insertion time,

> > > > issuingcachehits are displayed in light gray.


>
> > > > It would be useful to distinguish between 304s and non-request-issuing

> > > >cachehits, just so we can tell when a request was issued. Either


> > > > using different colors or indicating the response code for 304s would
> > > > suffice.
>

> > > > I've also added aCachetab for results in Net that shows fullcache


> > > > details for any cached file. I can pass this along if you're
> > > > interested.
>
> > > > -- Kyle
>
> > > > On Jul 24, 8:57 pm, John J Barton <johnjbar...@johnjbarton.com> wrote:
>
> > > > > Ok, very nicely described...now your patch does what in relation to
> > > > > these two cases?
>
> > > > > And: why did you ask for different styling for 304? As I understand
> > > > > it:
> > > > > #1 304: light gray

> > > > > #2 non-request-issuingcachehits & regular requests: dark gray.


> > > > > So the confusion comes from two cases of #2 styling?
>
> > > > > On Jul 24, 11:41 am, Kyle Scholz <kyle.sch...@gmail.com> wrote:
>
> > > > > > According to Firebug documentation, light gray timeline entries

> > > > > > indicate that a file was retrieved from thecache. A file may be
> > > > > > retrieved from thecachefor two reasons:
>
> > > > > > 1) The file exists incacheand Firefox issues an HTTP request with


> > > an
> > > > > > If-Modified-Since header. The remote web server responds with a 304
> > > > > > status code and an empty body.
>

> > > > > > 2) The file was placed incachewith aCache-Control policy that
> > > > > > enables Firefox to retrieve the file fromcachewithout making a
> > > > > > request to the remote web server -- as long as thecacheentry has

> > > > > > > > indicates that the newly createdcacheentry for this item


> > > should not
> > > > > > > > expire for an hour. Now click on a section heading like "World".
> > > > > > > > Firefox should NOT request main.js again. You can verify that it
> > > does
> > > > > > > > not with the LiveHTTPHeaders extension or Ethereal. Instead, the
> > > file

> > > > > > > > will delivered from thecachewithout issuing an HTTP request.

> > > > > > > > > > Manycachehits never trigger an HTTP request and they're


> > > currently
> > > > > > > > > > displayed in dark gray.
>

> > > > > > > > > > Below is a diff on net.js that uses thecachelookup in


> > > > > > > > > > getFileSizeFromCache() to determine if the file was served
> > > from the
> > > > > > > > > >cache. A file *should* have been served from thecacheif
> > > the entry
> > > > > > > > > > was available at the start of the entry was not updated as a
> > > result of
> > > > > > > > > > the request. I'm uncertain of the behavior when the caching
> > > policy
>

> ...
>
> read more »

John J Barton

unread,
Aug 3, 2007, 12:14:03 PM8/3/07
to Firebug
Thanks. Using the test case I was able to apply and verify the first
small diff you posted. But the longer diff has line numbers that don't
match mine. I guess you used the 1.05 source. I use the 1.1 source at
branches/explore/content/firebug/net.js revision 55.

John.

> ...
>
> read more »

Reply all
Reply to author
Forward
0 new messages