[chromium-dev] where is the code related to "Clear cache menu" in chrome://net-internals

[Simon YoungKi Hong]

Hi,

I'm trying to implement disk-cache-hard-limit.

To do that, I'm reading the code in ./net/disk_cache/.

During the navigation in the net-internals, I found the "Clear cache menu"

So, I think that point could be a good start point.

Could someone advice the code point?

Also any advice will be appreciate for implementing disk-cache-hard-limit.

Thank you!

Simon.

[Fred Akalin]

Does --disk-cache-size not do what you want?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
 
 
 

[Simon YoungKi Hong]

I want to limit the size strictly.
That means cache cannot over the limit I configure.
Current option is over that limit.

2013. 4. 25. 오후 4:20에 "Fred Akalin" <aka...@chromium.org>님이 작성:

[Simon YoungKi Hong]

My initial idea was calling TrimCache() when cache entry is created or data is written.

But when I call TrimCache() Cache directory is not cleaned immediately.

[Ricardo Vargas]

The limit is not a hard limit because enforcing a hard limit affects normal browsing. "Stopping" to do evictions while trying to load a page degrades the user experience.

TrimCache() would be the point to change that behavior.

[Simon YoungKi Hong]

Thank you Ricardo.

Does writing data into the entry occur during  the resource loading before sendng to renderer?
If that, it can degrade loading performance.

2013. 4. 26. 오전 3:08에 "Ricardo Vargas" <rva...@chromium.org>님이 작성:

[Simon YoungKi Hong]

When I call TrimCache(true), header_->num_bytes are set to zero.

I thought that value is overall cache size.

But when I check Cache dir size (set by --disk-cache-dir) after calling TrimCache(), it was not changed.

How could I decrease the directory size, too?

Thank you.

Simon.

[Ricardo Vargas]

The value reported on the header is the storage used on user data, not the overall size of the directory. If you start with an empty cache, both values are not far apart (unless you care about KB), but if the cache was already using considerable space, the total used space will not get close to zero. TrimCache(true) basically means "remove all entries from the cache"... the directory size may or may not go down.

[Simon YoungKi Hong]

I saw the Cache directory after using browser for some time.

There are many f_xxx files , 4 data files and 1 index file.

When TrimCache is called(), all f_xxx file is removed.

But the size data files and index files seems not changed.

When I click Clear cache button in net-internal, these files are also initialized.

(I found it is recreated not reduced)

These five files are just grow?

Thank you Ricardo for your kind response.

[Simon YoungKi Hong]

I read the "Disk Cache" in Design Document and and related codes.

I found that there are two kind of files block files and separate files.

When TrimCache() is called, separate files are removed but block files are just delete its entry.

So block files didn't shrimp again.

If my understand is right, we can't decrease the block file's size (only RestartCache() can!).

How about making most of entry as a separate file?

Then, we can more decrese the cache directory size. (Of course, there is trade-off) 

[Ricardo Vargas]

Block files are removed when empty (as long as they are not the first one for each type of file)... that happens when trimming but also during normal operation. They are never truncated though.

I'm not sure what is your motivation, or what exactly are you trying to do (if you want to modify your own version of Chromium etc). From the point of view of Chrome, there is no real user control of the size of the cache (command line options are not really user features), so the use case of the cache shrinking significantly at some point after use is not really strong.

We set the size limit relatively up front. Today, that should be done depending on the hardware and specific platform requirements. If you're running on a platform where the default logic to determine the cache size is not appropriate, just modify that logic.

We could compact and truncate block files as needed, but that is a complicated operation that is not fully justified if we don't expect the cache to shrink abruptly for a given user (as is the case today).

[Simon YoungKi Hong]

I'm trying to run chromium on embedded system.

We can expect cache write error when cache partition is full.

So, I think hard limit will be useful when chromium run  with fixed size nand partition for cache directory.

[Ricardo Vargas]

It would seem that for that use case there's no need to reclaim space from the cache... just set the initial size to a reasonable size. Note that it is possible to estimate the typical overhead for a given cache size... and if there's a semi-dedicated partition, a small adjustment to the logic that determines the space to use based on the available space should be enough.

[Simon YoungKi Hong]

Sorry Ricardo, I can't understand your reply. I have some questions...

What it means the "set initial size"? Is it size set by "--disk-cache-size"?

And what the typical overhead means? does this cache amount above total size( by setting --disk-cache-size)?

[Simon YoungKi Hong]

Dear Ricardo,

Please ignore my previous mail.

For example, I have a 130MB nand partition for disk cache.

If typical overhead is 20MB, I would set disk-cache-size to 110MB.

(As I understand, the meaning of the overhead is caching size befere TrimCache() is started after 110MB is over.)

So my disk cache size would not exceed 130MB by calling TrimCache(false).

Does I understand your meaning correctly?

Thank you.

Simon.

[Ricardo Vargas]

The "standard" way of selecting the cache size is to not pass a value at all. When the cache receives 0 as the size (for example, when no command line argument is used), there's a routine that calculates how much space to use (PreferedCacheSize()). That code assumes that the cache is sitting on a volume that is shared with other things, because the normal scenario is that the same volume will hold at least all the user's data. This means that if it sees 130 MB, it will try to be quite conservative about how much to use.

If you have a dedicated volume, the simplest fix would be to alter that function to use a higher percentage of the volume. But yes, that would be functionally equivalent to force a size of maybe 110 MB through the command line.

There are two types of overhead: one fixed, and one variable (for lack of a better name). For a cache that size, the fixed overhead should be about 288 KB. The variable overhead is harder to calculate, in part because it is not really overhead. The point is that some files grow at fixed increments, and there is some wasted space depending on the size of the data to store. For example, one of those files stores data whose total size is between 4 KB and 16 KB. That file grows at steps of 4 MB (when it is full, it is extended by 4 MB). However, having a file of 20 MB doesn't really mean 20 MB of overhead, even if that space is still in use after clearing the cache. The actual overhead is the part of that file that is not in use, and that should be close to 1/8th of that file (of course, right after cleaning the cache the file would not have used data so it would be 100% overhead).

The advantage of using PreferedCacheSize() and not a hard codded value is that each time a session starts, a new target size is calculated based on the current available size, so there's no real need to predict the overhead correctly.

[Simon YoungKi Hong]

Thank you Ricardo!

I understood the policy of disk cache regarding the max_size_.

As you told me that disk cache calculated max size conservatively (without using disk-cache-size options).

So I think it has no problem about exceeding the size of fixed partition. 

To measure more larger available size I can use (I want to use more than 80MB if possible in 128MB partition),

I tested with dedicated partitions(128MB) w/o disk-cache-size.

kDefaultCacheSize (80MB) is used as written in the source code (PreferedCacheSize()).

In my the test, total cache directory size sometimes grew to almost 100MB and reduced to almost 80MB during the navigation and this pattern is repeated.

So, I have a question.

When the TrimCache(false) is called?

After some code navigation, TrimCache(false) is called in the dtor of EntryImpl.

And it is originally called by HttpCache::OnProcessPendingQueue().

In that code, this commend is written "// If no one is interested in this entry, then we can deactivate it."

I don't have much knowledge in the ./net/http/.

Would you please explain a little bit about the meaning of this point ("if no one is interested")? Is this the end of writing or reading from entry?

When I know the exact point when TrimCache() is called, I could measure exact size what the maximum size I can use.

BR,

Simon.

[Ricardo Vargas]

A disk cache entry is alive until the last network request for that URL goes away (completed or cancelled). There may be multiple requests for a given URL, and they end up accessing the same EntryImpl object: either because the same URL is used multiple times on the same page or because another tab is accessing the same URL.

If you load a heavy page, there may be 100 individual requests, so there should be about 100 entries being accessed for that page. There may be multiple network reads for each of those requests, and we keep track of how each of those reads affect (increase) the total space used. However, we don't want to generate a ton of signals of "it is time to free up some space" while loading a page, so we only call TrimCache when we're done with an entry (the last request that accesses one resource). On the other hand, we will also notice that at this time we are loading 100 resources so we'll probably avoid adding more load to the system, and the net effect is that we'll postpone actual evictions until we are not so busy.

If your tests show that the cache is growing too much before going back to the intended size, you could try increasing kCleanUpMargin so that once we start evictions we free up more space, and/or reduce kMaxDelayedTrims so that we don't wait that long to force evictions when busy... however, everything has a price so the more aggressive evictions are, the higher the chance "real" accesses are slower because the system is busy deleting stuff.

[Simon YoungKi Hong]

Thank you! Ricardo.

Finally, I made one modification.
It is a delay time of starting TrimCache().
Chrome can start TrimCache() 5 mins later after starting browser.
I saw that caching is over our limit before reaching 5 mins because our limit is so small.
So we adjusted that value to lower.

Does this worth to making a patch triggering TrimCache() when limit is much over before 5 mins?

Thank you for your kind response.
Simon.

2013. 5. 3. 오전 4:17에 "Ricardo Vargas" <rva...@chromium.org>님이 작성:

[Ricardo Vargas]

Start up time is important so it is better to make sure that there's no extra work at a time when a lot of things are going on (maybe other applications are also starting up, with the whole OS). For a system with more predictable use patterns, reducing that timer sounds quite reasonable.

[Simon YoungKi Hong]

How about introducing this option --disk-cache-hard-limit?

If this option is set, chromium will call TrimCache() when Cache size is over the HighWaterMark(max_size + more) regardless of start up time.

Many thanks, Ricardo!