Ways to rebuild category cache

[Alexander Obuhovich]

In-Portal rebuilds category cache after a change has been made to a category.

Right now cache rebuild is always made. Following rules apply after category has been changed:

• if user has less then 30 categories, then cache rebuild is made automatically using progress bar
• if user has 30 and more categories, then user is asked if he wishes to perform cache rebuild right now using progress bar and he can deny rebuild
• if "quick category cache rebuild" setting is enabled, then category cache is rebuild in background without asking a user or showing a rebuild progress bar

On one of my projects I saw a customization, when new setting was added "Automatically Rebuild Category Cache" was added and it was disabled.

This was done, because there were 3000 categories and they were changed very often, so answering No all the time to "rebuild category cache now" question became very annoying.

I'm proposing to create new configuration setting "Category Cache Rebuild Mode", that will replace "quick category cache rebuild" option with following options:

• Manual - user need to click on rebuild button in catalog (will use progress bar)
• Silent - cache will be rebuild in background (no progress bar)
• Automatic - ask/don't ask user (based on category count) and rebuild (if he agrees) with progress bar

Default setting will be set to Automatic for backwards compatibility.

--
Best Regards,

http://www.in-portal.com
http://www.alex-time.com

[Phil]

I'm 100% for it! I had many problems with users not rebuilding cache. and I even see no benefit to have Automatic or Manual mode, if it can performed silently in background, without annoying user, why don't do it every time it's needed?

Envoyé avec Sparrow

[Dmitry A.]

Phil, I guess you never worked with 3000 categories project :)

This is where it's good to have these options.

DA

[Phil]

Dmitry, that's right, I don't have websites with so many categories.

The main website I'm working on, as webmaster, have 390 categories, and 10 languages. Let me tell you that at this point, it's even impossible to edit 5 categories at the same time, otherwise everything explode when I try to save changes! Not to say that Debug mode need to eat more than 50mb (!) to finish its output, giving the need to change mysql config, just to be able to work correctly... If you are interested by categories bug, I can provide you some recipes, even with my small websites :)

And categories "not appearing after creating" is a frequent bug, even on standard install with 10 new categories added. And until pressing "Refresh", no chance to view them (nor recipe here sadly).

That's why I propose to do it silently. Every time. For every sizes.

Envoyé avec Sparrow

[Alexander Obuhovich]

We introduced progress bar specially to prevent memory leak on silent category cache rebuild.

Not using that progress bar could cause memory leaks.

[Phil]

to deep into my example, memory leak happen before categories are saved (temporary tables not closed in DB).

Anyway, I think that rebuild categories as frequently as possible, and in a background process (that user couldn't stop), his something important.

What are the pro and cons of a background process vs. progress bar?

Envoyé avec Sparrow

[Alexander Obuhovich]

Until cache is rebuild you don't see new categories you've added.

[Phil]

Then cache rebuild should happen silently on every category creation, isn't it? +1 for silent cache rebuild :)

Envoyé avec Sparrow

[Alexander Obuhovich]

And you'll get 100% memory leak. That's why I'm against that.

[Dmitry A.]

I support proposed idea:

Create new configuration setting "Category Cache Rebuild Mode", that will replace "quick category cache rebuild" option with following options:

• Manual - user need to click on rebuild button in catalog (will use progress bar)
• Silent - cache will be rebuild in background (no progress bar)
• Automatic - ask/don't ask user (based on category count) and rebuild (if he agrees) with progress bar

Default setting will be set to Automatic for backwards compatibility.

The only question that I have to Alex - what are the options of updating silently ONLY current category level so it does show up or it will break cache date integrity in some ugly way? 

DA

[Alexander Obuhovich]

There was a code before, that only updated categories in path of changed category (this and all it's parents), but this code stopped working, when we introduced TreeLeft and TreeRight columns in category to speed up detection of all category children recursively.

Basically "TreeRight - TreeLeft" = "sub-category count on all levels", but when only direct parents (instead of whole catalog) was rebuild we got into situation, when these numbers were incorrect, because they only counted these direct parents and not all category children.

I'm not sure if there are users, who restrict (using CATEGORY.VIEW permissions) what categories are visible to what admin users, but they are none, then we can always show all categories in admin console (without VIEW permission check) and rebuild category cache in background in cron. Since we won't be checking VIEW permission (which is retrieved from cache) admins will see new categories instantly (maybe with incorrect sub-category counters) and cache would rebuild in background. Front-end users will see old category structure, since they still would rely on category VIEW permission checking.

[Dmitry A.]

I think we should have CATEGORY.VIEW permissions and perhaps other permissions NOT checked/used by default.

What's the need of this to users when it's really used (80-90% not used). We can make default setup somewhat lightweight and allow to enable full Permissions via System Settings or something.

What do you think?

[Alexander Obuhovich]

But if users do enable them, then we're back as square one. We still need to decide what to do, when these permissions are enabled.

[Phil]

+1

[Dmitry A.]

Funny, just today we came across similar issue in a one project where Category Import on Front-end didn't show new categories in Front-end since Category Cache Rebuild didn't run (at it would in Admin).

Anyway, I believe we should proceed the following way:

Create new configuration setting "Section Cache Rebuild Mode", that will replace "quick category cache rebuild" option with following options:

• Manual - user need to click on rebuild button in catalog (will use progress bar)
• Silent - cache will be rebuild in background (no progress bar)
• Automatic - ask/don't ask user (based on category count) and rebuild (if he agrees) with progress bar

Default setting will be set to Automatic for backwards compatibility.

In the future, if we create ability to DISABLE Category.View permissions in Admin we can make Automatic option to behave accordingly (so it's not even triggered or similar).

Agreed?

DA

[Dmitry A.]

So what's your take on this?

DA

[Alexander Obuhovich]

Agreed. Just need to create task.

Also to speed up cache rebuild we need to change the way how it is rebuild. For example right now we dig into category structure very much and that's why it's impossible to split that process in sub-processes that will complete task faster.

[Alexander Obuhovich]

Here is the task http://tracker.in-portal.org/view.php?id=1318.

[Alexander Obuhovich]

Here is the patch, ready for testing.