how to handle cache and tmp correctly

[v.]

I have written a couple of apps in F3 now and am really getting into it.

However I feel I do not entirely understand how I should handle cache and tmp files.

Should I care about them at all or should I let the framework work its magic?

I have noticed in 1 app that after only 1 month the temp/cache folder contains 40k entries.

After modyfying code, the cache (and/or tmp?) makes that changes go through unpredictebly and slowly.

How do you guys handle these things?

[bcosca]

There shouldn't be any adverse effect if you decide to empty these folders. You can even have the tmp/ folder in tmpfs if  your server allows.

[v.]

Hi Bcosca,

I know you can delete the folders without problems, the question was rather if one needs to (programmatically) take this into account or does the framework handle things itself entirely.

I was rather surprised to fin 40k items in the cache after only 1 month, They took up tens of MB.

[bcosca]

Unfortunately, the framework has no background process for detecting deprecated/revised/unused templates and cleaning up these folders so you have to empty these folders yourself.

[ikkez]

If you like, you can use a different cache engine instead of the folder caching solution, which can care about cleanup itself.

[v.]

I am reopening this question.



For obvious reasons I do not wish to go and delete all my apps' cache folders manually.


I am contemplating to automatically (using a cronjob) delete all files older than 4 or 5 days (?) in the said directories.




Is this a proper solution according to you experts?


[ved]

Hi,

Yes, that is a valid solution. (a similar example is Debian linux which by default uses a cronjob/systemd timer to remove old php session files since the webserver user doesn't have permissions on the session folder in order to run the garbage collector)

That being said, as @ikkez already stated, I'd also recommend using any other cache engine like APCu, Memcached or Redis instead of the filesystem. Those should be faster and run their own garbage collection to remove old expired cached entries so you could avoid the cronjob.

I personally use APCu, it works great and it's pretty much as fast as you can get if you're not doing load-balancing or sharing the cache with other servers. For those setups, memcached or redis would probably be the better solution.

Hope it helps, good luck.

[v.]

Hi Ved,

Thanks for your input. 

I have no experience with these cache engines at all.

How would that change the current code? Would I need to rewrite every single line that contains an f3->SESSION variable or is it a simple configuration setting to use another cache engine?

My tmp folder contains the "parsed" view templates (these do not use the cache I assume?) and a cache folder that .@, .schema and .sql files.

[ved]

Hi,

You wouldn't need to change any of your code. How is your CACHE variable configured now? If it's simply "true" and it's automatically using the filesystem, then that should be all that F3's auto-detection got.

To find out if your server supports any of the mentioned engines is to simply create an info.php file with "<?php phpinfo();" inside it and access it through the browser. Then, search (ctrl+f) for "APC", "Redis", or "Memcache" and see if you have those extensions enabled.

If you have full control over the server, like a vps or dedicated server then you should be able to install any of the needed php extensions for whatever engine you want.

If you're using some sort of shared hosting and/or have no idea what a php extension is, then you'll have to find out what your current server supports with phpinfo() as stated above.

For example, one of my servers with APCu support:

As a last alternative, if you really can't use any of the memory based cache engines, then go ahead and create your cronjob as initially planned and it should still solve your main issue.

Cheers

[v.]

 Hi Ved,

Thanks for the message. I already performed the phpinfo and at the moment my VPS only has memcached installed.

The f3 cache was indeed defined as TRUE in my config.ini file.

I have changed it to CACHE="memcache=localhost:11211"
Everything seems to work in the exact same way as before. The files in the cache folder also seem to be lookin the same. I am not sure if this means it is working or if it means the default f3 cache engine is still being used for some reason

[ved]

Hi,

You'll need not only the memcached server running, but your PHP must also have the memcached module installed. For example, on Debian the package name is something like php-memcached. You can see if it's installed with phpinfo().

If you have both, then F3 should be using memcached. To test, delete all files inside your cache folder and see if anything gets recreated. If it does, something's probably not working correctly.

You can also install some sort of web dashboard to monitor memcached keys like this or this or this. Then you'll be able to see what keys are being created on memcached.

Cheers

[v.]

Thanks again for your much appreciated help Ved.

Apparently memcached plugin for php was installed, but not memcached itself....

I have contacted my hoster, they propose Varnish in stead because they have more experience with it. Is this also compatible?

[ved]

Hi,

No, not really.

Varnish is an http caching proxy. It caches the responses your webserver sends out (that is, whatever gets outputted by your F3 app). It is not a server where you can save your application's cache. You also don't integrate it with your app, Varnish sits between the webserver and the visitor.

Memcached is an in-memory key/value server. You use it, as the name implies, to save some values into some keys. You integrate it with your app directly (or through F3 in this case).

They both have different purposes.

I'm guessing your host won't support any of the memory based caching engines so you'll probably have to stick with the filesystem.

Cheers

[v.]

Thanks ved,

I had my hoster install apcu, seems to work now: there is no cache folder anymore.

With this php cache dashboard I am able to see entries created by apcu (they seem to be the same as in the previous cache folder).

Do I understand correctly that this now stores the cache values in memory and no longer file-based?

Kind regards,

v

[ved]

Hi, no problem.

That is correct, your app should now be caching into memory.

As for the dashboard, the official apcu repo also includes a simple script for that but it should achieve the same results as the one you're using.

Cheers.

[v.]

Sorry for this, but I am reopening this thread a second time....

I am still trying to get a better grip on how cache is supposed to work.

I have noticed that my cache dashboard (I still use this one: https://github.com/JorgenEvens/php-cache-dashboard) shows APCu entries.They are all *.schema entries which seem to have an expiration of 60 seconds and @-entries that have an indefinite expiration. These @-entries seem to pile up after a while.with now nearly 600 of them after a couple of months.

Is this how it's supposed to be?

What are these @-entries and why are they not expiring? If I try to view their content they appear to be empty.

Deleting them seems to clear the active sessions, however, when a session is destroyd programmatically they remain in the cache. Should I programmatically delete them from cache? How?

I can also see that the framework files and my classes, but also the interpreted template files are being cached by PHP Opcache and not by APCu

The template files are also stored in my tmp folder.
Are these items not also supposed to be cached by apcu?

[ved]

Hi,

The .schema entries are set by the framework and contain the database table's schema. This is probably to get used by the data mappers and you shouldn't really worry about it.

The @ entries are your sessions. If you have the cache enabled and are initializing the session class (new Session()) then your sessions will get stored in your cache engine. In this case it's in APCu.

As for cleaning up of old/expired sessions, this is usually dealt by the php garbage collector, which will have X% chance of running on each request. The php.ini parameters that control this are: session.gc_probability and session.gc_divisor.

Some operating systems (for example, Debian and probably also Ubuntu) disable the PHP garbage collector and replace them with a cronjob or systemd timer that will then clean the session files from the system.

This is due to permission issues and work correctly if you're using the regular PHP sessions but doesn't really work when you're using F3's session handlers to save the sessions. In that case you'll have to setup your own session.gc_* parameters.

Here's an example:

// Enable session garbage collection with a 1% chance of
// running on each session_start()
ini_set('session.gc_probability', 1);
ini_set('session.gc_divisor', 100);

The way this works is gc_probability/gc_divisor = probability of the garbage collector run.

As for the templates, the compiled files should be saved on your tmp folder so that should be correct.

PHP's opcache is an internal opcode caching (so, not some sort of userland cache) and not a caching system that should be used directly by developers.

Cheers.

[v.]

Thanks ved,the number of beers owe you is seriously stacking up!

The server runs ubuntu, so I understand a bit better what is going on now. However, why the difference between regular php sessions and f3 sessions?

I would think that in the background it really is the exact same thing?

[ved]

Hey, no problem.

Debian, and I'm guessing it's derivatives like Ubuntu etc change the way sessions are handled mostly to harden security but it's better to just show you their description.

For example, I'm using php7.3 (from the deb.sury.org repository but it's the same thing with the default debian packages) and on my systems there is this file that reads:

/usr/share/doc/php7.3-common/README.Debian.gz:

(...)
Session storage
----------------------------------------------------------------------
  Session files are stored in /var/lib/php/sessions. For security
  purposes, this directory is unreadable to non-root users. This means
  that PHP running from Apache HTTP Server, for example, will not be
  able to clean up stale session files. Instead, we have a cron job
  run every 30 minutes that cleans up stale session files;
  /etc/cron.d/php. You may need to modify how often this runs, if
  you've modified session.gc_maxlifetime in your php.ini; otherwise,
  it may be too lax or overly aggressive in cleaning out stale session
  files.
  WARNING: If you modify the session handling in any way (e.g. put
  session files in subdirectories, use different session handler), you
  always have to check and possibly disable or modify the session
  cleanup cron job that is located in /etc/cron.d/php.
(...)

Then there's the cronjob/systemd timer that runs a bash script that's located at "/usr/lib/php/sessionclean" which is run as a user that has the needed permissions to delete the old sessions.

On your index.php try setting:

ini_set('session.gc_probability', 1);
ini_set('session.gc_divisor', 1);

Then load your website once and check APCu again. Your old sessions should be gone. Once you confirm that's working, just set the values back as 1 and 100 so that it runs on 1% of requests or adjust to your liking.

Cheers

[v.]

thank you again, learned something new and interesting!

[v.]

More session/cache questions...

My application has the session.gc_probability and session.gc_divisor set as proposed by ved.

Everything runs fairly smoothly.

However..... I have noticed users get logged out automatically sometimes, while they shouldn't (the application automatically logs out users after 10 minutes, but it appears that at random times a user will get logged out earlier, I know that at least once 2 logged in users were simultaneously logged out early).

Could this be due to the garbage collector that just clears all sessions with a 1/100 chance (due to the settings of php session.gc_prob and divisor)?

This is not what I meant to happen!

I have read up on how f3 handles sessions and cache and apparently it is possible to set a ttl for cache entries. I could not find anything on ttl for session entries though. Is that possible?

Is it possible to delete only the expired cache values? In this case it may be a good idea to use this to automatically avoid old sessions stacking up in apcu?

[ved]

Hi,

Could be a few things, how are you logging out your users automatically after 10m?

Also check your session.gc_maxlifetime parameter. The garbage collector shouldn't delete active sessions afaik.

EDIT: also note that the session handler will destroy the session if at any time your users change ip or user agent. You can use the onsuspect callback to check for this.

[v.]

Could be a few things, how are you logging out your users automatically after 10m?

On every pageload, the first thing my controller does is:

            if(time() - $this->f3->get('SESSION.timestamp') > $this->f3->get('auto_logout'))
            {
                $this->f3->clear('SESSION');
                $this->f3->reroute('/login');
            }
            else {
                $this->f3->set('SESSION.timestamp', time());
            }

Also check your session.gc_maxlifetime parameter.

Is set to 1440

The garbage collector shouldn't delete active sessions afaik.

 

EDIT: also note that the session handler will destroy the session if at any time your users change ip or user agent. You can use the onsuspect callback to check for this.

I know this, this is why I have not set the ip6 AAAA records for my application, this will mess up sessions I found. I can not find a decent explanation why 2 users would simultaneously be logged out before the session timestamp limit.

I am a bit hazy on the custom session error handler: where would this code need to go? In the index file? In the controller? 

I do not undertand how it works.

My index contains:

....
$f3->session = new Session();
$f3->run();

My Controller:

    protected $f3;
    protected $db;
   
    function __construct() {
        $f3=Base::instance();
        $db=new DB\SQL(
            $f3->get('db_dns') . $f3->get('db_name'),
            $f3->get('db_user'),
            $f3->get('db_pass')
        );
        $this->f3=$f3;
        $this->db=$db;    
    }

    function beforeroute()
    {
        if($this->f3->get('SESSION.logged_in'))
        {
....

Where does the session error handler go?

[ved]

Hey, 

Well this will obviously require you to do some more testing in order to figure out what is happening.

I'm afraid that all I can do at this point is throw some suggestions up in the air.

Setup some application logging (hint: Log) inside your app and log whenever you write to session, login, logout, clear, etc. (this can help debug what was made by your app and not by any os/php setting or cron, etc)

Comment out F3's session handler and test if the behavior is the same with the regular php sessions. (the native php sessions also should have no issues running ipv4 and ipv6 at the same time afaik)

Try increasing gc_maxlifetime and also check the session.cookie_lifetime. (doubtful..).

Make sure you're not running out of space in APC or that your APC settings aren't somehow deleting the session entries. (also doubtful, but helps to remember that APC is a cache engine and not really a reliable store)

Also take a look at this SO answer, which may or may not help on your specific case but seems similar to what you're trying to achieve (maybe).

Can't really think of any more ideas at the moment. :-\

Good luck