Performance issues: >100 per cent cpu with only < 100 concurrent users

[Yoeri Nijs]

Hi all,

Currently, I am facing some serious performance issues with my F3 site. I don´t know what is going wrong here. Maybe it is related to F3, but it is also possible it is something else. In case of the latter: maybe one is able to share some experience with me :)

First some technical specs. I am running my site on a vps with 8 cores and 8.192 MB. The host itself runs on Ubuntu, but I am running F3 isolated in a separate Apache Docker container (this one).

Docker is not limited, which means the container is able to use all resources of the host (theoretically). My Apache configuration is as follows:

...

Timeout 30

KeepAlive On

MaxKeepAliveRequests 500 

KeepAliveTimeout 3

...

Furthermore, I've enabled F3 on-disk cache option, and also caching most of the static pages. Furthermore, most database queries are cached for several minutes to speed up some things.

Now the hard part. I am performing some load tests with loadster.io and found that even with 50 concurrent users the site itself begins to slow down. p90 load time hits a wopping 6 seconds, and when I am trying to increase the number of concurrent users, I am actually DDoS-ing myself.

When I am running Docker stats, I see that cpu is above 500 per cent, which is quite high. Memory, however, keeps on  a healthy diet, so that's no issue.

I tried some things already to speed up some stuff. For instance, I cleaned up some Apache modules, cleaned up .htaccess (which holds some records to block bots), turned off cron jobs, disabled the caching engine, rewrote all file operations to database operations, but nothing helps.

Please, can someone point me to the right direction here?

[Yoeri Nijs]

One important note here: I found out that performance issues still exist when I just serve a static page with F3.

Op zaterdag 20 mei 2023 om 14:10:30 UTC+2 schreef Yoeri Nijs:

[ved]

Hello,

Not sure what is happening of course but some things to consider and/or try:

1. Configure your webserver to NOT send any static files through the php process. Not sure if it can be done in Apache if you're using mod_php (see #2)
2. Replace Apache and mod_php (if that's what you're using) with nginx (or another webserver) that uses php-fpm instead. With this setup you can make nginx serve all static files directly (it's very fast) and only process actual php files. You can also configure the php-fpm process a bit better than mod_php imho.
3. Go into your apache docker container and see what is actually using the 500% cpu. Is it actually the apache process?
4. You state that your docker can use all system resources. If you can limit it, try to do so. It may be that docker itself is taking resources from the server and making every local process struggle for cpu time. For example if it's a quad core, make docker/apache use only 2 cores and try it again.
5. Your Apache configuration can also be set too high which can also hog the machine. Try reducing the limits (MaxClients for instance) and see how it handles the load.
6. Check your apache error logs if there's any hint on what's happening
7. Check your host system's system logs (/var/log/messages, /var/log/debug, /var/log/syslog, etc) to see if there's any relevant errors there. Also check the logs inside the docker container for the same indicators.
8. Start your stress tests with a lower number of clients while observing the system behavior (again, check log files, check top/htop, etc) and then slowly increase until it starts getting saturated. That may help figuring out the cause.
9. Make sure it's not your database slowing everything down. The database is most often than not, the cause for slow php apps. Check database memory limits, cache sizes, try running mysqltuner (if using mysql or alternatives otherwise). Run your queries on the mysql client or phpmyadmin etc and see how long they take. If you're having 1second+ queries, it's normal for the parent webserver/php modules to "hang" for that amount of time which will inevitably create a big request queue that should deplete server resources.
10. Removing or increasing limits on all services is not the correct way to handle these issues. You need sensible limits on the webserver and database server so they don't compete for machine resources.

That's pretty much all I can think now from the top of my head. You can maybe post some more information about what exactly your php code is doing or letting us know more data about the server, logs, etc and perhaps we can be of more assistance.

Hope it helps, good luck.

Cheers.

[Yoeri Nijs]

Hi Ved,

Thanks for helping me out. I'm still investigating the root cause, but I may have some insights that are interesting to you as well.

First, I disabled all F3 related stuff, and only served a static html page with Apache. This did not help, which suggested that the problem was actually Apache and not php/F3/custom code.

Secondly, I entered the Docker container and found a lot of apache2 jobs with 'top' when running the load test. It seemed the processes were not limited by something.

I guess MaxClients may help since I found out that this setting was not configured in my Apache2 conf. What are the consequences if I implement this? Does it mean that website visitors retrieve a timeout or something like that when there is a lot of load?

Thanks!

Op zaterdag 20 mei 2023 om 18:40:44 UTC+2 schreef ved:

[ved]

Hey,

It's normal for multiple processes to be present when you're stress testing the webserver.

That being said, each process will use a certain amount of memory and consume resources in other ways.

For example, if logging is enabled, all those processes are going to be writing logs to disk at the same time, which can potentially overload the storage controller.

I would reduce maxclients on apache and see how it behaves during stress. This should prevent Apache from launching so many processes and help if that's actually the issue.

But note that I haven't really used Apache in many many years so I'm not up to date with latest configurations or changes.

Also, on linux, sysctl.conf settings may need to be tweaked for maximum webserver throughput.

Cheers

[Yoeri Nijs]

Another update about the performance issues here. I found the culprit. After trying static html on Apache and a simple F3 app without db connection (with and without template engine), I discovered that the number of db connections is why the cpu and the response time are ridiculously high. I don't know, however, how I can tackle this issue.

I tried two more things with the simple app, that just executed one simple query with the F3's db exec method. First, I tried to disable the query cache since it may slow down database queries. That did not help. Then, I created a static class that reuses database connections. This did not help as well.

Is there something that may help? Even when adding more cpu and mem, the problem remains.

Op maandag 22 mei 2023 om 23:52:25 UTC+2 schreef ved:

[ved]

Hi,

I'd recommend you tune your database to work within the limits of your system, taking into account that besides the database, you're also using it to run the webserver and whatever other services may be running.

If you're using mysql, then using something like mysqltuner may help adjust parameters in order to keep the database from exhausting server resources. Other databases may have similar tools.

Essentially you'll have to balance and finetune the amount of memory each apache and db process uses for your specific case and limit each one to work within the limits of your server. At some point you'll just have to scale horizontally and be done with it, but make sure you're not optimizing prematurely.

Good luck, cheers.