Apache2 hates my Django deployment method

[Mike Dewhirst]

Server Version: Apache/2.4.52 (Ubuntu 2022.04) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10

Server MPM: event

Server Built: 2022-09-30T04:09:50

Apache *seems* to hang onto something and eventually crashes after I reload it after deploying a Python file.

That is either scripted (delete site and re-export from repo) or drag and drop a py file using Filezilla on Windows.

It *seems* ok if I redeploy html templates.

I have asked on the Apache mailing list and the consensus seems to be that something is "hogging" the mpm-event workers and I have to figure out what and solve that.

The only solution I have is to restart Apache.

I have to think this is happening in other projects so I'm wondering if anyone has solved this?

Thanks for any feedback

Mike

-- 
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

[Jason]

AFAIK, restarting apache process on deployment is standard, is there a reason you don't want to do that?

Templates are picked up because they're not evaluated and compiled, rather they're processed dynamically so any changes there do not require a restart.

[Mike Dewhirst]

On 12/12/2022 1:33 am, Jason wrote:

AFAIK, restarting apache process on deployment is standard, is there a reason you don't want to do that?

With Ubuntu 2020.04 and the version of Apache which came with that, reload worked fine and I always thought that was the right way to redeploy.

The newer version of Apache or probably more particularly the newer mpm-event module had a bug whereby ...

[Sun Dec 11 02:09:18.196035 2022] [mpm_event:error] [pid 893:tid 139808992921472] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.

... and that consumed resources until Apache crashed. Hence my questions on the Apache list which resulted in adjusted mpm-event config recommended by gurus.

Currently, reloading triggers the above error about once per second.

Then issuing a restart generates ...

[Sun Dec 11 02:15:28.478942 2022] [mpm_event:notice] [pid 893:tid 139808992921472] AH00492: caught SIGWINCH, shutting down gracefully
[Sun Dec 11 02:15:31.840212 2022] [core:warn] [pid 893:tid 139808992921472] AH00045: child process 909 still did not exit, sending a SIGTERM

... and that [core:warn] message repeats for the same pid but a bunch of different child processes for however many there are until it changes to a [core:error] for the same bunch ...

[Sun Dec 11 02:15:37.883824 2022] [core:error] [pid 893:tid 139808992921472] AH00046: child process 7868 still did not exit, sending a SIGKILL
[Sun Dec 11 02:15:50.816477 2022] [mpm_event:notice] [pid 45163:tid 140392171792256] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations

... until Apache restarts.

So I reckon there is something wrong.

I now restart Apache on redeployment but it still goes through the [core:warn] and [core:error] before restarting if I have done a reload.

I guess I'll give up reloading.

Thanks for responding

Cheers

Mike

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/8853bc85-97fb-48d7-af94-0807f777c0fan%40googlegroups.com.

[Jason]

https://serverfault.com/questions/1058677/apache-2-4-37-reload-causes-error-scoreboard-is-full-not-at-maxrequestworkers 

and 

https://serverfault.com/questions/1084635/apache-mpm-workers-stuck-in-g-gracefully-finishing-growing-scoreboard-is-fu

seems reloading is problematic because

>When you gracefully reload Apache, you're terminating the containing process. What is happening here is Apache is making each thread wait until all the threads in the containing process have completed prior to restarting the container PID.

> So, in you're case, you've got a single thread contained in all the processes in that list that is still busy or stuck somehow.

[Mike Dewhirst]

On 12/12/2022 11:32 am, Jason wrote:

https://serverfault.com/questions/1058677/apache-2-4-37-reload-causes-error-scoreboard-is-full-not-at-maxrequestworkers 

and 

https://serverfault.com/questions/1084635/apache-mpm-workers-stuck-in-g-gracefully-finishing-growing-scoreboard-is-fu

seems reloading is problematic because

>When you gracefully reload Apache, you're terminating the containing process. What is happening here is Apache is making each thread wait until all the threads in the containing process have completed prior to restarting the container PID.

> So, in you're case, you've got a single thread contained in all the processes in that list that is still busy or stuck somehow.

I had seen the first of those links and abandoned reload in logrotate. I now copytruncate instead with no restarting. That was advice from the very friendly Apache list.

I don't think I have the brainspace to identify the actual cause. My best strategy (I think) is to restart on redeployment and wait for for Apache to self-heal. If it is Apache's fault.

Thanks for jumping in

Cheers

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d1fc3fb6-d902-437d-96eb-7a5f0a8a163an%40googlegroups.com.