Re: Python script not letting go of files

[Weatherby,Gerard]

Does the script exit when complete?

If you’re running on a Linux based system and have root access you can use lsof to see what processes have with files open. (Or use the psutil Python package).

From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of Mike Dewhirst <mi...@dewhirst.com.au>
Date: Tuesday, November 29, 2022 at 2:20 AM
To: pytho...@python.org <pytho...@python.org>
Subject: Python script not letting go of files
I have a script which fetches a production site directly from a
Subversion repo using svn export

It runs a bunch of commands by calling this little method ...

def trycmd(cmd, log):
retcode = -1
ret = f"Trying {cmd}"
try:
retcode = os.system(cmd)
ret = f"\n{cmd} -ok-> {retcode}"
except Exception as err:
ret = f"\n{cmd} -fail-> {err}"
log.write(remove_password(ret))
return retcode

This is the fetching script (omitting variables at the top) which
appears to be keeping a finger on files which Apache wants.

with open(fetchlog, 'a') as log:
ret = f"\n\nFetching {tag}"
log.write(ret)
cmd = f"sudo rm -Rf {site_root}"
if trycmd(cmd, log) == 0:
cmd = f"sudo svn export --force --username {usr} --password
{pw} {svn_repo} {site_root}"
if trycmd(cmd, log) == 0:
# get any new dependencies
cmd = f"sudo -H pip install -r {reqfile}"
if trycmd(cmd, log) == 0:
# run any migrations shipped from the repo
cmd = f"sudo python3 {site_root}/manage.py migrate
--noinput --settings={settings}"
if trycmd(cmd, log) == 0:
# shouldn't find anything
cmd = f"sudo python3 {site_root}/manage.py
makemigrations --noinput --settings={settings}"
if trycmd(cmd, log) == 0:
# should have been done already
cmd = f"sudo python3 {site_root}/manage.py
migrate --noinput --settings={settings}"
if trycmd(cmd, log) == 0:
# remove all static files from their Apache dir
cmd = f"sudo rm -Rf {static_root}/*"
if trycmd(cmd, log) == 0:
# copy all static files to the Apache
location
cmd = f"sudo python3
{site_root}/manage.py collectstatic --noinput --settings={settings}"
if trycmd(cmd, log) == 0:
# set all permissions
cmd = f"sudo {scripts}/perms_{host}.sh"
if trycmd(cmd, log) == 0:
cmd = "sudo service apache2
restart"
if trycmd(cmd, log) == 0:
ret = f"\nFinish {tag}\n\n"
log.write(ret)
else:
print("Apache didn't restart")
else:
print("Didn't set permissions")
else:
print("Didn't collectstaic")
else:
print("Didn't delete static files")
else:
print("Didn't migrate 2")
else:
print("Didn't makemigration")
else:
print("Didn't migrate 1")
else:
print("Didn't install requirements")
else:
print("Didn't get source")
else:
print("Didn't remove site")
exit()

The problem I'm trying to fix is that after an Apache reload Apache
seems hellbent on filling up its scoreboard and running out of
resources. The kind folk on the Apache mailing list say something is
hogging the workers.

You can see the last operation above is an Apache restart. It should be
an Apache reload. Reload however lets Apache run out of resources.

Do any of you Python folks see any blunders in the above code along the
lines of not letting go of py files or static assets?

mod_wsgi is configured with 100% defaults.
mod_mpm_event conf per advice from the Apache mailing list is ...

<IfModule mpm_event_module>
ServerLimit 32
StartServers 16
MinSpareThreads 400
MaxSpareThreads 800
ThreadLimit 64
ThreadsPerChild 50
AsyncRequestWorkerFactor 2
MaxRequestWorkers 800
MaxConnectionsPerChild 0
</IfModule>

Server Version: Apache/2.4.52 (Ubuntu 2022.04) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10
Server MPM: event
Server Built: 2022-09-30T04:09:50

DigitalOcean droplet 8BG RAM and lightly loaded.

Many thanks for any hints.

Cheers

Mike


--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

[Weatherby,Gerard]

"Weatherby,Gerard" <gweat...@uchc.edu> writes:
>Do any of you Python folks see any blunders in the above code along the
>lines of not letting go of py files or static assets?

Er, no, I just replied to the original poster.