Running a custom code after the server is up

[Bartosz Gańcza]

Hi everyone!

I am somewhat of a Django beginner and I can't seem to find an easy solution to what I need to do anywhere.

I have a web scraping code I wish to run in the background automatically (once) after the server is up and running. I use Docker to fire up the DB and the web server itself but can't seem to be able to configure it to also fire up the management command I configured that runs the scraping code (I guess using Docker for that is either beyond my current knowledge or is just not possible without using specialised tools like cron, which I don't fully understand).

Is there any way to do something like this in Django itself, without resorting to "ready" function? (I did that at first but it stops the server from running until the code completes and also runs it at least twice)

Best,

Bartosz

[PASCUAL Eric]

Hi,

IMHO the simplest way is to use cron.

Since you are working with Docker, one option is to add a container to your stack, sharing the same image as your Django app so that it can run the management command implementing your periodic process. Configure this container so that the command it runs is the cron daemon, and of course schedule your periodic task in the crontab.

Some people use Gunicorn start hook to fork a sub-process running the cron daemon. Others embed supervisor in a container and manage the web app process and the other ones with it. Although it works,  such approaches go against the rule which says that a Docker container should handle only one thing (either the app server or the cron stuff in you case) so that it can be managed independantly (scale, update image, restart...).

This is important when you deploy your stack under the control of an orchestrator such as Kubernetes, which will manage the containers (err. the "pods") automatically so that the state or the stack is the targeted one WRT replicas count, resource limits...

Hope this helps.

Eric

---

From: django...@googlegroups.com <django...@googlegroups.com> on behalf of Bartosz Gańcza <gar...@gmail.com>
Sent: Friday, November 2, 2018 3:31:38 PM
To: django...@googlegroups.com
Subject: Running a custom code after the server is up

 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAOjbnAWtCMWRck45hgZQqeaY-U86iEmiGs%2Btpq617V81_%2B6hkQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[bill.torc...@gmail.com]

If I understand your situation, you want to run a management command fairly promptly after Django-server reboot, and not again for the duration of the server's uptime.

Cron is useful and convenient for repeated tasks.  And it can be made to handle once-and-done tasks, if you keep a bit of state somewhere.

There's a lot I don't know about the practical side of your situation, so I ask this question from ignorance. Can you use the __init__method of a special purpose app in INSTALLED_APPS?  I don't know if the INSTALLED_APPS are created during server startup, or only as needed in response to an incoming URL.

If each class (or package?) in INSTALLED_APPS gets an object created at server-boot time, you can do it within one of your apps.

__init__ can start a separate thread that delays for a specific time interval and then does your web-scraping scraping.  __init__() itself cannot do the wait, because there is an implicit expectation that __init__() finishes in a relatively short period of time.

If the trigger event for starting the web-scraping is something like first-visit-to-your-home-page, you could probably use django signals.  But I've never used signals and I can't tell you anything more about that.

  ---  Bill