Periodic Execution of django functions or tasks

[MUGOYA DIHFAHSIH]

Hello fellow django developers, i am working on a web app in which reports are archived in the database at the end of every month automatically, i have searched on internet and many folks are suggesting celery which i have tried but not fruitful, is there any other library i use to accomplish this.

am using Python 3.8, django 2.2.8 and ubuntu

[David Nugent]

Not exactly automatic, but can be set up to do so, something along the lines:

https://wiki.postgresql.org/wiki/Month_based_partitioning

This solution is Postgresql specific, although I've done effectively the same with mysql with some help in the code.

Partitioning the db in this way makes it (almost) completely transparent to the application except when it comes to reporting. In mysql you can use the MERGE storage engine to aggregate a reporting period, which requires special rollover handling each month to manage. Otherwise crafted views is an alternate solution, again a requiring month rollover to regenerate.

Management of indexes is another issue - you lose index unique constraints across different months except with the partitioning date field that determines the table where records are inserted.

HTH, David

On 20 Oct 2020, at 00:53, MUGOYA DIHFAHSIH <dihfa...@gmail.com> wrote:

Hello fellow django developers, i am working on a web app in which reports are archived in the database at the end of every month automatically, i have searched on internet and many folks are suggesting celery which i have tried but not fruitful, is there any other library i use to accomplish this.

am using Python 3.8, django 2.2.8 and ubuntu

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAP%3DJD9wJGPFZaCJOefqzt48nqb%3DNbZNXoTtdLv5EzX%2BoUM880Q%40mail.gmail.com.

[Ryan Nowakowski]

On Mon, Oct 19, 2020 at 04:53:35PM +0300, MUGOYA DIHFAHSIH wrote:
> Hello fellow django developers, i am working on a web app in which reports
> are archived in the database at the end of every month automatically, i
> have searched on internet and many folks are suggesting celery which i have
> tried but not fruitful, is there any other library i use to accomplish this.
>

Yeah, celery is great! But sometimes it's too complex a beast for
simple things. Usually I write a custom Django management command[1].
In your case it would probably be named something like archivereports.
Then add a cron entry to run it at the end of every month.

[1] https://docs.djangoproject.com/en/2.2/howto/custom-management-commands/
[2] https://help.ubuntu.com/community/CronHowto

[Derek]

Ryan's suggestion is a very good one. 

If you don't need Django-specific commands, you can even write a stand-alone Python script which accesses your database to do the work. An alternative to cron that we use, on occasion, is https://pypi.org/project/APScheduler/ which runs in a container (a Python scheduler like this is also useful if you want to run on a non-Unix operating system). 

[Dan Davis]

If celery is too much, consider django-background-tasks.

To be totally honest, I am an old dog, and I generally do the following:

* Write a management command.

* Deploy a crontab to run my management command.

This is not so container or cloud friendly, but none of my cloud deployments run "long" tasks.  So, I've been able to translate this old pattern like this:

* Have a Lambda hit a web URL which runs the management command through django.core.management.call_command (for brief commands)

* Deploy to Lambda using Zappa and have a scheduled event in CloudWatch events run the command.

I think doing this same pattern with AWS Batch would work for longer commands - and only AWS is relevant to me.

Anyway, django-background-tasks or a simple management command are both less to deploy than celery.