Celery/Redis Django 2.0 Design Question

[Mark Phillips]

I am building a django 2.x site to:

* upload documents (images, pdfs, and videos)

* apply metadata to the documents (JSON metadata field)

* transform the documents (thumbnails, OCR, language translations, image conversion, facial recognition, image blurring, etc.) based on some of the metadata fields

* display the documents

Since the document transformations are fairly resource intensive and time consuming, I am thinking of using celery to build a state machine for the transformations, using celery tasks to transform the images, and celery signals to update the state of the document as the transformations complete successfully. I looked at django-fsm for this, but I think it will be better to run the transformations as celery tasks than to block the site. 

Does this plan make sense, or am I missing something regarding django and celery/redis.

Thanks!

Mark

[hunter...@gmail.com]

I think you are on the right track. I am doing something similar with photos and gps tracks uploaded by users and the celery/redis combo makes it easy to run a compute intensive task without blocking the site.

I'm not sure if the celery-signals approach is strictly necessary, however.  I am following the architecture laid out in cookiecutter-django (using docker) where the celery container is a copy of the django container.

That way, when the transformations are complete, the celery task can just issue a django style create or update command to access the database.

Good luck!

dave