python - handling HTTP requests asynchronously
41 views
Skip to first unread message
luke lukes
May 6, 2016, 8:37:54 AM5/6/16
to Django users
Hi everyone,
I need to generate a PDF report for each entry of a django queryset. There'll be between between 30k and 40k entries.
The PDF is generated through an external API. Since currently is generated on demand, this is handled synchronously via an HTTP request/response. That will be different for this task, since I think I'll use a django management command to loop through the queryset and perform the PDF generation.
Which approach should I follow for this task? I thought about 3 possibile solutions, although are technologies that I never used:
1) Celery: assign a task (http request with a different payload) to a worker, then retrieve it once it's done;
2) request-futures: using requests in a non-blocking way;
3) multiprocessing package, e.g. assigning a Pool of workers.
the goal is to use the API concurrently (e.g. send 10 or 100 http requests simultaneously, depending on how many concurrent requests the API can handle).
Anybody here that handled a similar task and can give advices on how to proceed on this?
I need to generate a PDF report for each entry of a django queryset. There'll be between between 30k and 40k entries.
The PDF is generated through an external API. Since currently is generated on demand, this is handled synchronously via an HTTP request/response. That will be different for this task, since I think I'll use a django management command to loop through the queryset and perform the PDF generation.
Which approach should I follow for this task? I thought about 3 possibile solutions, although are technologies that I never used:
1) Celery: assign a task (http request with a different payload) to a worker, then retrieve it once it's done;
2) request-futures: using requests in a non-blocking way;
3) multiprocessing package, e.g. assigning a Pool of workers.
the goal is to use the API concurrently (e.g. send 10 or 100 http requests simultaneously, depending on how many concurrent requests the API can handle).
the project runs on Python 2.7 and this task will occur approximately once a year.
Anybody here that handled a similar task and can give advices on how to proceed on this?
Avraham Serour
May 8, 2016, 7:18:32 AM5/8/16
to django-users
so it seems that what you need is not to handle http requests asynchronously, but to process async jobs requested by http.
if that is the case celery is a popular option and relatively easy to setup, I personally would use it for such things.
buton the other hand you are saying that the task in hand will be done once a year, so it has nothing to do with http request, the job is activated by time.
I wouldn't setup all this overhead for a once a year task, you don't even need async infrastructure, just set the command on cron and run
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a10436c0-2493-4889-b829-b4fc725f7e8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages