Run Django tasks with another programming language

[M Mihai]

Hi,

Right now I use https://github.com/Koed00/django-q and as broker, I use Redis to run some tasks on my Django website, but they're using 100% of the processor when I'm running 30 workers. 

So I'm looking for a way to run the tasks with another programing language(I would prefer golang or c#) and just update the database from python after the tasks were processed until now I've found only this package for golang: https://github.com/gocelery/gocelery but I was unable to make this work.

[Jani Tiainen]

Hi.

I greatly suspect that your problem isn't the Python itself but what you do in your workers.

Also what analysis made you to conclusion that using golang or c# would improve situation? In other words have you really measured and identified the slow part?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/c256b5fa-b3d9-4386-bfca-39ebf745c793%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[M Mihai]

I've done the same requests on golang using gorutins and they didn't use so much of my processor, I didn't test c# on the same vps, but I think c# will perform better in this case also.

Python is a wonderful programing language, but after I've tested the same requests I think I will be better of if I will do these requests using another programming language

On Friday, July 28, 2017 at 8:34:53 PM UTC+3, Jani Tiainen wrote:

Hi.

I greatly suspect that your problem isn't the Python itself but what you do in your workers.

Also what analysis made you to conclusion that using golang or c# would improve situation? In other words have you really measured and identified the slow part?

28.7.2017 19.21 "M Mihai" <1497...@gmail.com> kirjoitti:

Hi,

Right now I use https://github.com/Koed00/django-q and as broker, I use Redis to run some tasks on my Django website, but they're using 100% of the processor when I'm running 30 workers. 

So I'm looking for a way to run the tasks with another programing language(I would prefer golang or c#) and just update the database from python after the tasks were processed until now I've found only this package for golang: https://github.com/gocelery/gocelery but I was unable to make this work.

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

[Jani Tiainen]

Hi.

You should run performance profile analysis for python code to figure out what is eating cpu so much. It's easy to do and doesn't take long. That would also give you good understanding about the part that really consumes so much resources and can it be actually fixed or is it something really considered to be implemented by some other means.

So your main problem is that you don't know the real reason why your workers are slow. Figure that out and your problem is much easier to resolve either.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/90e33207-5def-4b82-8eef-6b38935648ca%40googlegroups.com.

[Jani Tiainen]

Furthermore, your findings would be helpful for others as well.

You’ve found something that is slow (or just hogs resources) would allow others to resolve similar issue(s) much more easy.

[M Mihai]

Well, I didn't do any profile analysis, but I'm using workers for other types of tasks and they do work great, the problem is when I try to send HTTP requests to few website per worker.

Also if I use the workers without sending the HTTP requests, using some fake data instead they work fine, so sending HTTP requests in the problem.

PS: I'm not trying to send them using only one worker, I'm using 30 of them.