Question about threading a view[REPOSTED]

[Arruda]

(I've created a topic like this a few minutes ago, but was using the old google groups, and now it's broken. So I created a new one using the new google groups).

Hi, I'll try to explain the best I can my problem, and I don't know if
what I'm trying to archive is the best way to get where I want, but
here is it:
I want that a specific view to run in a new thread, ex:
 def foo(request):
   do_something_slow()
  return httpResponse

But I DON'T want that a new thread is run inside the view, the view it
self should be runned in another thread, ex:

 def foo(request):
   t = thread(target=do_something_slow()....)
   t.daemon = True
   t.start()

  return httpResponse

This way when a user A access any page the site will load, even if a
user B is accessing the 'foo' view.
But the B user when access the view 'foo' will load it just a if if
was a normal view( will be slow and will render the response just
after do_something_slow() finished running).

I don't know if this is what I want, but I believe that there is
another way around:
when user B calls foo view, it will render to him a page with a JS(I
don't know JS either, so correct me if I'm talking nonsense) that say
its "Loading..."
And after the do_someting_slow() finished running if will change the
content of the page to whatever the result of "do_something_slow" was.

Thanks for the attention.

[Kurtis Mullins]

Did you check out django-celery yet?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/jc20eqhcl8oJ.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Arruda]

Some one also gave me that tip. 

I was reading it, but I couldn't see much clear how to use this in this case.

I saw that to execute a task I should run it by using ./manage.py ... and how can I do that in a view?

Another thing is that it seems to me a bit overkill isn't it, and a little to complicated to setup.

To test it in development I need to prepare a huge scenario so that it can work out, and also I can't imagine how to use this in tests. 

Did I understood it wrong? Ins't it a lot complicated to use?

Thanks

Em segunda-feira, 19 de março de 2012 16h43min03s UTC-3, Kurtis escreveu:

Did you check out django-celery yet?

To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.

[Jani Tiainen]

Hi,

Basic idea is to send a message to message broker from your view to tell "I want to run this long running task X".

After that your view can return (with task id). After that your client can start asking from second view for long running response.

In the background you have to have some means to query message broker for requested tasks and execute them. For example 

using a cron job.

That's where django-celery comes in. It can create tasks and it can run tasks within Django framework using various message

brokers - rabbitmq being probably most popular.

So first you define task with django-celery. Then in your view you send message to message broker that "execute that task".

And then you must have some background running process to read messages from broker and execute them.

What comes to testing usually you mock responses in these kind of tests. Of course you can still test your long running process but

in unit tests you don't test that communication between broker and running system.

HTH,

Jani Tiainen

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/9Xk3RQ-y6pgJ.

To post to this group, send email to django...@googlegroups.com.

To unsubscribe from this group, send email to django-users...@googlegroups.com.

[Marc Patermann]

Arruda schrieb (19.03.2012 19:18 Uhr):

> I don't know if this is what I want, but I believe that there is
> another way around:
> when user B calls foo view, it will render to him a page with a JS(I
> don't know JS either, so correct me if I'm talking nonsense) that say
> its "Loading..."
> And after the do_someting_slow() finished running if will change the
> content of the page to whatever the result of "do_something_slow" was.

I think this is what the "Asynchronous" in AJAX stands for.
http://en.wikipedia.org/wiki/Ajax_%28programming%29
https://code.djangoproject.com/wiki/AJAX

Marc

[Jani Tiainen]

That is not related to initial problem (long running tasks on server).

Even you can use AJAX calls to perfrom something asyncronously, it still
requires open connection to server. If you have tasks that runs long,
let's say 20 minutes your AJAX call will definitely be terminated (timed
out). So that is not a right solution.

Using AJAX to query long running status makes a lot of sense though.

--

Jani Tiainen

[Arruda]

Thanks every one, I've manage to work this around.

The solution might not be the best, but it's working:

In the view I call the do_something_slow() in a new thread, and in the end of it(after the role process is done) it change the model of a class to add the feedback.

While the thread is running it renders the template to the user(with a "Loading..." message).

Then I've made a ajax_get_feed_back() view to get this feedback.

In the template, I made a simple JS that every second tries to get the feedback via ajax.

And if it gets the feedback it stops requesting it.

=)

But I still wanted to know how to use the celery in this case =/

But got very lost in that, does any one have a example(code) of something similar to do with it?

In the docs I only found examples of process that are kind of tasks... to run every hour, or things like that.

[Javier Guerra Giraldez]

On Tue, Mar 20, 2012 at 9:52 AM, Arruda <felipe.arr...@gmail.com> wrote:
> But I still wanted to know how to use the celery in this case =/

the idea is to use a Queue:

you have a separate process (typically implemented in a manage
command) that stays running, and waits for messages in the queue.

when the web app wants to do some slow processing, writes any needed
parameters to the queue and returns with a 'wait...' message

the background process receives the queue item and does the process
without tying up the web response.

the first implementation almost everybody does is called "Ghetto
queues", it consist of writing the parameters to the database in a
'todo' table, and a cron process that periodically checks these todo's
and processes them. it works, but falls down at a certain load.
after that, you need a real queue manager; either RabbitMQ, Redis
Queues, or a lot of other options.

Celery is a python package that allows you to simply add a '@task'
decorator to any function; so that when you call them, they're not
executed, but instead the argument list is pushed to a queue. it also
creates an admin command that does the queue reading and actually
calls the functions with the arguments it founds on there.

--
Javier

[Tom Evans]

On Tue, Mar 20, 2012 at 2:52 PM, Arruda <felipe.arr...@gmail.com> wrote:
> Thanks every one, I've manage to work this around.
> The solution might not be the best, but it's working:
> In the view I call the do_something_slow() in a new thread, and in the end
> of it(after the role process is done) it change the model of a class to add
> the feedback.
> While the thread is running it renders the template to the user(with a
> "Loading..." message).
> Then I've made a ajax_get_feed_back() view to get this feedback.
>
> In the template, I made a simple JS that every second tries to get the
> feedback via ajax.
> And if it gets the feedback it stops requesting it.
> =)
>
> But I still wanted to know how to use the celery in this case =/
> But got very lost in that, does any one have a example(code) of something
> similar to do with it?
> In the docs I only found examples of process that are kind of tasks... to
> run every hour, or things like that.
>

I don't have any easily shareable code, but the basic principle is like this:

Django request comes in
View wants to handle some code in the background
View creates task object, with an id
View renders page, includes task id
Rendered page polls the web server with the id, waiting for the task to finish
Webserver returns data associated with id when task is complete
Javascript inserts data into the page

Meanwhile..

When the task is created, a message is sent to the message broker.
There are several backend task processors attached to the message
broker, and the message is delivered to one of them.
The task processor runs the delayed task, and once complete returns the data.

Cheers

Tom

[Jani Tiainen]

Hi,

Your solution will work perfectly in single process environment (For
example with manage.py runserver )

But when you put your app behind webserver you usually invoke several
processes. For example if you're using single round-robin loadbalancing
over 5 different Django instances.

What would happen if Django instances are restarted after n cycles (like
our wsgi django instances are restarted after 1000 requests). Can you
guarantee that your thread survives?

If you do "massive" scaling: what happens if you have multiple servers,
how can you guarantee that consecutive requests from single user goes to
same server and same process?

That is why there exists things like django-celery and rabbitmq (message
broker). How they work is already explained to you.

20.3.2012 16:52, Arruda kirjoitti:
> Thanks every one, I've manage to work this around.
> The solution might not be the best, but it's working:

> In the view I call the *do_something_slow()* in a new thread, and in the

> end of it(after the role process is done) it change the model of a class
> to add the feedback.
> While the thread is running it renders the template to the user(with a

> *"Loading..."* message).
> Then I've made a *ajax_get_feed_back() *view to get this feedback.

>
> In the template, I made a simple JS that every second tries to get the
> feedback via ajax.
> And if it gets the feedback it stops requesting it.
> =)
>
> But I still wanted to know how to use the celery in this case =/
> But got very lost in that, does any one have a example(code) of
> something similar to do with it?
> In the docs I only found examples of process that are kind of tasks...
> to run every hour, or things like that.
>

> Em segunda-feira, 19 de mar�o de 2012 15h18min09s UTC-3, Arruda escreveu:
>
> *(I've created a topic like this a few minutes ago, but was using

> the old google groups, and now it's broken. So I created a new one

> using the new google groups).*

> Hi, I'll try to explain the best I can my problem, and I don't know if
> what I'm trying to archive is the best way to get where I want, but
> here is it:
> I want that a specific view to run in a new thread, ex:

> *def foo(request):
> do_something_slow()
> return httpResponse*

>
> But I DON'T want that a new thread is run inside the view, the view it
> self should be runned in another thread, ex:

> *

> def foo(request):
> t = thread(target=do_something_slow()....)
> t.daemon = True
> t.start()
>
> return httpResponse

> *

> This way when a user A access any page the site will load, even if a
> user B is accessing the 'foo' view.
> But the B user when access the view 'foo' will load it just a if if
> was a normal view( will be slow and will render the response just
> after do_something_slow() finished running).
>
> I don't know if this is what I want, but I believe that there is
> another way around:
> when user B calls foo view, it will render to him a page with a JS(I
> don't know JS either, so correct me if I'm talking nonsense) that say
> its "Loading..."
> And after the do_someting_slow() finished running if will change the
> content of the page to whatever the result of "do_something_slow" was.
>
> Thanks for the attention.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/django-users/-/ST6KqE0uPqsJ.

[Arruda]

Thanks every one, and yes, I'm heaving problem with the project in production... 

I'm using fstcgi and nginx, and when it's called that view it just don't run in background, or run once and then it's stops...

So I'm thinking of using django-celery... but just have one more question...

Javier, you said that with celery you can use the @task thing ins't it?

But to make the celery work in the production I need to make some crazy confs in a virtualhost to it work?

Or just use the @task + manage.py? (I still have hope that it's stupid easy as that to use it xD).

Thanks!

Em terça-feira, 20 de março de 2012 12h01min47s UTC-3, Javier Guerra escreveu:

[Arruda]

Thanks everyone, I'd like to say that I ended up using django-celery + RabbitMQ server.

But had some problems trying to make celery run as daemon.

After sometime I discovered what need to be done(didn't like the docs in this part... found them misleading), everything is working great.

Even made a post about this:

http://www.arruda.blog.br/programacao/django-celery-in-daemon/

Thanks

> django-users+unsubscribe@googlegroups.com.

[Shawn Milochik]

The best solution for this, in my opinion and experience, is to use
supervisord. It's easy, requires no root access, and allows you to
control all your long-running processes (Django app, celery, Redis,
MongoDB, etc.) from one place.