RFC: Asynchronous Signal Receivers
142 views
Skip to first unread message
zvoase
Jan 4, 2009, 2:46:08 AM1/4/09
to Django developers
Hi Developers,
Basically, I wanted to ask for some devel's opinions on something.
When using the Django signals framework (which, by the way, is a very
good way of decoupling reusable apps whilst allowing for deep
integration), I sometimes find that a signal receiver does not need to
be called in synchronicity with the rest of the receivers. To
demonstrate, I'm going to explain a common scenario when it might be
useful to have *asynchronous* signal receivers, and why I feel there
should be a special provision made for them in the signals framework.
Let's suppose I have a comments app, and I write a signal which is
'fired' when a new comment is posted. Now let's assume I also write a
mail notification app which listens for this signal. When a comment is
posted on an object, the owner of the object is notified via e-mail.
For argument's sake, let's also assume that this app is *not* using a
mail queue which is flushed several times a day via a cron job - the
owner should be notified as soon as possible after the publication of
the comment. This means that the mail notifier's receiver function
must send the e-mail when called. This raises the issue that e-mail
may take a while (i.e. from a few seconds to a couple of minutes) to
send; it is after all operating over a network connection. This is
unacceptable, as we would ideally like to minimise latency in the HTTP
response. At the moment, the Signal class processes receivers in
serial; one must finish execution before the next may begin. It is
not, however, *critical* that this e-mail is sent before the next
receiver is processed; it may be put into the background in another
thread, and finish within a reasonable amount of time.
An obvious solution to this problem is to use threading or forking,
thus placing the sending of the e-mail into the background. From my
receiver function I can do this by creating a new thread and starting
it, then returning after starting the thread without ensuring that the
sending of the e-mail has been completed. However, we run into a snag
whereby a thread is being created, started and left alone, without any
conceivable way of 'reigning it back in'. This could result in major
memory leaks or undetected infinite loops, and there is no obvious way
of creating a thread from within the receiver function *without* this
problem.
So, taking the idea of threading but reworking it slightly, there is
another way to deal with the problem. If the Signal object were aware
that a particular receiver function (or a collection thereof) were to
be run in the background, then it could essentially start all of these
tasks in tandem and also create a controller, which could be passed
back to the sender of the signal. This controller will contain all the
thread instances, and might have a 'join' method which would wait
until they had all finished executing; an infinite loop or memory leak
would therefore be controllable, or at least detectable, as the
threads would be referenced within the scope of the sender.
Applying this solution to the problem of e-mail sending, I would write
a function as before that sent an e-mail in the usual way. I would
then use the typical 'signal.connect' method to connect my receiver to
the signal, but pass in an 'async=True' keyword argument. Then, when a
comment is posted, a new e-mail-sending thread is created, along with
a thread controller (itself probably just a subclass of list with a
'join' method, or something along those lines). This would send the e-
mail and finish execution, without holding up my view function where
I've added the comment.
I hope I've made my rationale for this feature clear. I've sent it to
the mailing list first because I think it's something which could be
debated (at a good length). The inclusion of asynchronous signal
receivers will help Django's signals framework in becoming a mature
message-queueing solution, and I think that the use of signals is
invaluable to properly decoupling functionality whilst maintaining a
state of integration between a diverse sets of Django applications.
Regards,
Zack
Basically, I wanted to ask for some devel's opinions on something.
When using the Django signals framework (which, by the way, is a very
good way of decoupling reusable apps whilst allowing for deep
integration), I sometimes find that a signal receiver does not need to
be called in synchronicity with the rest of the receivers. To
demonstrate, I'm going to explain a common scenario when it might be
useful to have *asynchronous* signal receivers, and why I feel there
should be a special provision made for them in the signals framework.
Let's suppose I have a comments app, and I write a signal which is
'fired' when a new comment is posted. Now let's assume I also write a
mail notification app which listens for this signal. When a comment is
posted on an object, the owner of the object is notified via e-mail.
For argument's sake, let's also assume that this app is *not* using a
mail queue which is flushed several times a day via a cron job - the
owner should be notified as soon as possible after the publication of
the comment. This means that the mail notifier's receiver function
must send the e-mail when called. This raises the issue that e-mail
may take a while (i.e. from a few seconds to a couple of minutes) to
send; it is after all operating over a network connection. This is
unacceptable, as we would ideally like to minimise latency in the HTTP
response. At the moment, the Signal class processes receivers in
serial; one must finish execution before the next may begin. It is
not, however, *critical* that this e-mail is sent before the next
receiver is processed; it may be put into the background in another
thread, and finish within a reasonable amount of time.
An obvious solution to this problem is to use threading or forking,
thus placing the sending of the e-mail into the background. From my
receiver function I can do this by creating a new thread and starting
it, then returning after starting the thread without ensuring that the
sending of the e-mail has been completed. However, we run into a snag
whereby a thread is being created, started and left alone, without any
conceivable way of 'reigning it back in'. This could result in major
memory leaks or undetected infinite loops, and there is no obvious way
of creating a thread from within the receiver function *without* this
problem.
So, taking the idea of threading but reworking it slightly, there is
another way to deal with the problem. If the Signal object were aware
that a particular receiver function (or a collection thereof) were to
be run in the background, then it could essentially start all of these
tasks in tandem and also create a controller, which could be passed
back to the sender of the signal. This controller will contain all the
thread instances, and might have a 'join' method which would wait
until they had all finished executing; an infinite loop or memory leak
would therefore be controllable, or at least detectable, as the
threads would be referenced within the scope of the sender.
Applying this solution to the problem of e-mail sending, I would write
a function as before that sent an e-mail in the usual way. I would
then use the typical 'signal.connect' method to connect my receiver to
the signal, but pass in an 'async=True' keyword argument. Then, when a
comment is posted, a new e-mail-sending thread is created, along with
a thread controller (itself probably just a subclass of list with a
'join' method, or something along those lines). This would send the e-
mail and finish execution, without holding up my view function where
I've added the comment.
I hope I've made my rationale for this feature clear. I've sent it to
the mailing list first because I think it's something which could be
debated (at a good length). The inclusion of asynchronous signal
receivers will help Django's signals framework in becoming a mature
message-queueing solution, and I think that the use of signals is
invaluable to properly decoupling functionality whilst maintaining a
state of integration between a diverse sets of Django applications.
Regards,
Zack
alex....@gmail.com
Jan 4, 2009, 3:01:37 AM1/4/09
to Django developers
I don't follow the argument against simply having the signal spawn a
thread. Why does the main thread ever need to regain control of it,
if the signal sender needs the response than ultimately the signal
can't occur asynchronously anyways, since sooner or later you'll need
to simply wait for the thread to complete it's execution. Further if
this is really an issue one can always have their signal return a Pipe
between the spawned thread and the main thread and within their own
application poll the pipe for new data. All of this is pretty easy to
wrap up in a 4 or 5 line decorator.
Alex
thread. Why does the main thread ever need to regain control of it,
if the signal sender needs the response than ultimately the signal
can't occur asynchronously anyways, since sooner or later you'll need
to simply wait for the thread to complete it's execution. Further if
this is really an issue one can always have their signal return a Pipe
between the spawned thread and the main thread and within their own
application poll the pipe for new data. All of this is pretty easy to
wrap up in a 4 or 5 line decorator.
Alex
flo...@gmail.com
Jan 4, 2009, 3:36:44 AM1/4/09
to Django developers
If you ask me, a message queue sits quite firmly outside the realm of
what a web framework should be responsible for providing. I think you
are also underestimating the amount of effort and thought that goes
into writing a robust and scalable message queue. For one simple
example: what if Apache decides to kill or restart the Django process
after a certain amount of requests have been processed (and I know
that many people have their servers set to do this)? The threads
which were are running asynchronous tasks would be killed as well,
losing any tasks which hadn't yet been completed.
However, you do bring up a good point: being able to do things
asynchronously is one of many key factors in being able to write truly
scalable code. It's for that reason, I think, that Brian Rosner and
James Tauber have been working on, not a re-implementation of a
message queue, but instead a pluggable connector to communicate with
more robust message queues in their django-engine project [1]. I
haven't kept close tabs on that project, but I think it's far from
ready for prime-time.
If something like django-engine were to be more fully fleshed-out and
made more robust, then I think it would make a much better case for
inclusion in contrib, rather than try to turn django's signals into
something they weren't meant to be. And I'm pretty sure Brian and
James accept patches and idea contributions :)
Anyway, that's just my 2c.
-Eric Florenzano
[1] http://code.google.com/p/django-engine/
what a web framework should be responsible for providing. I think you
are also underestimating the amount of effort and thought that goes
into writing a robust and scalable message queue. For one simple
example: what if Apache decides to kill or restart the Django process
after a certain amount of requests have been processed (and I know
that many people have their servers set to do this)? The threads
which were are running asynchronous tasks would be killed as well,
losing any tasks which hadn't yet been completed.
However, you do bring up a good point: being able to do things
asynchronously is one of many key factors in being able to write truly
scalable code. It's for that reason, I think, that Brian Rosner and
James Tauber have been working on, not a re-implementation of a
message queue, but instead a pluggable connector to communicate with
more robust message queues in their django-engine project [1]. I
haven't kept close tabs on that project, but I think it's far from
ready for prime-time.
If something like django-engine were to be more fully fleshed-out and
made more robust, then I think it would make a much better case for
inclusion in contrib, rather than try to turn django's signals into
something they weren't meant to be. And I'm pretty sure Brian and
James accept patches and idea contributions :)
Anyway, that's just my 2c.
-Eric Florenzano
[1] http://code.google.com/p/django-engine/
zvoase
Jan 4, 2009, 4:12:05 AM1/4/09
to Django developers
First, I'll address Alex's concerns. Basically, an async decorator
would work fine for some things, but it helps to be able to co-
ordinate the threads. Look at it this way; when something uses TCP,
for example, it's going to take a certain finite amount of time for
the TCP request to occur. Without asynchronicity, that time is going
to be spent by the interpreter doing nothing, meaning other
computations which might take some time and *could* be going on now,
aren't. So even though you're waiting for them all to complete at the
end, you're still saving time. Even if there was a simple decorator
which could be used, maybe it's a good idea to include that decorator
in Django; I could see it being useful to some people.
Now on to Eric's comment:
> If you ask me, a message queue sits quite firmly outside the realm of
> what a web framework should be responsible for providing. I think you
> are also underestimating the amount of effort and thought that goes
> into writing a robust and scalable message queue.
There is already at least one other web framework I know of (http://
appcelerator.org/) which provides message queuing functionality as an
important part of the framework itself; I think that we're going to
start seeing a lot more of this sort of thing in web frameworks over
the next few years. Message passing is, as you say, a consideration
when trying to scale. I'm also not saying that Django should try to re-
implement something like Apache ActiveMQ, etc., but that a system
which tried to encapsulate some of these concepts wouldn't do the
framework any harm.
> For one simple example: what if Apache decides to kill or restart the Django process
> after a certain amount of requests have been processed (and I know
> that many people have their servers set to do this)? The threads
> which were are running asynchronous tasks would be killed as well,
> losing any tasks which hadn't yet been completed.
To be honest, I hadn't thought about that; that indeed would make it
difficult to implement async signals. However, the aggregated 'join'
method could be called after the receivers have all finished, as
Apache will not terminate the Django process until all requests have
been handled; so there are probably ways around this.
Also, I didn't know about engine, thanks for telling me about it. The
concept looks quite good, and I'm very intrigued to find out more
now...
Thanks for your feedback so far. This is why I sent it to the mailing
list in the first place :)
would work fine for some things, but it helps to be able to co-
ordinate the threads. Look at it this way; when something uses TCP,
for example, it's going to take a certain finite amount of time for
the TCP request to occur. Without asynchronicity, that time is going
to be spent by the interpreter doing nothing, meaning other
computations which might take some time and *could* be going on now,
aren't. So even though you're waiting for them all to complete at the
end, you're still saving time. Even if there was a simple decorator
which could be used, maybe it's a good idea to include that decorator
in Django; I could see it being useful to some people.
Now on to Eric's comment:
> If you ask me, a message queue sits quite firmly outside the realm of
> what a web framework should be responsible for providing. I think you
> are also underestimating the amount of effort and thought that goes
> into writing a robust and scalable message queue.
appcelerator.org/) which provides message queuing functionality as an
important part of the framework itself; I think that we're going to
start seeing a lot more of this sort of thing in web frameworks over
the next few years. Message passing is, as you say, a consideration
when trying to scale. I'm also not saying that Django should try to re-
implement something like Apache ActiveMQ, etc., but that a system
which tried to encapsulate some of these concepts wouldn't do the
framework any harm.
> For one simple example: what if Apache decides to kill or restart the Django process
> after a certain amount of requests have been processed (and I know
> that many people have their servers set to do this)? The threads
> which were are running asynchronous tasks would be killed as well,
> losing any tasks which hadn't yet been completed.
difficult to implement async signals. However, the aggregated 'join'
method could be called after the receivers have all finished, as
Apache will not terminate the Django process until all requests have
been handled; so there are probably ways around this.
Also, I didn't know about engine, thanks for telling me about it. The
concept looks quite good, and I'm very intrigued to find out more
now...
Thanks for your feedback so far. This is why I sent it to the mailing
list in the first place :)
alex....@gmail.com
Jan 4, 2009, 4:27:09 AM1/4/09
to Django developers
Ok, but you're original concern was that what happens if say you go to
send an email and "raises the issue that e-mail may take a while (i.e.
from a few seconds to a couple of minutes) to send;", ultimately if
you join on that thread than you've reduce the overhead insofar as the
new length of the request is now length(sending the email) - amount of
the time the rest of the stuff you do before you join() takes, but if
it's taking 30 seconds to send the email, you're still ultimately
going to be taking a minimum of 30 seconds to serve that request.
Alex
send an email and "raises the issue that e-mail may take a while (i.e.
from a few seconds to a couple of minutes) to send;", ultimately if
you join on that thread than you've reduce the overhead insofar as the
new length of the request is now length(sending the email) - amount of
the time the rest of the stuff you do before you join() takes, but if
it's taking 30 seconds to send the email, you're still ultimately
going to be taking a minimum of 30 seconds to serve that request.
Alex
Graham Dumpleton
Jan 4, 2009, 5:10:34 AM1/4/09
to Django developers
On Jan 4, 8:12 pm, zvoase <crack...@gmail.com> wrote:
> > For one simple example: what if Apache decides to kill or restart the Django process
> > after a certain amount of requests have been processed (and I know
> > that many people have their servers set to do this)? The threads
> > which were are running asynchronous tasks would be killed as well,
> > losing any tasks which hadn't yet been completed.
>
> To be honest, I hadn't thought about that; that indeed would make it
> difficult to implement async signals. However, the aggregated 'join'
> method could be called after the receivers have all finished, as
> Apache will not terminate the Django process until all requests have
> been handled; so there are probably ways around this.
complete before shutting down child worker processes. For a non
graceful restart or shutdown, Apache will give the process about 3-4
seconds to exit and if it doesn't exit then it will kill it. Thus if
there were active requests which take longer than that to complete,
then they will be interrupted.
There is a bit more to it than this but exact details depend on which
hosting mechanism you are using in conjunction with Apache.
Graham
zvoase
Jan 4, 2009, 9:15:24 AM1/4/09
to Django developers
On Jan 4, 10:27 am, "alex.gay...@gmail.com" <alex.gay...@gmail.com>
wrote:
going to need to fix that as a bottleneck; this might be a nice
solution to a problem, but it won't be able to fix *everything*. Not
to mention that if it's taking 30s for 1 email, a batch system is
going to take aaaaages to flush the e-mail queue.
wrote:
> Ok, but you're original concern was that what happens if say you go to
> send an email and "raises the issue that e-mail may take a while (i.e.
> from a few seconds to a couple of minutes) to send;", ultimately if
> you join on that thread than you've reduce the overhead insofar as the
> new length of the request is now length(sending the email) - amount of
> the time the rest of the stuff you do before you join() takes, but if
> it's taking 30 seconds to send the email, you're still ultimately
> going to be taking a minimum of 30 seconds to serve that request.
That's true, but then if e-mail is taking 30s to send you're probably
> send an email and "raises the issue that e-mail may take a while (i.e.
> from a few seconds to a couple of minutes) to send;", ultimately if
> you join on that thread than you've reduce the overhead insofar as the
> new length of the request is now length(sending the email) - amount of
> the time the rest of the stuff you do before you join() takes, but if
> it's taking 30 seconds to send the email, you're still ultimately
> going to be taking a minimum of 30 seconds to serve that request.
going to need to fix that as a bottleneck; this might be a nice
solution to a problem, but it won't be able to fix *everything*. Not
to mention that if it's taking 30s for 1 email, a batch system is
going to take aaaaages to flush the e-mail queue.
zvoase
Jan 4, 2009, 9:19:48 AM1/4/09
to Django developers
On Jan 4, 11:10 am, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
*any* hopes of things finishing nicely. In this case, Apache is
forcefully terminating a process; ideally a set-up where processes
were being killed in the middle of their execution is not going to use
asynchronous *anything*, and also will want to have all DB work going
on in transactions; having an Apache which just keeps killing stuff is
going to be dangerous for code no matter what. And it shouldn't be
taking 4 seconds for the user to receive a request, so by the end of
that time all work should be done. If something *needs* to be done
asynchronously that's going to take more than several seconds, then it
should probably be delegated to a daemon running separately from the
web server.
wrote:
> Not true, Apache doesn't necessarily wait for all current requests to
> complete before shutting down child worker processes. For a non
> graceful restart or shutdown, Apache will give the process about 3-4
> seconds to exit and if it doesn't exit then it will kill it. Thus if
> there were active requests which take longer than that to complete,
> then they will be interrupted.
>
> There is a bit more to it than this but exact details depend on which
> hosting mechanism you are using in conjunction with Apache.
>
> Graham
But if we're doing a non-graceful restart, then we can say goodbye to
> complete before shutting down child worker processes. For a non
> graceful restart or shutdown, Apache will give the process about 3-4
> seconds to exit and if it doesn't exit then it will kill it. Thus if
> there were active requests which take longer than that to complete,
> then they will be interrupted.
>
> There is a bit more to it than this but exact details depend on which
> hosting mechanism you are using in conjunction with Apache.
>
> Graham
*any* hopes of things finishing nicely. In this case, Apache is
forcefully terminating a process; ideally a set-up where processes
were being killed in the middle of their execution is not going to use
asynchronous *anything*, and also will want to have all DB work going
on in transactions; having an Apache which just keeps killing stuff is
going to be dangerous for code no matter what. And it shouldn't be
taking 4 seconds for the user to receive a request, so by the end of
that time all work should be done. If something *needs* to be done
asynchronously that's going to take more than several seconds, then it
should probably be delegated to a daemon running separately from the
web server.
Malcolm Tredinnick
Jan 4, 2009, 6:54:13 PM1/4/09
to django-d...@googlegroups.com
Your response here and in a couple of earlier mails are pointing you in
the right direction. You've arrived at the point where asynchronous
signals are not a good idea. The approach is to synchronously set up an
entry in your external queue system to process anything that is going to
take a long time. The signal processing is thus very short and the
long-term operations are in no way tied to webserver process lifecycles
or client wait times or anything. It's directly analogous to top-half /
bottom-half interrupt handlers in operating systems, for example.
I don't see the requirement for extra signal infrastructure complexity
in anything you've written in this thread. All your examples seem much
better handled by dispatching to an external processing queue.
Malcolm
zvoase
Jan 4, 2009, 8:30:48 PM1/4/09
to Django developers
On Jan 5, 12:54 am, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
Yeah, I suppose so :) Well thanks a lot everyone, you've helped make
things a bit more clear, and this thread is also probably going to be
useful for future reference for anyone who's thinking of bloating the
signals framework ;-)
Regards,
Zack
wrote:
things a bit more clear, and this thread is also probably going to be
useful for future reference for anyone who's thinking of bloating the
signals framework ;-)
Regards,
Zack
Reply all
Reply to author
Forward
0 new messages