Twisted: 1 thread in the reactor pattern
jacopo
I would like to understand more about what happens behind the
curtains.
Say I have a client and a server on two different machines, the server
gets callRemote()’s in an asynchronous way, these requests are parked
in a queue and then served sequentially (not in parallel – correct me
if I am wrong). If everything is implemented in a single thread, how
is it possible that while the processor is engaged in the processing
triggered by callRemote()’s at the same time the reactor is ready to
listen/accept new events and put them in a queue? To me it looks like
there should be at least 2 processes, one for the reactor and on for
the rest.
In the documentation they keep stressing how one of the peculiarity of
the reactor pattern is the single thread, but I can not figure out
how.
Any suggestion would be welcome.
For the time being I recommend this doc which has been the most useful
to me so far.
http://www.artima.com/weblogs/viewpost.jsp?thread=230001
Thanks, Jacopo
exa...@twistedmatrix.com
>I am diving into Twisted and Perspective Broker (PB) in particular and
>I would like to understand more about what happens behind the
>curtains.
>Say I have a client and a server on two different machines, the server
>in a queue and then served sequentially (not in parallel 13 correct me
>if I am wrong).
Since, as you point out below, there is only one thread, the remote
methods can only be invoked one at a time.
However, rather central to the asynchronous operation of Twisted
libraries and applications, the remote method itself may return before
the remote call has been completely serviced. So while only one remote
method will run at a time, each of the two remote calls may run
concurrently at some point before they are responded to.
>If everything is implemented in a single thread, how
>is it possible that while the processor is engaged in the processing
>triggered by callRemote() 19s at the same time the reactor is ready to
>listen/accept new events and put them in a queue? To me it looks like
>there should be at least 2 processes, one for the reactor and on for
>the rest.
It isn't possible. While the remote methods are running, other events
are not being serviced. This is what is meant when people describe
Twisted as a "*cooperative* multitasking" system. Event handlers (such
as remote methods) run for a short enough period of time that it doesn't
matter that the reactor is prevented from accepting new connections (or
what have you) for the duration of their execution.
Jean-Paul
jacopo
Jean -Paul, not sure I have understood.
Say I have one server S and two clients C1 and C2 (all on separate
machines).
(a) C1 requests a remote call of f1() to S, f1() requires 5 minutes of
processing.
(b) S puts f1() in a queue and returns immediately a Deferred to
C1.
(c) Now f1() starts and keeps S’s processor busy for 5 mins
(d) after few seconds C2 requests a remote call f2() to S.
(e) On S the processor is already engaged with f1() but still
“someone” on S is able to accept the request from C2, put it in a
queue (after f1()) and return a Deferred to C2.
(f) At some point after f1() is finished f2() will start
I believe (b) is what you say “run for a short enough period of time
that it doesn't
matter that the reactor is prevented from accepting new connections
(or
what have you) for the duration of their execution.” ?!
Don’t we have (c) and (e) running at the same time here? How is it
possible to have only one Thread?
Thanks,
Jacopo
exa...@twistedmatrix.com
>>
>>It isn't possible. While the remote methods are running, other events
>>are not being serviced. This is what is meant when people describe
>>Twisted as a "*cooperative* multitasking" system. Event handlers
>>(such
>>as remote methods) run for a short enough period of time that it
>>doesn't
>>matter that the reactor is prevented from accepting new connections
>>(or
>>what have you) for the duration of their execution.
>>
>>Jean-Paul
>
>Jean -Paul, not sure I have understood.
>Say I have one server S and two clients C1 and C2 (all on separate
>machines).
>
>(a) C1 requests a remote call of f1() to S, f1() requires 5 minutes
>of
>processing.
>(b) S puts f1() in a queue and returns immediately a Deferred to
>C1.
>(d) after few seconds C2 requests a remote call f2() to S.
>(e) On S the processor is already engaged with f1() but still
>queue (after f1()) and return a Deferred to C2.
>(f) At some point after f1() is finished f2() will start
>
>that it doesn't
>matter that the reactor is prevented from accepting new connections
>(or
If you have a function that takes 5 minutes to run, then you're blocking
the reactor thread for 5 minutes and no other events are serviced until
the function finishes running.
You have to avoid blocking the reactor thread if you want other events
to continue to be serviced. There are various strategies for avoiding
blocking. Different strategies are appropriate for different kinds of
blocking code.
Jean-Paul
jacopo
>
> - Show quoted text -
Even if the server is engaged in a 5 minutes processing other arriving
requests of callRemote() are queued and Deferreds are returned
immediately. So the reactor is responding somehow.
Isn’t it?!
On the Server I am using: reactor.listenTCP(port, pb.PBServerFactory
(MyClass()))
Could you suggest me any doc to better understand?
Thanks, Jacopo
exa...@twistedmatrix.com
>On Sep 24, 7:54 pm, exar...@twistedmatrix.com wrote:
>>On 07:10 am, jacopo.pe...@gmail.com wrote:
>> >On Sep 23, 5:57 pm, exar...@twistedmatrix.com wrote:
>> >[snip]
>>
>>If you have a function that takes 5 minutes to run, then you're
>>blocking
>>the reactor thread for 5 minutes and no other events are serviced
>>until
>>the function finishes running.
>>
>>You have to avoid blocking the reactor thread if you want other events
>>to continue to be serviced. There are various strategies for avoiding
>>blocking. Different strategies are appropriate for different kinds of
>>blocking code.
>>
>>Jean-Paul- Hide quoted text -
>>
>>- Show quoted text -
>
>Even if the server is engaged in a 5 minutes processing other arriving
>requests of callRemote() are queued and Deferreds are returned
>immediately.
Nope, they're not. The bytes representing the new requests sit in the
socket buffer until the function finishes processing and the reactor
gets an opportunity to read them.
>
>Could you suggest me any doc to better understand?
If you haven't read
http://twistedmatrix.com/projects/core/documentation/howto/async.html
yet, that may be a good idea.
Jean-Paul