multiplex loop bypass, redundant syscalls to iopoll

[הלאַוואַק סטאַנדאַ‎]

Hi dude,

Thanks for awesome library, works great so far.
I've been also poking around in the code and before I go to bed, I want to ask.

can you explain a little the multiplexing thing, which is in base.py:167
1) what i observed is, that if there are still data to be received on socket (result ends up in True), you're bypassing whole event/IO POLLIN loop for notifications about newly received not yet processed data on this socket.
i don't understand, why would you want to skip the loop?
what's the designed purpose of multiplex?
can you please advice why shouldn't I turn it off?

2) also in the second case (if the result ends up in False), then you're adding duplicated socket's FD into the IO loop over and over again. I understand that duplicated sockets in pool don't matter, but i think another redundant syscall for every received message does.

3) and is this thing still alive? :-)

link to source:
https://code.google.com/p/cogen/source/browse/trunk/cogen/core/proactors/base.py#167

regards,
-sh

[Ionel Maries Cristian]

Hey,

1) Bypassing the event loop was an optimization, for example when you expect to receive large amount of data then you want to keep reading till you exhausted the OS's buffers without asking the event loop every time if the fd is ready, because, most probably is is already ready.

2) You have a point here but the fd wouldn't be in the event loop before adding it anyway. It's unregistered after the token is completed. If it already there then it's a bug somewhere :)

3) Well ... not really. There's no activity for almost 4 years, there wasn't much interest in this generator based approach ...

Thanks,
-- Ionel M.

--
You received this message because you are subscribed to the Google Groups "cogen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cogen+un...@googlegroups.com.
To post to this group, send email to co...@googlegroups.com.
Visit this group at http://groups.google.com/group/cogen.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[הלאַוואַק סטאַנדאַ‎]

Hello again,

Thanks for awesome reply,

On Monday, August 12, 2013 2:00:26 PM UTC+2, Maries Ionel Cristian wrote:

Hey,

1) Bypassing the event loop was an optimization, for example when you expect to receive large amount of data then you want to keep reading

Makes sense, that's exactly what i observed while stracing it.
 

till you exhausted the OS's buffers without asking the event loop every time if the fd is ready, because, most probably is is already ready.

2) You have a point here but the fd wouldn't be in the event loop before adding it anyway. It's unregistered after the token is completed. If it

already there then it's a bug somewhere :)

I've been monitoring the fd's unregistration and it didn't go through it.
 

3) Well ... not really. There's no activity for almost 4 years, there wasn't much interest in this generator based approach ...

This part is interesting. Why do you think it's not? My completely biased opinion based on nothing is, that it's so much more superior and pure to gevent/eventlep/greenlet (stack based switch) or tornado's based async calls solutions. There're also not so many alternatives, I've been checking also weightless, but it seems it's doing so much redundant unnecessary stuff and I'm really bothered by the PEP380 C extension. Then there is multitask, which seems like the only reasonable solution after cogen, but multitask seems to be dead as well.

[Ionel Maries Cristian]

On Tue, Aug 13, 2013 at 12:01 AM, הלאַוואַק סטאַנדאַ <obl...@gmail.com> wrote:

 

3) Well ... not really. There's no activity for almost 4 years, there wasn't much interest in this generator based approach ...

This part is interesting. Why do you think it's not? My completely biased opinion based on nothing is, that it's so much more superior and pure to gevent/eventlep/greenlet (stack based switch) or tornado's based async calls solutions. There're also not so many alternatives, I've been checking also weightless, but it seems it's doing so much redundant unnecessary stuff and I'm really bothered by the PEP380 C extension. Then there is multitask, which seems like the only reasonable solution after cogen, but multitask seems to be dead as well.

Maybe it was too purist for it's own good:

• Having to use generators for everything made everything convoluted - because there was no "yield from" - and there still isn't any in python2 - you have to tra​de readability and encapsulation for performance. Calling a couroutine/generator from another coroutine/generator has some overhead.
• Hard to integrate with 3rd party blocking code. For all the wrongs this is where gevent/eventlet/greenlet shines. Just monkeypatch some stuff and everything (appears) to work.
• Platform support too broad, supporting epoll, poll, select, kqueue, iocp (which is and behaves wildly different than the others) created some overhead. Some data structures could be simpler if cogen were only to support say, epoll. I even had a PyQt event loop at some point ... but I ditched it after a big refactor.
  

​I would look at pep-3156 ​and http://code.google.com/p/tulip/​ ... maybe that will make the generator based apporoach better understood and popular ...