Re: friendlyflow
Erik Groeneveld
(hope cc to cogen works, as it seems picky on the exact email address)
I am very glad that you are paying attention to both Weightless and
Cogen. The community is indeed fragmented, but I believe Cogen,
Weightless and perhaps MultiTask (see
http://www.weightless.io/weightless, Related Work) are serious pieces
of software. Some sort of joint effort would be good for all.
As for the code base, I really do not care that much for the
Reator/Acceptor in Weightless. I can trade them in for another ones.
My original idea also included the exploration of true async I/O and
kernel buffering (sendfile, splice etc), but I haven't been able to
work on that part. So Acceptor/Reactor are really not so special.
The most important parts I do care about are 'compose' and 'gio'.
Both of these enable the dynamic building of efficient pipelines
(Jackson SP) based on generalized generators (coroutines). The 'gio'
component ties a ('compose'd) generator to a socket, that's all. It
could be done with Cogen as well I think. The use of splice/tee etc
should be transparent to gio.
The 'compose' part is finished. It is a way of decomposing a program
into generators. It is quite stand-alone as well.
The gio component is new and just completely rewritten (as all other
components were a couple of times) and I am just about to take it into
production at one place to see it work. The component is still under
heavy research, it could change, although I am quite confident about
the interface at this point.
My suggestion would be to
- tie composen and gio to Cogen,
- replace my Weightless with Cogen/Weightless on all places
- Cogen/Weightless must be somehow compatible with the way we use
Weightless at this moment (I am thinking about timer support for
example)
- take it easy with the gio part, it is not finished like compose,
things must not go to fast
Any comments? Plan a pair programming session?
Best regards,
Erik
E.J. Groeneveld
Seek You Too
twitter, skype: ejgroene
mobiel: 0624 584 029
On Sun, Apr 12, 2009 at 17:51, Klaas van Schelven <kl...@vanschelven.com> wrote:
> Hi Andrew, Erik,
>
> First off, I hope you don't mind me cc-ing to Erik: he's very knowledgeable
> on the subject and could I figured putting the two of you in contact would
> be valuable for the both of you. Erik: welcome to the conversation.
>
> To say I've worked with cogen is a bit too much honor: I've opened the
> package and played with it a little bit.
>
> Cogen and friendly flow pretty much take the same approach, though cogen is
> both much more advanced and has a more complicated codebase. The approach is
> to use Python's yield to yield flow, and otherwise maintain pythonesque ways
> of passing data through the program.
> Weightless is more radical in that it passes data in the yield calls (and
> simultaniously yields control). I've created Friendly Flow at the time to
> sketch my intuition of a more clean separation of concerns. However, that
> project never served a role in a bigger whole so must be considered play
> code. Weightless has several production servers and 5 figure projects
> running on top of it.
>
> All projects demand a high level of involvement to be of value given the
> subject matter and their maturity. In any case I would recommend downloading
> and playing with all of them a little bit. In fact, for this purpose even
> Friendly Flow is useful: because it's so rudimentary it's codebase is very
> easy to understand (though I'm biased).
>
> May I inquire what kind of plans do you have with these projects?
>
> ciao, (Erik: see you next week),
> Klaas
>
> On Sat, Apr 11, 2009 at 1:32 PM, Andrew McNabb <amc...@mcnabbs.org> wrote:
>>
>> I really value your perspective since you've worked with both Cogen and
>> Weightless. Are there any big differences in philosophy between the
>> projects, or would there be any chance of them joining at some point?
>> Are they at about the same level of maturity?
>>
>>
>> On Sat, Apr 11, 2009 at 11:32:48AM -0400, Klaas van Schelven wrote:
>> > Hi
>> >
>> > The project is no longer active. I've subscribed to the cogen mailing
>> > list.
>> >
>> > Weightless is still active, I'm working on it whenever Erik
>> > Groeneveld
>> > pays me to do so :-) (weightless is used in "Meresco", which is an
>> > Open
>> > Source project developed by "Seek You Too" whenever they get paid to
>> > do
>> > so)
>> >
>> > Klaas
>> >
>> > On Thu, Apr 9, 2009 at 6:39 PM, Andrew McNabb
>> > <[1]amc...@mcnabbs.org>
>> > wrote:
>> >
>> > I was looking at networking systems using Python generators, and
>> > friendlyflow came up. �Is this project still active? �If so, have
>> > you
>> > considered collaborating with cogen and/or weightless? �It seems
>> > like
>> > this particular developer community is very fragmented.
>> >
>> > Thanks.
>> > --
>> > Andrew McNabb
>> > [2]http://www.mcnabbs.org/andrew/
>> > PGP Fingerprint: 8A17 B57C 6879 1863 DE55 �8012 AB4D 6098 8826 6868
>> >
>> > --
>> > [3]www.xaba.nl
>> > 06 811 599 10
>> > Skype: klaasvanschelven
>> > Twitter: vanschelven
>> >
>> > References
>> >
>> > Visible links
>> > 1. mailto:amc...@mcnabbs.org
>> > 2. http://www.mcnabbs.org/andrew/
>> > 3. http://www.xaba.nl/
>>
>> --
>> Andrew McNabb
>> http://www.mcnabbs.org/andrew/
>> PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
>
>
>
> --
> www.xaba.nl
> 06 811 599 10
> Skype: klaasvanschelven
> Twitter: vanschelven
>
Erik Groeneveld
as it comes to how coroutines work. Cogen has:
request = yield sock.read()
yield sock.write("GOOD BYE")
while Weightless has:
request = yield
yield "GOOD BYE"
This has been a hot topic in my head, and I have rewritten the code a
few times to do the first or the second, and in the end I concluded
that there is no difference! With weightless one could write:
class Socket(object):
def read(self):
data = yield
raise StopIteration(data)
def write(self, data):
yield data
sock = Socket(...)
and then write the same code as with Cogen:
request = yield sock.read()
yield sock.write("GOOD BYE")
The trick is in compose. Compose allows a coroutine to yield other
coroutines that continue the pipeline. The reason why I care so much
about writing this:
request = yield
yield "GOOD BYE"
is because of the fact that I do not think about processes and
scheduling, but in terms of pipelines/JSP. These pipelines could be
attached to sockets, but also to files, or to nothing at all during
unittesting.
This looks like a major difference, but I think it isn't. Weightless
supports the read/write style as well, and Cogen can wrap a socket and
present the data to a generator just like gio does.
BTW, two different thinking modes (scheduling vs pipelines) have been
bothering me last years. It it hard to separate them, and actually,
by switching every now and then, it help to get clear what the code I
typed actually became..
Best regards,
Erik
E.J. Groeneveld
Seek You Too
twitter, skype: ejgroene
mobiel: 0624 584 029
Ionel Maries Cristian
While I don't want to ruin it for you I must note that cogen doesn't decompose
generators at all - all the generators are wrapped in a special class that the scheduler
and proactors know how to call.
While it does have some disadvantages (additional complexity) it separates some
aspects - on could just plug in another coroutine architecture (like greenlets) by just
providing another Couroutine class with similar interface.
So where you would yield a generator instance in weightless, you would yield a
Coroutine instance in cogen. And the scheduler just calls that instance like other
instances (like a socket read request) and stuff happens behind the scenes.
Way back there were some discussions on making a coroutine specification and
have interchangeable components and sadly, it ended with a divergent discussion on
this topic (wrapping the generators or not).
The original proposal is here: http://groups.google.com/group/python-coro/
Well anyway, there are plenty of advantages and disadvantages on each side
and I'm still not sure I made the right thing.
I still want some feedback on this topic from you guys.
Another problem is that cogen doesn't work in a pipeline style like weightless.
Well, I need to think a bit on that one - the pipeline stuff in weightless looks
very good.
The proactor code right now it's a bit coupled with the scheduler but it's
remediable (it was worse - network calls all over the place, now they are
just in the proactor)
I think we could build something on top of the proactor if you don't like a
scheduler-based architecture :)
I'll look more on the weightless code in the upcoming days - I don't have yet
a good overview of it to make good suggestions on how to approach this.
-- ionel
Andrew McNabb
>
> May I inquire what kind of plans do you have with these projects?
I'm a disappointed Twisted user. I'm a bit tired of the spaghetti flow
of callbacks, the pervasive overengineering, and the web/web2 thing (no
HTTP/1.1 support). Anyway, I think the generator approach is
intriguing. As I mentioned in the emails that started this discussion,
I think that everyone in the Python generators world are doing really
interesting things, and I would love to see them succeed. I am even
willing to help, but it would be easier to motivate myself if there were
a single project with a bigger development community.
Andrew McNabb
>
> Oh, I almost forgot the main difference between Cogen and Weightless
> as it comes to how coroutines work. Cogen has:
>
> request = yield sock.read()
> yield sock.write("GOOD BYE")
>
> while Weightless has:
>
> request = yield
> yield "GOOD BYE"
> BTW, two different thinking modes (scheduling vs pipelines) have been
> bothering me last years. It it hard to separate them, and actually,
> by switching every now and then, it help to get clear what the code I
> typed actually became..
This really is an interesting difference, and I can definitely see the
argument for both. Would it be possible to have both of them available,
perhaps using a decorator to distinguish between them? I'm not familiar
enough with either framework yet to have any intuition about whether
this is feasible.
Erik Groeneveld
> On Sun, Apr 12, 2009 at 07:09:52PM +0200, Erik Groeneveld wrote:
>>
>> Oh, I almost forgot the main difference between Cogen and Weightless
>> as it comes to how coroutines work. Cogen has:
>>
>> request = yield sock.read()
>> yield sock.write("GOOD BYE")
>>
>> while Weightless has:
>>
>> request = yield
>> yield "GOOD BYE"
>
>> BTW, two different thinking modes (scheduling vs pipelines) have been
>> bothering me last years. It it hard to separate them, and actually,
>> by switching every now and then, it help to get clear what the code I
>> typed actually became..
>
> This really is an interesting difference, and I can definitely see the
> argument for both. Would it be possible to have both of them available,
> perhaps using a decorator to distinguish between them? I'm not familiar
> enough with either framework yet to have any intuition about whether
> this is feasible.
decorator. In fact, the gio component just turns callback based I/O
into a generator. You could equally well let it execute command that
are being yielded.
BTW The reason why Weightless' Reactor is just a simple reactor
performing callbacks on I/O events is exactly because after a year of
refactoring, I discovered that the Reacctor/Scheduler really does not
need to know anything about generators at all. So I seperated the
concerns.
Erik
Andrew McNabb
acquainted with cogen.
On Sun, Apr 12, 2009 at 09:24:43PM +0300, Ionel Maries Cristian wrote:
> Hey guys,
>
> While I don't want to ruin it for you I must note that cogen doesn't
> decompose
> generators at all - all the generators are wrapped in a special class that
> the scheduler
> and proactors know how to call.
> While it does have some disadvantages (additional complexity) it separates
> some
> aspects - on could just plug in another coroutine architecture (like
> greenlets) by just
> providing another Couroutine class with similar interface.
> So where you would yield a generator instance in weightless, you would
> yield a
> Coroutine instance in cogen. And the scheduler just calls that instance
> like other
> instances (like a socket read request) and stuff happens behind the
> scenes.
While it sounds like a really nice idea to be able to plug in a
different architecture like greenlets, I'm not completely convinced that
generators and greenlets are completely compatible. For example, it
looks like "yield from" is going to be added to Python 2.7 and 3.1.
This would allow a coroutine to "call" another coroutine; it's basically
generator composition being added to the language.
Is it really possible to fully support generators (with composition) and
greenlets at the same time, or would this require sacrificing both? In
other words, would cogen be a "jack of all trades, master of none"? The
more I think about it, the more I'm convinced that composition (yield
from) is the "right" way to do generators in Python.
By the way, I don't think this precludes having a scheduler
architecture. In fact, I don't think the scheduler would have to change
much at all to support "yield from". I think that the scheduler would
wrap each added generator in a Coroutine instance as part of the process
of adding it to the scheduler. If there were a stack of generators
linked with "yield from", only the first would be wrapped because the
other ones would be linked transparently.
I don't think it would be a radical change, but it might sacrifice
support for greenlets in order to support composed generators better.
Erik Groeneveld
I though I'd serve you better by making both the Weightless web-site
and the code better readable (especially the site was bad). The most
notable change is that I removed all references to asynchronous I/O
and related things.
Your questions (and those of others) really made me conclude that the
way I/O is performed is not the main contribution of Weightless.
Instead, it is the way Weightless deals with the data that makes it
unique.
If Cogen deals with sending and receiving data efficiently, using
Reactors or Proactors etc, then Weightless can deal with processing
the data efficiently. The interesting thing is that there is no
impedance mismatch: Weightless neatly fits the asynchronous nature of
Cogen.
The web-site (http://weightless.io) now makes clear what it is about:
1. decomposing programs into coroutines using compose
2. creating pipelines using the observer pattern
3. connecting file descriptors (sockets etc) to pipelines using gio
The 3rd step involves turning callbacks or continuations into
feeding/reading generators, which could be based on any the I/O
classes of Cogen.
The steps 1 and 2 refer to using coroutines to process data in a way
Micheal Jackson described in 1975. This is substantially different
from using generators to do I/O scheduling, as Cogen does. To make it
clear, I believe there are two problems:
1. I/O scheduling
2. data-procosessing
Since both can be solved with generators, and therefor, both solutions
neatly fit each other, I confused them for years, but I now believe
they are really distinct problems.
The new yield-from proposed in PEP 380 seems more aimed about the
scheduling problem than at the data-processing problem. On
python-ideas, I tried to make a case for the data-processing problem
as another type of applications for using generators, but I completely
failed. Perhaps because me myself is only beginning to understand the
stuff well.
Anyway, I though I would be good to let you know the changing
proposition I am making with Weightless. I think I makes Weightless
and Cogen an even better fit doesn't it?
Now that I've said that, I'll have to make time to let Weightless run
atop of Cogen.
Erik
E.J. Groeneveld
Seek You Too
twitter, skype: ejgroene
mobiel: 0624 584 029