Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

fiber(cooperative multi-threading)

8 views
Skip to first unread message

Akihiro KAYAMA

unread,
Dec 22, 2007, 7:10:39 AM12/22/07
to pytho...@python.org

Hi all.

I found cooperative multi-threading(only one thread runs at once,
explicit thread switching) is useful for writing some simulators.
With it, I'm able to be free from annoying mutual exclusion, and make
results deterministic.

For this purpose, and inspired by Ruby(1.9) fiber, I wrote my own
version of fiber in Python.

It just works, but using native Python threads for non-preemptive
threading is not cost-effective. Python has generator instead but it
seemed to be very restricted for general scripting. I wish I could
write nested (generator) functions easily at least.

Is there any plan of implementing real (lightweight) fiber in Python?

------------------------------------------------------------
import threading

class Fiber(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)

self.semaphore_running = threading.Semaphore(0)
self.semaphore_finish = None
self.val = None

self.setDaemon(True)
self.start()
self.start = self.start_fiber

def start_fiber(self):
self.semaphore_finish = threading.Semaphore(0)
self.semaphore_running.release()
self.semaphore_finish.acquire()

def run(self): # override
self.semaphore_running.acquire()
self.main()
if self.semaphore_finish is not None:
self.semaphore_finish.release()

def switchto(self, fiber, val=None):
fiber.val = val
fiber.semaphore_running.release()
self.semaphore_running.acquire()
return self.val

def main(self): # should be overridden
pass

class F1(Fiber):
def main(self):
print "f1 start"
self.switchto(f2)
print "f1 foo"
v = self.switchto(f2)
print "f1 v=%s world" % v
self.switchto(f2, "OK")
print "f1 end"

class F2(Fiber):
def main(self):
print "f2 start"
self.switchto(f1)
print "f2 bar"
result = self.switchto(f1, "Hello, ")
print "f2 result=%s" % result
print "f2 end"
self.switchto(f1)

f1 = F1()
f2 = F2()

print "start"
f1.start()
print "end"

-- kayama

Arnaud Delobelle

unread,
Dec 22, 2007, 8:53:25 AM12/22/07
to

I am not really familiar with ruby but these fibers seem to be some
sort of coroutines. Since python 2.5, generators can be sent values,
this can be used to implement what you want. I have had a got at it
for fun and this is what I came up with: note that there is no need
for a specific class, or for threads. I have tested this only on your
example, but it gives the same result as your code :)

-------------------------------------

# A 'fiber' is a generator function that yields tuples (fiber, value)
# or (fiber,). The 'run' function starts the the fiber and carries on
# with the fiber it yields. If the fiber also yields a value, then
# this value is sent to the 'next' fiber.

def run(fiber):
it, val = fiber(), []
fibers = { fiber: it }
try:
while True:
n = it.send(*val) if val else it.next()
co, val = n[0], n[1:]
it = fibers.get(co)
if it is None:
fibers[co] = it = co()
except StopIteration:
return val[0] if val else None

# Now the two 'fibers'

def f1():
print "f1 start"
yield f2,
print "f1 foo"
v = yield f2,


print "f1 v=%s world" % v

yield f2, "OK"
print "f1 end"

def f2():
print "f2 start"
yield f1,
print "f2 bar"
result = yield f1, "Hello, "


print "f2 result=%s" % result
print "f2 end"

yield f1,

print "start"
run(f1)
print "end"

--------------------------------------

HTH

--
Arnaud

Duncan Booth

unread,
Dec 22, 2007, 9:37:25 AM12/22/07
to
Arnaud Delobelle <arn...@googlemail.com> wrote:

> I am not really familiar with ruby but these fibers seem to be some
> sort of coroutines. Since python 2.5, generators can be sent values,
> this can be used to implement what you want. I have had a got at it
> for fun and this is what I came up with: note that there is no need
> for a specific class, or for threads. I have tested this only on your
> example, but it gives the same result as your code :)

I think it was Microsoft who invented the term 'fiber' as a synonym for
coroutine. See http://msdn2.microsoft.com/en-us/library/ms682661.aspx

Unfortunately generators only save a single level of stack-frame, so they
are not really a replacement for fibers/coroutines. The OP should perhaps
look at Stackless Python or Greenlets. See
http://codespeak.net/py/dist/greenlet.html

Arnaud Delobelle

unread,
Dec 22, 2007, 9:55:36 AM12/22/07
to
On Dec 22, 2:37 pm, Duncan Booth <duncan.bo...@invalid.invalid> wrote:

> Arnaud Delobelle <arno...@googlemail.com> wrote:
> > I am not really familiar with ruby but these fibers seem to be some
> > sort of coroutines.  Since python 2.5, generators can be sent values,
> > this can be used to implement what you want.  I have had a got at it
> > for fun and this is what I came up with: note that there is no need
> > for a specific class, or for threads.  I have tested this only on your
> > example, but it gives the same result as your code :)
>
> I think it was Microsoft who invented the term 'fiber' as a synonym for
> coroutine. Seehttp://msdn2.microsoft.com/en-us/library/ms682661.aspx

OK

> Unfortunately generators only save a single level of stack-frame, so they
> are not really a replacement for fibers/coroutines. The OP should perhaps
> look at Stackless Python or Greenlets. Seehttp://codespeak.net/py/dist/greenlet.html

Still what I am proposing offers a level of cooperative multi-
threading which may be enough for the OP. In fact it is not clear to
me at the moment what can be done (sensibly :) with the OP's Fiber
class that cannot be achieved with the run() function I suggested.
This makes me think that there is something important that I am
failing to take into consideration; I would be grateful if it was
pointed out to me.

--
Arnaud

Akihiro KAYAMA

unread,
Dec 23, 2007, 4:16:31 AM12/23/07
to pytho...@python.org

Thanks for your replies.

In article <37c5a53c-1a39-47af...@i72g2000hsd.googlegroups.com>,
Arnaud Delobelle <arn...@googlemail.com> writes:

arnodel> def f1():
arnodel> print "f1 start"
arnodel> yield f2,
arnodel> print "f1 foo"
arnodel> v = yield f2,
arnodel> print "f1 v=%s world" % v
arnodel> yield f2, "OK"
arnodel> print "f1 end"
arnodel>
arnodel> def f2():
arnodel> print "f2 start"
arnodel> yield f1,
arnodel> print "f2 bar"
arnodel> result = yield f1, "Hello, "
arnodel> print "f2 result=%s" % result
arnodel> print "f2 end"
arnodel> yield f1,

This is the most simple example. In real programming, things are more
complicate so I will want to refactor it like below:

def foo(fiber, s, arg=None)
print s
return yield fiber, arg

def f1():
foo(f2, "start") # XXX returns generator object
v = foo(f2, "foo")
foo(f2, "v=%s world" % v, "OK")

But current Python generator specification requires me:

def f1():
for x in foo(f2, "foo"): yield x
for x in foo(f2, "foo"): yield x
# XXX v = ... (I don't know how to do this)
for x in foo(f2, "v=%s world" % v, "OK"): yield x

I think it is not straitforward. Single level function which generator
impose is impractical for real use.

In article <Xns9A0E94C1BC...@127.0.0.1>,
Duncan Booth <duncan...@invalid.invalid> writes:

duncan.booth> Unfortunately generators only save a single level of stack-frame, so they
duncan.booth> are not really a replacement for fibers/coroutines. The OP should perhaps
duncan.booth> look at Stackless Python or Greenlets. See
duncan.booth> http://codespeak.net/py/dist/greenlet.html

I am happy if I could use convenient coroutine features via standard
or simple extension library. py.magic.greenlet may be what I'm
looking for, but I wonder why this is named "magic" :-)

-- kayama

Duncan Booth

unread,
Dec 23, 2007, 5:43:57 AM12/23/07
to
Akihiro KAYAMA <kay...@st.rim.or.jp> wrote:

> Thanks for your replies.


>
> But current Python generator specification requires me:
>
> def f1():
> for x in foo(f2, "foo"): yield x
> for x in foo(f2, "foo"): yield x
> # XXX v = ... (I don't know how to do this)
> for x in foo(f2, "v=%s world" % v, "OK"): yield x
>
> I think it is not straitforward. Single level function which generator
> impose is impractical for real use.

Not just impractical, there are plenty of situations where you simply
cannot do that at all.

Here's a greenlet example (from
http://socal-piggies.org/presentations/grig/2005_07_21/):

from py.magic import greenlet
import xml.parsers.expat

def send(arg):
greenlet.getcurrent().parent.switch(arg)

def start_element(name, attrs):
send(('START', name, attrs))
def end_element(name):
send(('END', name))
def char_data(data):
data = data.strip()
if data: send(('DATA', data))

def greenparse(xmldata):
p = xml.parsers.expat.ParserCreate()
p.StartElementHandler = start_element
p.EndElementHandler = end_element
p.CharacterDataHandler = char_data
p.Parse(xmldata, 1)

def iterxml(xmldata):
g = greenlet(greenparse)
data = g.switch(xmldata)
while data is not None:
yield data
data = g.switch()

if __name__ == "__main__":
for data in iterxml("somexmldata"):
# do something with data


The greenlet here calls expat, but the 'yield' happens inside an expat
callback. To get this to work you would need to rewrite expat to use its
callback as a generator. Expat is coded in C, so you also need to find
some way to get it to save the state of the C functions when it has
yielded.

I think this example should answer Arnaud's question ("In fact it is not


clear to me at the moment what can be done (sensibly :) with the OP's
Fiber class that cannot be achieved with the run() function I

suggested.") Arnaud's run function cannot be used by anything which needs
to save C stack state.

> I am happy if I could use convenient coroutine features via standard
> or simple extension library. py.magic.greenlet may be what I'm
> looking for, but I wonder why this is named "magic" :-)

I think it is called "magic" because it does something which is at first
glance impossible: saving the C stack state as well as the Python stack.
Possibly it does something underhand to achieve an impressive effect.
I haven't looked at the implementation to see how it does it: creating
threads and then scheduling them cooperatively would be one way but I
expect it just switches the C stack around between different areas of
memory.

A long time ago I ported the BCPL coroutine library and there is one
function in the middle (the one which does the stack switch) that
definitely felt like magic.

One interesting question raised by all this is whether those of us using
Windows can usefully call Microsoft's Fiber functions through ctypes and
get the benefit of coroutines without having to load any additional C
extensions at all.

Michael Sparks

unread,
Dec 23, 2007, 10:47:32 AM12/23/07
to
Duncan Booth wrote:

> Unfortunately generators only save a single level of stack-frame, so they
> are not really a replacement for fibers/coroutines. The OP should perhaps
> look at Stackless Python or Greenlets. See

On the surface of things, the single level aspect *LOOKS* like a problem,
but in fact is actually really useful. The reason is because it encourages
a generator to be relatively simple and focused encouraging reuse.

cf the components listed here:
* http://kamaelia.sourceforge.net/Components

Also, because they're simple, you can link every component in that list
either directly with all the others or via a trivial filtering component.

Also the fact that you have to build your own scheduler means you can do the
equivalent of saying to the scheduler "Don't run me next, run this next".
This gives you pretty much the full power of co-routines but allows for
very heavy reusability and independence of testing.

An example application that makes moderate use of this "bounce back"
coroutines is a greylisting server you can find here:
* http://kamaelia.sourceforge.net/KamaeliaGrey

I'd (almost) argue that the single level aspect of generators is perhaps
their best feature.


Michael.
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/


Michael Sparks

unread,
Dec 23, 2007, 11:04:20 AM12/23/07
to
Hi,


> It just works, but using native Python threads for non-preemptive
> threading is not cost-effective. Python has generator instead but it
> seemed to be very restricted for general scripting. I wish I could
> write nested (generator) functions easily at least.
>
> Is there any plan of implementing real (lightweight) fiber in Python?

Please take a look at Kamaelia. Generators work extremely well, and are
extremely useful to work with:

* http://kamaelia.sourceforge.net/Home

(I need to update/improve the website...)

If you *must* have nested generators (it's rarely useful) you can do that,
but just need to have your "thing running the generators" understand
a "don't run me next, run this next" message. Our message is called
WaitComplete, and you can see it used here (in a non-trivial useful
system):

https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Apps/Kamaelia-Grey/App/greylisting.py

(I could point at trivial examples, but prefer something real)

The shortest overall description is here:
* http://kamaelia.sourceforge.net/t/TN-LightTechnicalIntroToKamaelia.pdf

Cookbook examples:
* http://kamaelia.sourceforge.net/Cookbook

List of dozens of reusable (all with each other) components:
* http://kamaelia.sourceforge.net/Components

How to build your own (non-optimised) core:
* http://kamaelia.sourceforge.net/MiniAxon/

Ruby (mini) version: (for comparison :-)
https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/trunk/Code/Ruby/miniaxon.rb

Experimental process based (as well as thread & generator based) version:

http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1196129474

(Going to move to process based components for static segmentation of an
application across process boundaries to give explicit multicore support.
May move to automatic distribution at some point, but static is an easy win
as the above example shows :-)

It's not as lightweight as stackless python's microthreads, but it's pretty
close. The scaling style is pretty similar as well - cf:

http://www.rhonabwy.com/wp/2007/11/13/generator-based-concurrent-programming-in-python/
I also blogged about that here:
http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1195688924

(Someone curious about comparing Kamaelia to stackless. Interesting to see
the same scaling curve, but unsurprisingly stackless wins :-) Kamaelia
however works with standard python from version 2.2.late onwards :-) )

Interestingly recently discovered that Kamaelia's been slowly reinventing a
system the UK's MOD have been using for around 30 years to make concurrent
systems easier to build.... (mascot) :-)

Merry Christmas :-)


Michael

Duncan Booth

unread,
Dec 23, 2007, 2:53:14 PM12/23/07
to
Michael Sparks <m...@cerenity.org> wrote:

> Duncan Booth wrote:
>
>> Unfortunately generators only save a single level of stack-frame, so
>> they are not really a replacement for fibers/coroutines. The OP
>> should perhaps look at Stackless Python or Greenlets. See
>
> On the surface of things, the single level aspect *LOOKS* like a
> problem, but in fact is actually really useful. The reason is because
> it encourages a generator to be relatively simple and focused
> encouraging reuse.
>

Ah, perhaps Python should similarly limit function call nesting to one
level so as to keep things simple and encourage reuse.

Michael Sparks

unread,
Dec 23, 2007, 7:31:05 PM12/23/07
to
Duncan Booth wrote:

> Michael Sparks <m...@cerenity.org> wrote:
>
>> Duncan Booth wrote:
>>
>>> Unfortunately generators only save a single level of stack-frame, so
>>> they are not really a replacement for fibers/coroutines. The OP
>>> should perhaps look at Stackless Python or Greenlets. See
>>

>> On the surface of things, the single level aspect LOOKS like a


>> problem, but in fact is actually really useful. The reason is because
>> it encourages a generator to be relatively simple and focused
>> encouraging reuse.
>>
>
> Ah, perhaps Python should similarly limit function call nesting to one
> level so as to keep things simple and encourage reuse.

Bit of a grumpy response? ... and a straw man argument ? Generators can be
used to build co-operative multitasking (eg WSGI is implicitly co-op
multitasking), whereas function calls are used to build sequential code. I
agree for sequential code generators being single level can be rather
limiting. However, the thread is about co-operative multitasking.

Decades of experience has shown that the best way to build concurrent
systems (MASCOT[1], Unix, MPI, Occam, Erlang, even VHDL & some mashups)
is to have small things loosely connected. Generators encourage making
little things which then encourages connecting them together. It seems,
to me, to help. (Which is all I said really)

[1] Only recently heard about MASCOT - best reference is here:
http://async.org.uk/Hugo.Simpson/
(MASCOT is over 30 years old... (fun thing of the week that I've been
pointed at for me :-) )

It's the same basic ethos that makes Unix pipelines useful, and is why
WSGI is useful. Simple transformational things which work best when they're
pipelined together. The fact that you can't trivially nest generators in a
way that feels nice[2] IMO naturally encourages making small things, which
end up being reusable.

[2] I don't personally feel for i in X(): yield i is particularly nice.

To be clear - I REALLY didn't like the fact that generators were single
layer when I first saw them - it seemed a huge limitation. (Indeed as huge
a limitation as only having single level function calls, or only single
layer of nesting for namespaces, etc)... I even asked a question [3] on
that point before choosing python to write Kamaelia in... The fact that
it's been a benefit rather than a problem rather surprised me.

[3]
http://groups.google.com/group/comp.lang.python/tree/browse_frm/thread/fcd2709952d23e34/8a0b9ba0e3beb108

That's rather different from the recursion limit (which is a limit on
function call depth) that really *has* been a pain at times. (Something
which tail recursion would really help with)

Incidentally, you may note that you helped me back then (which I'm thankful
for :-), so you can kinda view this as me reporting back "actually, it's
turned out to be helpful rather than a pain" :-) Took a while to figure
that out though ;)

Your mileage may vary :)

Incidentally, it was probably your response (message 7) in that thread
that was responsible for me deciding to see where things could go by
trying python :-)

Merry Christmas,


Michael. (hoping that's a suitably positive response :-)

"Martin v. Löwis"

unread,
Dec 23, 2007, 7:53:55 PM12/23/07
to Akihiro KAYAMA
> Is there any plan of implementing real (lightweight) fiber in Python?

I have no such plan, and I don't know of anybody else's plan, either.

Regards,
Martin

Duncan Booth

unread,
Dec 24, 2007, 6:24:02 AM12/24/07
to
Michael Sparks <m...@cerenity.org> wrote:

> To be clear - I REALLY didn't like the fact that generators were
> single layer when I first saw them - it seemed a huge limitation.
> (Indeed as huge a limitation as only having single level function
> calls, or only single layer of nesting for namespaces, etc)... I even
> asked a question [3] on that point before choosing python to write
> Kamaelia in... The fact that it's been a benefit rather than a problem
> rather surprised me.

Ok, I've now spent a little while browsing Kamaelia and it does look
impressive. I'll need a bit longer to look at it in more detail.

Let me try again:

There are problems for which generators saving only a single stack frame
are an excellent solution. e.g. a situation like yours where if I
understand it you may have thousands of separate generators all in
existence simultaneously.

There are also problems where full blown coroutines are appropriate. The
example I quoted earlier of turning a parser from one which generates a lot
of callbacks to one which 'yields' tokens is the usual example given.

There are also some situations where threads are appropriate: just not
nearly as many situations as some people think.

I'm happy with generators as they are: they're a great solution to a
problem that most of us didn't realise we had until the solution came
along. That doesn't mean that I wouldn't also like to have a separate
coroutine library, but I would use it to replace situations that might
otherwise have to use threads, not to replace generators.

> Incidentally, you may note that you helped me back then (which I'm
> thankful for :-), so you can kinda view this as me reporting back
> "actually, it's turned out to be helpful rather than a pain" :-) Took
> a while to figure that out though ;)
>
> Your mileage may vary :)
>
> Incidentally, it was probably your response (message 7) in that thread
> that was responsible for me deciding to see where things could go by
> trying python :-)
>

Thanks you. I really do intend *most* of my posts to be helpful.

Michael Sparks

unread,
Dec 24, 2007, 11:03:23 AM12/24/07
to
Duncan Booth wrote:
[ snip sections about why the single layer aspect can be helpful ]

> I'm happy with generators as they are: they're a great solution to a
> problem that most of us didn't realise we had until the solution came
> along. That doesn't mean that I wouldn't also like to have a separate
> coroutine library, but I would use it to replace situations that might
> otherwise have to use threads, not to replace generators.

Oh agreed. For that, it would probably be nice to see greenlets (or
something similar) make it into the standard library, but I suspect
that's too much of a jump at the moment.

I think for me, what we tend to do is prototype something as a stub
program, and either turn that into a threaded component or a generator
component. If I was to do something halfway I'd probably look at greenlets
at the moment, though so far we've not found a use for that. (which is
relatively suprising)

Any time it's looked useful, generally it's actually meant "spin this
control out to another smaller component". That's generally then resulted
in that spun out component being reusable.

For example, probable border case for coroutines is in this:
http://kamaelia.svn.sourceforge.net/viewvc/kamaelia/branches/private_MPS_Scratch/Apps/Kamaelia-Grey/App/greylisting.py?revision=3684&view=markup

We do do something that implies full coroutine support would be useful -
specifically the calls:
yield WaitComplete(self.getline(), tag="_getline1")
...
yield WaitComplete(self.getline(), tag="getline2")

To:
31 def getline(self):
32 control_message = ""
33 while 1:
34 while not self.anyReady():
35 self.pause();
36 yield 1
37 while self.dataReady("control"):
38 control_message = self.recv("control")
39 if isinstance(control_message, socketShutdown):
40 self.client_connected = False
41 if self.dataReady("inbox"):
42 self.logging_recv_connection()
43 return
44 else:
45 if not self.client_connected :
46 self.breakConnection = True
47 return
48 yield 1

(This is of course faking full co-routines by getting the caller to call
something else temporarily)

The thing is, if we split that out to an external component, there's
potential benefits.

*shrug* IMO, code should be written for clarity first, so simply being able
to have this as pure co-routines may be useful. However, being able to tag
the calls like this:
yield WaitComplete(self.getline(), tag="_getline1")
...
yield WaitComplete(self.getline(), tag="getline2")

Meant that I could run an introspector:
506 Pipeline(
507 PeriodicWakeup(),
508 WakeableIntrospector(),
509 ).activate()

And was able to see that some clients were misbehaving at different time
points, but also seeing clearly able to see what state the component was
waiting/hanging in when dealing with broken clients, and able to add in
timeouts trivally. cf:

*debug* 'THREADS'['Axon.Microprocess.microprocess_7068_getline1',
'Kamaelia.Chassis.Graphline.Graphline_7062',
'Kamaelia.Chassis.Pipeline.Pipeline_7',
... snip ...
'Kamaelia.Internet.ConnectedSocketAdapter.ConnectedSocketAdapter_7059',
'Kamaelia.Internet.Selector.Selector_11',
'Kamaelia.Internet.TimeOutCSA.ActivityMonitor_7060',
'Kamaelia.Internet.TimeOutCSA.ResettableSender_7061',
'__main__.GreylistServer_8',
'__main__.PeriodicWakeup_5',
'__main__.TCPS_10',
'__main__.WakeableIntrospector_6']

Now you could of course design a coroutine system for python to do that
and I'd hope if anyone did, they'd think of it being useful to trace the
where a co-routine is currently stuck :-)

All that said, my personal primary aim for kamaelia is to try and make it
into a general toolkit for making concurrency easy & natural (as well
as efficient) to work with. If full blown coroutines turn out to be part
of that c'est le vie :-)

I think it's a bit like the arguments around lambda being limited to a
single expresson vs a throwaway def. Both are useful, being stuck with one
or the other is more limiting than is perhaps ideal.

> Thanks you. I really do intend most of my posts to be helpful.

Likewise :-)

Have fun :-)


Michael.

Bjoern Schliessmann

unread,
Dec 24, 2007, 6:30:53 PM12/24/07
to
Michael Sparks wrote:

> All that said, my personal primary aim for kamaelia is to try and
> make it into a general toolkit for making concurrency easy &
> natural (as well as efficient) to work with. If full blown
> coroutines turn out to be part of that c'est le vie :-)

I must admit I mostly didn't follow this thread, but it sparked my
interest in Kamaelia. I already did the MiniAxon tutorial and I
plan to try out Kamaelia with a project I had discussed here a
while ago (which I had to suspend until recently because of not
enough spare time; it's about simulating complex relais circuits).
Please continue the great work.

Regards & merry christmas everyone,


Björn

P.S.: In the MiniAxon tutorial, I noticed a formatting problem and a
bunch of typos on one page but the feedback link doesn't work; are
you interested in a list?

--
BOFH excuse #14:

sounds like a Windows problem, try calling Microsoft support

icca...@gmail.com

unread,
Dec 28, 2007, 7:19:56 AM12/28/07
to
Hi!

Since your interest in fibers/coroutines is related to writing
simulators, you should try the SimPy package (simpy.sf.net), which is
a process-based discrete event simulator that uses generators as
processes.

Michael Sparks

unread,
Dec 28, 2007, 2:12:48 PM12/28/07
to
Bjoern Schliessmann wrote:

> Michael Sparks wrote:
>
>> All that said, my personal primary aim for kamaelia is to try and
>> make it into a general toolkit for making concurrency easy &
>> natural (as well as efficient) to work with. If full blown
>> coroutines turn out to be part of that c'est le vie :-)
>
> I must admit I mostly didn't follow this thread, but it sparked my
> interest in Kamaelia. I already did the MiniAxon tutorial and I
> plan to try out Kamaelia with a project I had discussed here a
> while ago (which I had to suspend until recently because of not
> enough spare time; it's about simulating complex relais circuits).
> Please continue the great work.

Many thanks for your kind words - the work is continuing :-)

Also, I'd be interested in hearing how your project gets on - it sounds
like the sort of thing that Kamaelia should be able to help with. (If it
doesn't/can't, then it's a bug IMO :)


> Regards & merry christmas everyone,
>
>
> Björn
>
> P.S.: In the MiniAxon tutorial, I noticed a formatting problem and a
> bunch of typos on one page but the feedback link doesn't work; are
> you interested in a list?

I'm always interested in feedback! The fact the feedback link doesn't work
for you is particularly useful - I'll look into that!

Best Regards,


Michael.

Michael Sparks

unread,
Dec 28, 2007, 2:49:52 PM12/28/07
to
Duncan Booth wrote:

> There are also problems where full blown coroutines are appropriate. The
> example I quoted earlier of turning a parser from one which generates a
> lot of callbacks to one which 'yields' tokens is the usual example given.

For completeness, I looked at the other half of the thread at the expat
parser, and decided to write that in the style we use for Kamaelia - it
ends up looking like this:

import xml.parsers.expat
import Axon
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Console import ConsoleEchoer

class Parser(Axon.ThreadedComponent.threadedcomponent):
data = "<h1> Default </h1>" # Can be overridden by kwargs as normal

def start_element(self, name, attrs):
self.send(("START", name, attrs), "outbox")

def end_element(self, name):
self.send(("END", name), "outbox")

def char_data(self, data):
data = data.strip()
self.send(("DATA", data), "outbox")

def main(self):
p = xml.parsers.expat.ParserCreate()
p.StartElementHandler = self.start_element
p.EndElementHandler = self.end_element
p.CharacterDataHandler = self.char_data
p.Parse(self.data, 1)
self.send(Axon.Ipc.producerFinished(), "signal")

Pipeline(
Parser(data="<body><h1>Hello</h1> world <p>Woo</p></body>"),
ConsoleEchoer(),
).run()

You'll note we don't use generators for "Parser" in this context. This also
isn't 100% identical to form you use since we don't turn this into an
iterator (obviously :).

We do also use 1 more thread than the greenlet approach though. Pipeline
& ConsoleEchoer are generator based though, as is the scheduler that
runs them :-)

Have fun :-)


Michael.

Bjoern Schliessmann

unread,
Dec 28, 2007, 4:51:37 PM12/28/07
to
Michael Sparks wrote:
> Also, I'd be interested in hearing how your project gets on - it
> sounds like the sort of thing that Kamaelia should be able to help
> with. (If it doesn't/can't, then it's a bug IMO :)

That was also my impression. :) My last tries had similarities to
Axon, but I never was able to formulate it this consistently (no
generators, no "microthreads", very very basic scheduler).

Quite recently, I felt a little lost in the documentation. The docs
shipped with Axon seem to be slightly out of date (e. g. "pydoc
Component" says I mustn't override Component.main). The newer docs
(<http://kamaelia.sourceforge.net/Docs/Axon/>) don't seem to be
shipped with it at all (they help much though ;) ).



> I'm always interested in feedback! The fact the feedback link
> doesn't work for you is particularly useful - I'll look into that!

It says

"""Authorisation DENIED !

Sorry, guest - you are not allowed to perform this action :
Do something undefined !
"""

Regards,


Björn

--
BOFH excuse #111:

The salesman drove over the CPU board.

0 new messages