Generator Local
Erik Groeneveld
(I suggest switching to the cogen mailing list, ok?)
I noticed the (struggle with?) kind of thread-local variables for
generators in Cogen. Yet another problem in common. We implemented
this a few times, and after lots of debate settled for a lookup of a
local variable on the callstack. If a coroutine defines a variable
for example:
def coroutine(self, ...):
__local__
othercode()
then in othercode(), you can find the __local__ variable by inspecting
the call stack. The debate was about how dirty this was. In the end
we concluded that putting a 'generator-local' on the call stack like
this was the proper thing to do because the whole purpose of using
generators/coroutines is to replace expensive thread state with the
lightweight state of a call stack. The VM makes sure that each
generators state is preserved on the callstack by creating a stack
frame object for it. The call stack == coroutine state. Hence
'coroutine-locals' belong on this call stack as well. We think...
Because traversing the call stack is extremely expensive (the VM will
dynamically create dictionary abstractions containing different kinds
of locals every time you use frame.f_locals) we have created a
C-extension that does it way faster. Weightless now uses the fast one
when available and falls back to the slow python version for
traversing the call stack.
It would be nice if the VM had hooks for such things wouldn't it?
What is your opinion?
Best regards,
Erik
E.J. Groeneveld
Seek You Too
twitter, skype: ejgroene
mobiel: 0624 584 029
Ionel Maries Cristian
wrote code that pass state around if needed.
I don't have a "context" that decides what happens with the
input and output of a coroutine - it's just the scheduler and magic objects
that get in and out of the coroutine - and they know how to take care of
themselves.
You could call them request/response objects, asynchronous completion
tokens, operations etc. They hold the logic that starts asynchronous
requests and passes back results in the coroutine. They also take care
of the state.
Then again, if you wrap generators you could easily slap some state/locals
on the wrapper.
Btw, is that C-extension public?
I also noted that weightless is missing a setup script now?
-- ionel
Andrew McNabb
>
> Because traversing the call stack is extremely expensive (the VM will
> dynamically create dictionary abstractions containing different kinds
> of locals every time you use frame.f_locals) we have created a
> C-extension that does it way faster. Weightless now uses the fast one
> when available and falls back to the slow python version for
> traversing the call stack.
I would be a bit disappointed if it were required to have a C extension
to run. There's a lot of value in being able to distribute pure Python
code. However, I'm not opposed to having a slower pure-Python
implementation and a faster C-extension implementation that is used if
available.
My main concern is that it's hard to read code that has magic frame
stuff going on under the hood. As a user, I would like to avoid that
sort of approach if possible.
Forgive me if I'm revealing my lack of experience with pipelines, but it
occurs to me that an alternative approach might be a class for local
state, like so:
class MyGenerator(BaseGenerator):
def __call__(self, *args):
self.mystate = 'something'
yield 'stuff'
In this case, any externally visible state could be saved as attributes
of the class.
Am I understanding the problem or am I completely missing it?
> It would be nice if the VM had hooks for such things wouldn't it?
Would yield-from (mentioned in another thread) help address this problem
with pipelines?
--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
Erik Groeneveld
>
> On Mon, Apr 13, 2009 at 01:22:08PM +0200, Erik Groeneveld wrote:
>>
>> Because traversing the call stack is extremely expensive (the VM will
>> dynamically create dictionary abstractions containing different kinds
>> of locals every time you use frame.f_locals) we have created a
>> C-extension that does it way faster. Weightless now uses the fast one
>> when available and falls back to the slow python version for
>> traversing the call stack.
>
> I would be a bit disappointed if it were required to have a C extension
> to run. There's a lot of value in being able to distribute pure Python
> code. However, I'm not opposed to having a slower pure-Python
> implementation and a faster C-extension implementation that is used if
> available.
is VM'ish and may easily hamper performance when written in Python.
Some stuff is even not possible in Python. like adjusting the
tracebacks after an Exception.
> My main concern is that it's hard to read code that has magic frame
> stuff going on under the hood. As a user, I would like to avoid that
> sort of approach if possible.
comprehensible. To accomplish that however, might involve not so
straight forward code, as any VM has, I am afraid.
> Forgive me if I'm revealing my lack of experience with pipelines, but it
> occurs to me that an alternative approach might be a class for local
> state, like so:
>
> class MyGenerator(BaseGenerator):
> def __call__(self, *args):
> self.mystate = 'something'
> yield 'stuff'
>
> In this case, any externally visible state could be saved as attributes
> of the class.
>
> Am I understanding the problem or am I completely missing it?
But you are not missing the point, I am. I have been confusing thread
locals with context information. Context info is a completely
different programming challenge: were do you store context information
like security, transaction, thread information? And how to access it?
This is what I try to do with my 'local', but that 'local' is actually
a 'context' object. But since it is stored in a generator local I
confused a bit. Sorry.
But both problems need to be solved:
- generator locals
- context info
Erik