def f1():
print "1-1"
print "1-2"
print "1-3"
def f2():
print "2-1"
print "2-2"
print "3-3"
and when my application runs, I want to execute these functions in
"lock-step", so that the output looks like:
1-1
2-2
1-2
2-2
1-3
2-3
If I were using prolog I'd write a meta-interpreter; with lisp/scheme I
might use macros and do some source-code transformation.
What's the most Pythonic way to do this?
If I assume that each statement in the function bodies occupy only one
physical line, I could read the user functions in, and output them to a new
file, interspersing "yield"s. If, quite reasonable, a user spans a statement
over more that one line:
def f3():
d = {
"a" : "One string",
"b" : "Another string"
}
print d
then I'm in the "parsing python" game.
(I recall the malevolent and wicked ternary operator example that was
posted here which fiddled with function source code in the metaclass... I'm
still picking up pieces of my brain from the floor)
I'd be very grateful for suggestions. Is the compiler module my friend?
I would love to end up with:
while 1:
try:
for f in function_list:
f.next()
except StopIteration:
print "wow!"
Thanks
Chris Wright
I think the problem is underspecified. What do you mean by the quoted
phrase "lock-step"? Your example includes only simple print statements,
which generate output to sys.stdout. What are you really trying to
accomplish? Do you want individual bytecodes to be executed one at a
time from a series of functions? Do you want any output to be interlaced
on a line-by-line basis? Any call to any I/O routine? Need more detail.
-Peter
OK. Thanks. I was just giving some simple examples...but I hope this is
better:
From the python grammar:
funcdef ::= "def" funcname "(" [parameter_list] ")" ":" suite
I want the statements in <suite> to be executed on a statement by statement
basis, with no particular restrictions on *what* those statements are. It's
the execution stepping I'm interested in... the I/O in my example was just
that. I understand that there will be all the problems on contention/race
etc (just like if the functions were going in separate threads)
> I want to execute these functions in
> "lock-step", so that the output looks like:
>
> 1-1
> 2-2
> 1-2
> 2-2
> 1-3
> 2-3
Have you made a mistake in this output? Should it be this?
1-1
2-1
1-2
2-2
1-3
2-3
If yes, how about this?
#-------------------------
import itertools
def seq(which, iterable):
for i in iterable:
yield "%d-%d" % (which, i)
times = [1,2,3]
seq1 = seq(1, times)
seq2 = seq(2, times)
for v1, v2 in itertools.izip(seq1, seq2):
print v1
print v2
#--------------------------
If you want to generalise the number of iterators, how about this?
#--------------------------
def lizip(iterables):
iterables = map(iter, iterables)
while True:
result = [i.next() for i in iterables]
yield tuple(result)
def seq(which, iterable):
for i in iterable:
yield "%d-%d" % (which, i)
y = [1,2,3] ; max_x = 5
iters = [seq(x, y) for x in range(1, max_x+1)]
for vertslice in lizip(iters):
for v in vertslice:
print v
#--------------------------
Note that these examples don't use "coroutines". Instead they use
python generators, which are classified as "semi-coroutines".
--
alan kennedy
-----------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/mailto/alan
He wants to have f1 and f2 alternate executing instructions:
# pseudo code
def f1():
instruction1-1
instruction1-2
def f2():
instruction2-1
instruction2-2
So, for the code above, he would like to see something like the following
execution routine:
start execution of f1
execute instruction1-1
pause execution of f1
start execution of f2
execute instruction2-1
pause execution of f2
resume execution of f1
execute instruction1-2
pause execution of f1
resume execution of f2
execute instruction2-2
pause execution of f2
Do a piece of one, then do a piece of the other, then do another piece of
the former, and so on...
While the following code does not provide a solution to the OP's request, it
does show something very much like the behaviour that's being asked for.
class LockStepper:
def __init__(self, id):
self.id = id
self.i = 1
self.partner = None
def next(self):
if self.i > 10:
return
print "instruction %s-%s"%(self.id, self.i)
self.i += 1
self.partner.next()
f1 = LockStepper(1)
f2 = LockStepper(2)
f1.partner = f2
f2.partner = f1
# begin lock-step execution
f1.next()
The idea here is to think of each LockStepper as though it were a function,
and to think of the print statement in each of their next() methods as a
line of instruction inside the body of that function. The LockStepper
executes the first line of it's "function body", then pauses. It's partner
executes the first line of it's function body, then pauses. The partner's
partner was the original LockStepper, so it "resumes execution"
(self.partner.next()) and executes it's second line of instructions. And so
on...
If someone better at generators than I could put together a version of this
where the LockStepper's have a __call__() method that yields execution back
and forth between a partner LockStepper, then we'd be closer to simulating
the behaviour, but not to solving the problem, which asks for this behaviour
using arbitrary functions (not necessarily LockSteppers). And, I've know
idea how to do that :)
Hope that clears things up a little,
Sean
Okay, that's clear. It can't be done. :-) Now on to the next step:
what are you actually trying to do. That is, *why* do you want the
behaviour you say you want? That might give us a few ideas about what
can really be done.
Basically, what you describe would require the interpretation of
Python at a higher level than it is interpreted now. It is actually
*compiled* code, turned from source (the lines you refer to) into
bytecodes, which are effectively machine code for the Python virtual
machine. It is at the level of bytecodes that one might actually
start to get the control you're talking about.
Another approach would be if you reduced the problem from arbitrary
statements to some kind of subset. Maybe evaluation of expressions
only, without loops or most statements? Of course, that might be
far too limited for you, but you haven't said why you want to do this
yet. If you could take that approach, you would basically retrieve
the source line by line and pass it to "exec" or maybe eval().
You could also just define your own mini-language, and write an
interpreter for that which has the behaviour you need.
Or, depending again on why you want all this, you could require
construction of the code using generators or something, as others
are suggesting.
-Peter
class LockStepper:
def __init__(self, id, nstmts):
self.id = id
self.stmt = 1
self.nstmts = nstmts
def next(self):
print "instruction %s-%s"%(self.id, self.stmt)
self.stmt += 1
def hasnext(self):
return self.stmt <= self.nstmts
class Switcher:
def __init__(self, *steppers):
self.steppers = steppers
def __call__(self):
while True:
for stepper in self.steppers:
if stepper.hasnext():
stepper.next()
if self._done():
break
def _done(self):
return not [f for f in (f1,f2) if f.hasnext()]
f1 = LockStepper(1, 4)
f2 = LockStepper(2, 7)
switch = Switcher(f1,f2)
switch()
# that should have been
def _done(self):
return not [f for f in self.steppers if f.hasnext()]
> OK. Thanks. I was just giving some simple examples...but I hope this
> is better:
>
> From the python grammar:
>
> funcdef ::= "def" funcname "(" [parameter_list] ")" ":" suite
>
> I want the statements in <suite> to be executed on a statement by
> statement basis, with no particular restrictions on *what* those
> statements are. It's the execution stepping I'm interested in... the
> I/O in my example was just that. I understand that there will be all
> the problems on contention/race etc (just like if the functions were
> going in separate threads)
The only way I can think to implement this 'easily' would be to use the
parser module to parse your source code, then walk the parse tree looking
for statements inside one of your functions and add a yield_stmt after
every other statement. Then compile the tree into bytecode and execute it.
Of course your users may be surprised to find that they cannot directly
call any of their functions from another function. You could do a further
rewrite on the tree to convert obvious calls to your users' functions into
for loops.
Why?
--
Duncan Booth dun...@rcp.co.uk
int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3"
"\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure?
Tricky - without jumping in before the code is compiled by python or
similar then I can't think of any way to do this. I can however think of a
perlesque way to do it. (ie should work, might not be considered nice)
One possible way out is if you're willing to change the way you write &
run code /slightly/.
Rather than write:
def func(arg):
arg.foo()
arg.bla()
while arg.foobar():
print arg.state()
arg.bla()
If you write this in your source files:
def func(arg):
arg.foo();
arg.bla();
while arg.foobar():
print arg.state();
arg.bla();
Before you run these files you could pre-process them to change all
occurances of ";" to "; yield <num>". Clearly you could do this
pre-processing in python and then exec the code. ie your preprocessor
would change the above code fragment to:
def func(arg):
arg.foo(); yield 1
arg.bla(); yield 2
while arg.foobar():
print arg.state(); yield 3
arg.bla(); yield 4
Then to run your functions in lockstep, you simply:
gens = [ x() for x in [func1, func2, func3, func4 ]
while gens:
newgens = []
foreach gen in gens:
try:
gen.next()
newgens.append(gen)
except StopIteration:
pass
gens = newgens
(Working on the assumption the generators for different functions will run
for differing periods of time)
About the only way I can see this working "simply" is using this sort of
pre-processing trick. I doubt it'd be considered Pythonic. (Personally I
view as more like a perlesque trick) It does however leave your code
looking single threaded (in fact I suppose it still is if you don't
preprocess it), and doesn't require any odd code structuring. (It does
require care if you use semi-colons inside strings/etc)
HTH,
Michael.
There is a 'Flow' module in Twisted which uses generators to
accomplish this sort of thing. Basically, Twisted has a
queue of functions that are called, and by yielding "cooperate"
object, you re-schedule the given generator for execution
on this queue (allowing other code blocks to execute).
from __future__ import generators
from twisted.flow import flow
from twisted.internet import reactor
coop = flow.Cooperate()
def f1():
print "1-1"
yield coop
print "1-2"
yield coop
print "1-3"
def f2():
print "2-1"
yield coop
print "2-2"
yield coop
print "2-3"
d = flow.Deferred(flow.Merge(f1(),f2()))
d.addCallback(lambda _: reactor.stop())
reactor.run()
I'm working on converting many 'data sources' such as PostreSQL
and OpenLDAP so that their async interface works this way.
See http://www.twistedmatrix.com/documents/howto/flow.html
for more information on the flow module.
Best,
Clark
P.S. If you wished, you could probably use flow without Twisted.
You'd have to do three things: (a) create an event queue object, such
as twisted.internet.reactor, (b) provide a notification when something
scheduled is done, aka twisted.internet.defer, and (c) provide some
way to handle exceptions in an async manner, aka twisted.python.fail.
Although, once you have all of these things in place... why not just
use the rest of Twisted... *grin* In other words, it is easy in
theory to implement this correctly, but not that easy if you want
to have it work with sockets and properly handle exceptions...
So, by the time you do all of this, you will have re-invented the
core of Twisted... and then you won't have a built-in http server
nor a community to support you.
> From the python grammar:
>
> funcdef ::= "def" funcname "(" [parameter_list] ")" ":" suite
>
> I want the statements in <suite> to be executed on a statement by
> statement basis, with no particular restrictions on *what* those
> statements are. It's the execution stepping I'm interested in...
Might your requirements be better met by a debugger, which is feeding
step instructions to a normal python program? This could be done
extending the Pdb debugger class in module Lib/pdb.py to implement
your own single step method. This could control various executing
suites of code, according to your chosen scheduling algorithm.
Or might you be better off to actually spawn separate threads, and
have each thread carry out each instruction, line by line, and then
enter a blocking wait for either a "continue" token from a
Queue.Queue(), or "Continue" message signalled on a
threading.Condition()?
> the I/O in my example was just that.
Or is that you want to control the states of your objects, depending
on the readiness status of various I/O channels? And you want to run
everything in one thread? In which case you might be looking for
ultra-lightweight threads, based on generators, combined with an
asynchronous I/O detection and reaction platform, such as asyncore,
twisted or medusa.
> I understand that there will
> be all the problems on contention/race etc (just like if the
> functions were going in separate threads)
If the objects which are being distributed out to your
(ultra-lightweight) threads are local to the functions/objects (i.e.
not shared with other functions/objects) that process them, then there
is no contention, and thus no locking required.
How many separate "threads" of execution are you seeking to create?
Check out this article:
http://www-106.ibm.com/developerworks/opensource/library/l-pythrd.html?dwzone=opensource
on weightless threads, a simpler version of "Flow"
Now, my question is, could an arbitrary function's bytecode be liberally
sprinkled with "yeilds" at run-time to achieve this effect without
having to alter the source code?
Oops! ;-)
To make ammend, I send you this solution:
def f1():
for i in range(1,4):
print "1-%s" % i; yield i
def f2():
for i in f1():
print "2-%s" % i
f2()
This print
1-1
2-1
1-2
2-2
1-3
2-3
However I am not sure I understand
correctly what you have in mind, so
this solution may well be not relevant
at all!
Michele
I don't have an answer, but I have the START of an approach. Michael Sparks
is ahead of me here (see his excellent answer
http://mail.python.org/pipermail/python-list/2003-July/173694.html), but
for a different approach try this:
G:\>python
ActivePython 2.2.1 Build 222 (ActiveState Corp.) based on
Python 2.2.1 (#34, Apr 15 2002, 09:51:39) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> def f():
... print '1-1'
... print '1-2'
... print '1-3'
...
>>> import dis
>>> dis.dis(f.func_code)
0 SET_LINENO 1
3 SET_LINENO 2
6 LOAD_CONST 1 ('1-1')
9 PRINT_ITEM
10 PRINT_NEWLINE
11 SET_LINENO 3
14 LOAD_CONST 2 ('1-2')
17 PRINT_ITEM
18 PRINT_NEWLINE
19 SET_LINENO 4
22 LOAD_CONST 3 ('1-3')
25 PRINT_ITEM
26 PRINT_NEWLINE
27 LOAD_CONST 0 (None)
30 RETURN_VALUE
Here we can SEE the code, and it's nicely broken into lines by
the SET_LINENO bytecode. What I'd LIKE to do now is to somehow
go from f.func_code (a code object) to a Python list of bytecodes.
Then we could easily split the list on SET_LINENO codes, intersperse
with calls to other functions, and sew it all up into a larger
list. Then we would just need to exec those bytecodes, and BINGO...
problem solved.
But I don't know how to convert a code object into a list of
bytecodes, nor how to convert it back to a code object so I can
pass it to exec.
Any better gurus able to enlighten me?
-- Michael Chermside
import inspect
def f1():
print "1-1"
print "1-2"
print "1-3"
def f2():
print "2-1"
print "2-2"
print "3-3"
def merge(f1,f2):
s1=inspect.getsource(f1).splitlines()[1:]
s2=inspect.getsource(f2).splitlines()[1:]
lines=['def mergedfunction():\n']
for line1,line2 in zip(s1,s2):
lines.append(line1+'\n')
lines.append(line2+'\n')
dic={}; exec ''.join(lines) in dic
return dic['mergedfunction']
f=merge(f1,f2)
f()
Output:
1-1
2-1
1-2
2-2
1-3
3-3
Exercise left for the reader: show the pittfalls of this approach.
Michele
> I have an application in which I want the users to be able to create
> python functions:
>
> def f1():
> print "1-1"
> print "1-2"
> print "1-3"
>
> def f2():
> print "2-1"
> print "2-2"
> print "3-3"
>
> and when my application runs, I want to execute these functions in
> "lock-step", so that the output looks like:
>
> 1-1
> 2-2
> 1-2
> 2-2
> 1-3
> 2-3
Does this do what you want?
Given an input string:
SOURCE = """\
def f1():
print "1-1"
for ch in ('a', 'b', 'c'):
print "1-2",ch
print "1-3"
def f2():
print "2-1"
print "2-2"
print "2-3"
"""
it produces:
1-1
2-1
1-2 a
2-2
1-2 b
2-3
1-2 c
1-3
N.B. It executes all top-level functions in the input, interleaving statements.
If a function calls another function, the called function will be executed
at full speed. A function could call another function 'line at a time' by
wrapping it's generator in for loop, but it would have to be explicit.
----- begin steplock.py -----
# Execute functions in steplock
import symbol, token
from symbol import *
from token import *
import parser
import compiler
import types
class Atom:
def __init__(self, value):
self.value = value
def __repr__(self):
return str(self.value)
def pretty(astList):
res = []
el = astList[0]
if el in symbol.sym_name:
res.append(Atom(symbol.sym_name[el]))
elif el in token.tok_name:
res.append(Atom(token.tok_name[el]))
else:
res.append(el)
for el in astList[1:]:
if isinstance(el, (list, tuple)):
res.append(pretty(el))
else:
res.append(el)
return type(astList)(res)
def printAst(ast, depth=2):
if not isinstance(ast, (list, tuple)):
ast = ast.tolist()
import pprint
pp = pprint.PrettyPrinter(depth=depth)
pp.pprint(pretty(ast))
class TreeWalker:
def __init__(self):
pass
def dovisit(self, node):
t = node[0]
nodetype = "default"
if t in symbol.sym_name:
nodetype = sym_name[t]
elif t in token.tok_name:
nodetype = tok_name[t]
fname = "visit_" + nodetype
if hasattr(self, fname):
return getattr(self, fname)(node)
return self.visit_default(node)
def visit_default(self, node):
# Visit an otherwise boring node.
res = [ node[0] ]
for child in node[1:]:
if isinstance(child, (list, tuple)):
res.append(self.dovisit(child))
else:
res.append(child)
return tuple(res)
class TopWalker(TreeWalker):
def __init__(self):
self.generators = []
def visit_file_input(self, node):
newnodes = self.visit_default(node)
assert newnodes[-1][0] == 0 and newnodes[-2][0] == 4
newnodes = newnodes[:-2] + tuple(self.generators) + newnodes[-2:]
return newnodes
#def visit_stmt(self, node):
# printAst(node, depth=255)
# return self.visit_default(node)
def visit_funcdef(self, node):
self.generators.extend(MakeGenerator(node))
return node
YIELD_STMT = (stmt, (simple_stmt, (small_stmt, (flow_stmt, (yield_stmt, (NAME, 'yield'),
(testlist, (test, (and_test, (not_test, (comparison,
(expr, (xor_expr, (and_expr, (shift_expr, (arith_expr, (term, (factor,
(power, (atom, (NAME, 'None')))))))))))))))))), (NEWLINE, '')
))
def MakeGenerator(funcdefnode):
funcname = funcdefnode[2][1]
newfuncname = funcname + '_generator'
suite = funcdefnode[-1]
walker = FunctionWalker()
newsuite = walker.dovisit(suite)
NewFunc = (funcdef, (NAME, 'def'), (NAME, newfuncname)) + funcdefnode[3:-1] + (newsuite,)
NewFunc_stmt = (stmt, (compound_stmt, NewFunc))
#return (NewFunc_stmt, )
ASSIGN_stmt = (stmt, (simple_stmt, (small_stmt, (expr_stmt, (testlist,
(symbol.test, (and_test, (not_test, (comparison, (expr, (xor_expr,
(and_expr, (shift_expr, (arith_expr,(term, (factor, (power,
(atom, (NAME, funcname)),
(trailer, (DOT, '.'), (NAME, 'generator'))))))))))))))),
(EQUAL, '='),
(testlist, (symbol.test, (and_test, (not_test, (comparison, (expr, (xor_expr,
(and_expr, (shift_expr, (arith_expr,(term,
(factor, (power, (atom, (NAME, newfuncname))))))))))))))))),
(NEWLINE, '')))
DEL_stmt = (stmt, (simple_stmt, (small_stmt,
(del_stmt, (NAME, 'del'),
(exprlist, (expr, (xor_expr, (and_expr, (shift_expr,
(arith_expr, (term, (factor, (power, (atom, (NAME, newfuncname))))))))))))),
(NEWLINE, '')))
return (NewFunc_stmt, ASSIGN_stmt, DEL_stmt)
class FunctionWalker(TreeWalker):
def visit_funcdef(self, node):
return node # Ignore nested functions!
def visit_suite(self, node):
#print "Visiting suite of %d children" % (len(node)-1)
res = [ node[0] ]
for el in node[1:]:
if el[0] == stmt:
if el[1][0] == compound_stmt:
res.append(self.dovisit(el))
else:
res.append(el)
res.append(YIELD_STMT)
else:
res.append(el)
return tuple(res)
def test():
SOURCE = """\
def f1():
print "1-1"
for ch in ('a', 'b', 'c'):
print "1-2",ch
print "1-3"
def f2():
print "2-1"
print "2-2"
print "2-3"
"""
ast = parser.suite(SOURCE)
astList = ast.totuple(1)
walker = TopWalker()
newList = walker.dovisit(astList)
code = parser.sequence2ast(newList)
namespace = { '__builtins__': __builtins__ }
exec code.compile('dummy.py') in namespace
functions = [ namespace[key].generator()
for key in namespace
if isinstance(namespace[key], types.FunctionType) ]
while functions:
for f in functions[:]:
try:
f.next()
except StopIteration:
functions.remove(f)
if __name__=='__main__':
test()
----- end steplock.py -----
><thewr...@ozemail.com.au> wrote in
>news:g7xRa.305$fF....@nnrp1.ozemail.com.au:
>
>> I have an application in which I want the users to be able to create
>> python functions:
>>
>> def f1():
>> print "1-1"
>> print "1-2"
>> print "1-3"
>>
>> def f2():
>> print "2-1"
>> print "2-2"
>> print "3-3"
>>
>> and when my application runs, I want to execute these functions in
>> "lock-step", so that the output looks like:
>>
[...]
>
[...]
>----- begin steplock.py -----
># Execute functions in steplock
[...]
>if __name__=='__main__':
> test()
>----- end steplock.py -----
>
>--
>Duncan Booth dun...@rcp.co.uk
>int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3"
>"\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure?
OP:
From: <thewr...@ozemail.com.au>
Newsgroups: comp.lang.python
Subject: Co-routines
Lines: 63
Date: Thu, 17 Jul 2003 23:07:36 +1000
SOLUTION:
From: Duncan Booth <dun...@NOSPAMrcp.co.uk>
Newsgroups: comp.lang.python
Subject: Re: Co-routines
Date: Fri, 18 Jul 2003 14:57:35 +0000 (UTC)
Organization: RCP Consultants
Lines: 248
Ok, I'm impressed ;-)
Regards,
Bengt Richter