pipes.py and its limits
val
i'm looking at pipes.py, a great pipeline implementation
in python (BTW, who is the author?), and trying to figure
out its limits. The description says that the commands
to be appended to the pipeline template have to be
shell functions. I'd like to try to build the dynamic pipelines
using the python functions or callable objects.
In other words, i'd like to modify the pipeline -
its functional components on-the-fly depending on
the results of running the pipeline (its output).
I'm sure the real Pythoneers know how to do that.
Any pointers/help would be highly appreciated.
thanx,
val
Oren Tirosh
Take a look at http://tothink.com/python/dataflow for a library that
implements pipelines (I call them flows) using Python iterators and
generators.
I also have a library for intergrating external commands more easily
into Python scripts. http://tothink.com/python/shell
I'm working on merging the two so it will be possible to create flows
with mixed Python functions and external commands.
I do not intend to support modification of the flow while it is running
because its elements may be delicate stateful components that have
different input and output rates, buffering, etc. Changing it on the
fly could lose data. But one of the elements in the flow can change its
functionality or even embed a separate flow object inside it if you
want.
Oren
val
Oren you are great.
First, coz you are here to help when one needs it.
Thanx. (This is the Fantastic List,
nothing like that in the whole world)
I spent a couple of weeks torturing the poor
google trying to locate your 'flows'
(i had a fuzzy idea that they exist, i tried
pipeline, meta, sequence, etc but never flows..)
Anyway, i'm going to follow your pointers and
study all that material.
thank you very much,
val