Computational pipelines for python

[Tim Lahey]

I ran across this package,

http://code.google.com/p/ruffus/

That provides for support for computational
pipelines in Python. It has some nice features
for support of a task pipeline and visualization
of the pipeline using Graphviz.

Cheers,

Tim.

---
Tim Lahey
PhD Candidate, Systems Design Engineering
University of Waterloo
http://www.linkedin.com/in/timlahey

[Pat LeSmithe]

Tim Lahey wrote:
> I ran across this package,
>
> http://code.google.com/p/ruffus/
>
> That provides for support for computational
> pipelines in Python. It has some nice features
> for support of a task pipeline and visualization
> of the pipeline using Graphviz.

On a somewhat related, highly speculative note: It should be possible
to convert the worksheet into a free-form workspace. The classic
one-column setup works well, but different arrangements could be useful
in some situations.

The browser's cell list is now an array of IDs, but we can also keep
track of cell coordinates (2D or 3D), orientations, input-output
connections (see below), types, etc. Of course, we should save this
data on the server, too, in order to restore pieces to their places.

A click on the workspace or a toolbar might create a new stem cell [1].
Another click (or a paste event) could convert it to an ordinary input
cell, a text cell, an interactive cell, or something completely different.

In the case of "interacts" [2], each such cell might have a set of
inputs and outputs. These could be scalars or more intricate Sage
objects, images, sounds, etc. Nevertheless, a cell's widgets, including
ordinary functions, should be able to read / write the variables in a
standard way.

Efficiently making, maintaining, and displaying the graph of cell
connections probably requires some thought and a lot of work. It'd be
great to have both plain text and drag-and-drop options [3].

Ideally, we'd also be able to specify the conditions under which a
given cell fires, e.g., when any input is updated, after all inputs are
updated, or manually. The preparser, special cell variables (e.g.,
_in_X, _out_Y), notebook's callback system, and/or jQuery [4] may
help here.

Not that I'm volunteering, so feel free to pile it on... Thoughts?

[1] http://groups.google.com/group/sage-devel/msg/1f5658399f500c24

[2]
http://groups.google.com/group/sage-devel/browse_thread/thread/0b91cbc63479555b/f3aa93231beedfa1?#f3aa93231beedfa1

[3] A desktop example is the signals-and-slots mode in Qt Designer
(/usr/bin/designer-qt4 on Fedora Linux):

http://doc.qtsoftware.com/4.5/designer-connection-mode.html

I think VisTrails

http://www.vistrails.org/index.php/Main_Page

also works in this way.

[4] jQuery, or one of it's plug-ins, can handle certain live events:

http://docs.jquery.com/Events/live
http://plugins.jquery.com/project/Listen
http://plugins.jquery.com/project/Intercept
http://plugins.jquery.com/project/livequery

[William Stein]

(1) Too bad you're not volunteering, since judging by what I've seen
you do so far, you're *definitely* capable of pulling something off
like you describe above. I could even imagine you doing it in a few
weeks time.

(2) It is fun to use jquery to make input/out cell blocks
drag/movable. This is really quite trivial with jquery, and it allows
you to do a sort of demo/mockup of a tiny bit of what you're
describing above.

(3) What you're describing is pretty cool because it reminds me more
of Mathcad or Adobe Illustrator, which are two programs I've used a
lot before. When I was an undergrad in 1991-1993 I had a copy of
Mathcad which I frequently used (I never liked Mathematica then). I
liked how one could lay out mathematics in a 2d array, just like an
illustrator document (say).

Anyway, I definitely don't see the current Mathematica/Google
Notebook-style design of the notebook as at all the be-all end-all of
the notebook. It was mainly a quick very usable design that we could
make stable.

I'm funded to work fulltime on the notebook and symbolic manipulation
stuff from mid-September through mid-December (yeah Univ of
Washington!), and need all the ideas people can suggest to keep me
busy. :-)

-- William

> [1] http://groups.google.com/group/sage-devel/msg/1f5658399f500c24
>
> [2]
> http://groups.google.com/group/sage-devel/browse_thread/thread/0b91cbc63479555b/f3aa93231beedfa1?#f3aa93231beedfa1
>
> [3] A desktop example is the signals-and-slots mode in Qt Designer
> (/usr/bin/designer-qt4 on Fedora Linux):
>
> http://doc.qtsoftware.com/4.5/designer-connection-mode.html
>
> I think VisTrails
>
> http://www.vistrails.org/index.php/Main_Page
>
> also works in this way.
>
> [4] jQuery, or one of it's plug-ins, can handle certain live events:
>
> http://docs.jquery.com/Events/live
> http://plugins.jquery.com/project/Listen
> http://plugins.jquery.com/project/Intercept
> http://plugins.jquery.com/project/livequery
>
>
> >
>

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[Kevin Horton]

On 1 Jul 2009, at 16:52, William Stein wrote:

> On Wed, Jul 1, 2009 at 5:38 PM, Pat LeSmithe<qed...@gmail.com> wrote:
>>
>> On a somewhat related, highly speculative note: It should be
>> possible
>> to convert the worksheet into a free-form workspace. The classic
>> one-column setup works well, but different arrangements could be
>> useful
>> in some situations.

Great ideas.

> I'm funded to work fulltime on the notebook and symbolic manipulation
> stuff from mid-September through mid-December (yeah Univ of
> Washington!), and need all the ideas people can suggest to keep me
> busy. :-)
>

I'd love to have interacts that work on public worksheets.

--
Kevin Horton
Ottawa, Canada