I realized WebPipes is kind of similar to Storm's model, without the distributedness.
The core abstraction in Storm is the "stream". A stream is an unbounded sequence of tuples. Storm provides the primitives for transforming a stream into a new stream in a distributed and reliable way. For example, you may transform a stream of tweets into a stream of trending topics.
The basic primitives Storm provides for doing stream transformations are "spouts" and "bolts". Spouts and bolts have interfaces that you implement to run your application-specific logic.
A spout is a source of streams. For example, a spout may read tuples off of a Kestrel queue and emit them as a stream. Or a spout may connect to the Twitter API and emit a stream of tweets.
A bolt consumes any number of input streams, does some processing, and possibly emits new streams. Complex stream transformations, like computing a stream of trending topics from a stream of tweets, require multiple steps and thus multiple bolts. Bolts can do anything from run functions, filter tuples, do streaming aggregations, do streaming joins, talk to databases, and more.
https://github.com/nathanmarz/storm/wiki/Tutorial
A spout is kind of like our triggers, and a bolt is like a block.
The major differences are:
1. Storm is "distributed". You specify how many instances of each spout/bolt you want, and how inputs to those instances should be grouped. This lets Storm to spread computation across multiple servers like in MapReduce.
2. Instead of piping around individual values, you pipe tuples. Each bolt inputs a single tuple (which has multiple fields). This simplifies the dependency resolution of a pipeline executor since there's only one input (with multiple fields). However the bolts' input/output fields must match, so it's harder to have a collection of generic bolts, which is really what we want in WebPipes.
Anyway, it would be cool to build Storm topologies with the pipes editor, but maybe that's a separate project.
Another related thought I had was the node-webpipes API isn't HTTP specific at all. Maybe the library could expose other transports, like TCP sockets, WebSockets, ZeroMQ, even Storm bolts. Then it's no longer WebPipes, but more like JSONPipes or something.
I'm not proposing anything in particular, just wanted to throw these ideas out there (but it's 4am so they may not make any sense)
-tom