Two kinds of pipelines?

[Jeff Lindsay]

Pipelines are currently only considered composite blocks. By that definition, they have inputs and outputs. 

However, think of the outer most pipeline for a simple IFTTT style pipeline ... there are no inputs and no outputs. Suddenly a pipeline, in certain situations, is the only block that has no inputs or outputs. 

But it doesn't stop there. 

Composite pipelines have a name because as a block they're going to be reused. However, the outer pipeline I end up designing in the pipeline editor isn't made to be reusable. In fact, it might even have a UUID and have my email associated with it. The UUID is to identify the "instance" of this particular pipeline, and the email is helpful when the pipeline executer runs into problems. Neither of those things would be part of a composite block pipeline.

So it seems like maybe there are two kinds of pipelines. One is a block, one isn't. One is reusable, one is a singleton. Suddenly the model becomes more confusing...

--
Jeff Lindsay
http://progrium.com

[Jeff Lindsay]

Although conceptually there might be two kinds of pipelines, we might still get away with one pipeline format without overloading it too much.

In the singleton instance, the UUID can be the name of the pipeline. This convention would make it pretty clear when one is a singleton vs a reusable block ... along with not having inputs/outputs. 

Both can have an email address. The email for a block pipeline is the maintainer. Perhaps all blocks should have emails. This can be used, assuming the definition is accessible, by the executer to let the maintainer know if anything is wrong (slow response times), but also in whatever public repositories for people to know who the maintainer is. 

Then of course, singleton pipelines would have an email address to email the owner if the pipeline failed to run for some reason.

[Tom Robinson]

I'm not sure I see the need for a distinction.

- pipeline is a description of one or more blocks

- block pipeline has inputs and outputs

- trigger pipeline has no inputs

- action pipeline has no outputs

- "singleton" pipeline has no inputs or outputs

In the pipeline editor you could add a singleton block, but you'd have no way to connect it to anything.

Actually, now that I think about it, that might be a useful thing to be able to do. Say you design an application consisting of multiple pipelines, it would be useful to collect all of those pipelines together, and pass it to an executor to register every trigger.

Maybe a pipeline could also have some "environment variables" that could contain configuration data shared by all sub-pipelines (API keys, etc).

(p.s. sorry if you get this twice, I hadn't registered for the Google Group)

[Jeff Lindsay]

So let me give some concrete examples of each type of pipeline. 

First, the reusable block pipeline (1):

1) I'm creating a new kind of action block as a pipeline because I'm not a developer. It's going to tweet it's "text" input in leetspeak. It's made of two blocks, a leetspeak block and a tweet action. The leetspeak input is connected to the input of the pipeline. The output of the leetspeak block is wired to the tweet action. I name my pipeline "leettweet" and share the URL with others to use a block in their pipelines. 

Now, the "singleton" or "outer" pipeline (2):

2a) In this pipeline I want to make an email address that I can email and it will tweet the subject in leetspeak. It uses an email receiver trigger and the leettweet pipeline/block we made before. This pipeline is just for me. It doesn't have any inputs or outputs.

2b) I'm using an IFTTT style editor to make another simple pipeline. It has a cron trigger and an email action. Neither of these blocks are composite blocks. I attach the (made up) "time" output of the cron trigger to the "subject" input of the email action. I set the "to" input of the email action to a single value: my email address. Like the last one it's a "container" pipeline. Also, it's not reusable because it's made just for me. 

[Tom Robinson]

I understand the difference, I just don't think we really need different formats. In both cases you're describing the connections between blocks.

[Jeff Lindsay]

I understand the difference, I just don't think we really need different formats. In both cases you're describing the connections between blocks.

Except we could easily say a pipeline definition is an extended block definition. But it's not necessarily. Maybe it can be. I'm more concerned about having consistent language to talk about the difference. Notice all the different ways I talk about it in the examples.

-jeff

 

--