Retrying a generator pipeline from a child stage?

[Mark Larimer]

Say a parent generator pipeline spawns a child stage. If the child
stage discovers that something has gone wrong, what's the recommended
way to have the child tell the parent to abort and retry from the
start? I see "raise common.Abort()", but that doesn't look like it
will retry the parent. Can I catch the Abort in the parent like I
would a regular exception and start the parent generator pipeline over
somehow?

Along similar lines...if pipeline A spawns stage B, and pipeline B
spawns stage C, is there any way for C to tell B to abort and retry
(but not send the abort/retry all the way to up to A)?

Thanks.
--mark

[Brett Slatkin]

Hey Mark,

All there is right now along these lines is raise pipeline.Retry, but
that will only cause the current child to retry. Forcing the parent to
retry also is interesting. The problematic thing here is when child C
tells parent B to retry, what should happen to any siblings of C? Once
the generator pipeline of B runs, you've got some expectations of
idempotence. So I don't see how you could keep them if you don't also
abort C's siblings. And then what do you do if some of C's siblings
have run, but not all?

Anyways, it would help a lot if you had a more concrete use-case we
could discuss? Maybe this is a common pattern we should try to
address. Thanks,

-Brett

[sco...@referabuyer.com]

Hey Guys,

I'm resurrecting this ancient thread because this is exactly the question I've come here to ask.  How do  I retry a parent pipeline within a child pipeline?  Or, how do I retry a parent pipeline based on the output of its child?  Here's my particular use case:

Pipeline A submits a query to BigQuery.  Then a child pipeline A-B is created that waits for the query to finish.  Then a child pipeline A-C is created that depends on A-B and checks the query for errors.  If A-C finds errors then I want to retry pipeline A.  Whether the retry is triggered in A-C or in A doesn't really matter to me.  

I've been having a hard time figuring out how to accomplish this, but it seems like a basic thing so I'm definitely feeling "There must be a better way!".

Any help is greatly appreciated.

[Josh Whelchel (loudr.fm)]

You have the option of yielding a duplicate of pipeline A if A-C finds errors. Then you have a recursive pipeline, but can easily accomplish what you're looking for.