question: creating subdag operators from existing dags

[lm]

Hi,

I have existing dags "a", "b", and "c", they all have different start dates and I want to be able to ensure they run in strict chronological order going forward. The approach currently is scheduling them in order, but we've found it's difficult to ensure each job will finish in time to prevent overlaps. The other requirement is that dag "b" needs to be able to fail without preventing dag c from running but we want dag "c" to run only after dag "b" has either succeeded or failed. At the same time we want the success status to be recorded as failed when dag "b" fails, with the ability to re-run when some external problem has been fixed. 

Questions:

1. is there a way to create only an "execution done" dependency between two jobs, without forfeiting accurate recording of success and failure in the airflow history for any of the jobs?

2. do subdags help with this problem at all?

3. and an unrelated question about subdags: what are the implications of composing subdags into a superdag whereby all the subdags and superdags have different start dates?

Thanks. 

[lm]

update: ok looks like what i really want is externaltasksensor.

so my next question is, what problems do subdags address?

[Arthur Wiedmer]

At Airbnb, 

We use them :

- to create reusable patterns (for isntance, stage data into the system, do data quality checks and then land them into the final location),

- to use a specific and separate type of executor for a group of tasks.

I would love to hear more use cases.

Best,

Arthur

[Maxime Beauchemin]

Thanks to @nicktrav who just documented SubDags here:

http://pythonhosted.org/airflow/concepts.html#subdags

[lm]

the new concepts docmentation is awesome. thanks to all who contributed to that.