BaseOperator on_failure_callback issue or something else ?

[Steven Yvinec-Kruyk]

My understanding is that BaseOperator.on_failure_callback simply requires a Python Callable ... I'm doing something a bit hacky but I believe it should work.

Rather than recreate the SlackAPIPostOperator I am wrapping it in a function in order to get the executable callable to use as the on failure callback.

Python 2.7.10 (default, Sep 23 2015, 19:22:00) 

[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> from datetime import datetime

>>> from airflow.operators import SlackAPIPostOperator

>>> def on_failure_callback():

...     slack_failure_notification = SlackAPIPostOperator(

...         task_id='Slack_Failure_Notification',

...         token='XXXX',

...         channel='#airflow',

...         text=":skull: - {time} - {dag} has failed".format(dag='Some Dag',

...                                                           time=datetime.now().strftime(

...                                                               "%Y-%m-%d %H:%M:%S"), ),

...         owner='Failure Handler',

...     )

...     return slack_failure_notification.execute

... 

>>> callback = on_failure_callback()

>>> callback()

The code above works in the shell. In my dag I have set on_failure_callback=callback but it does not appear that it's ever called (I would receive a Slack Notification if it did) Is there something inherently wrong with this approach or am I missing something ?

Any insight here would be appreciated.

Thanks

[Maxime Beauchemin]

Here's the call:

https://github.com/airbnb/airflow/blob/255a1ab1451138960afca12e440ab596fb0a1557/airflow/models.py#L1037

There might be more to it, but you need to specify one argument to your function that can receive the context. Here's the list of what you should expect to receive in the context dict:

http://pythonhosted.org/airflow/code.html#macros

Max

[Steven Yvinec-Kruyk]

Awesome ... That did it ! ... Thanks Max ! Should have caught that one myself.