PythonOperator: context vs op_args

[Brenda Bell]

I'm trying to figure out how the various operator args work for the PythonOperator. The following code works:

def print_context(ds, *args, **kwargs):

    pprint(args)

    pprint(kwargs)

    print(ds)

    return 'Whatever you return gets printed in the logs'

workflow = DAG('foo', default_args=default_args)

run_this = PythonOperator(

    task_id='print_the_context',

    provide_context=True,

    python_callable=print_context,

    dag=workflow)

I'm assuming that ds is directly available to print_context because provide_context=True. But if I try to add additional positional parameters, I get errors.

The code:

def print_context(ds, x, *args, **kwargs):

    pprint(args)

    pprint(kwargs)

    print(ds)
    print(x)

    return 'Whatever you return gets printed in the logs'

workflow = DAG('foo', default_args=default_args)

run_this = PythonOperator(

    task_id='print_the_context',

    provide_context=True,

    python_callable=print_context,

    dag=workflow,
    op_args=['1',])

And the error:

Traceback (most recent call last):

  File "/Library/Python/2.7/site-packages/airflow/models.py", line 1000, in run

    result = task_copy.execute(context=context)

  File "/Library/Python/2.7/site-packages/airflow/operators/python_operator.py", line 65, in execute

    return_value = self.python_callable(*self.op_args, **self.op_kwargs)

TypeError: print_context() takes at least 2 arguments (1 given)

Am I limited to using one or the other? Does op_args hide the context?

[Maxime Beauchemin]

Reading the code for the operator (it's only a few lines of code), Airflow passes the context as kwargs.

https://github.com/airbnb/airflow/blob/master/airflow/operators/python_operator.py#L59

What python does there is a little tricky to explain here, but you can fix your code above by either:

* swapping ds and x in the function header, so that that your op_args[0] gets assigned to x as an *arg

* use `op_kwargs={'x': '1'}` in the PythonOperator constructor instead of op_args for a more explicit approach

[Brenda Bell]

That was a big help. Thanks. It all works now.