Escape code for curly bracket in BashOperator?

1,624 views
Skip to first unread message

Chris Peters

unread,
Oct 7, 2015, 7:44:01 AM10/7/15
to Airflow
In the following example in line c (the bash command) which looks like

c = 'echo LOG_HOUR_LOAD={ {{ ti.xcom_pull("hour_calc")[2] }} , {{ ti.xcom_pull("hour_calc")[3] }} } >> parameters.txt;'

I'm trying to render something like:

'echo LOG_HOUR_LOAD={20151006 , 20151008} >> parameters.txt;'

However the first and last curly bracket causes the dag task to fail. I have tried various escape characters without result ( \ and \\ ). Does anyone know how to write:

LOG_HOUR_LOAD={ {{ ti.xcom_pull("hour_calc")[2] }} , {{ ti.xcom_pull("hour_calc")[3] }} } to parameters.txt using the BashOperator

Here's the full code example:

import airflow
import time
from datetime import datetime, timedelta

def calc(ds, execution_date, ti, **kwargs):
ed = int(time.mktime(execution_date.timetuple()))
hour = ((int(ed)/3600) * 3600) - 3600
hour_1 = hour - 3600
hour_p1 = hour + 3600
log_hour_load = time.strftime("%H", time.gmtime(hour))
log_hour_before = time.strftime("%H", time.gmtime(hour_1))
log_hour_p1 = time.strftime("%H", time.gmtime(hour_p1))

return hour, log_hour_load, log_hour_before, log_hour_p1


args = {'owner': 'webstats', 'start_date': datetime(2015, 10, 7, 9, 40, 0), 'depends_on_past': False}
dag = airflow.DAG(dag_id='webstats', schedule_interval=timedelta(hours=1), default_args=args)

t1 = airflow.operators.PythonOperator(task_id='hour_calc', provide_context=True, python_callable=calc, dag=dag)

c = 'echo LOG_HOUR_LOAD={ {{ ti.xcom_pull("hour_calc")[2] }} , {{ ti.xcom_pull("hour_calc")[3] }} } >> parameters.txt;'
t2 = airflow.operators.BashOperator(task_id='parameters', bash_command=c, dag=dag)
t2.set_upstream(t1)


AH

unread,
Oct 7, 2015, 1:00:11 PM10/7/15
to Airflow
Perhaps try escaping it like this? 
{{ '{' }}
Although I didn't know that jinja had trouble with single brackets so maybe that's not the problem?

Maxime Beauchemin

unread,
Oct 8, 2015, 12:15:34 AM10/8/15
to Airflow
Your solution is right, in other cases you can also use raw blocks.
http://jinja.pocoo.org/docs/dev/templates/#escaping

Chris Peters

unread,
Oct 8, 2015, 2:46:37 AM10/8/15
to Airflow
Thanks. It worked after changing to:  {{"{"}}

Reply all
Reply to author
Forward
0 new messages