Not able to run single task in a dag.(Composer 1.0.0)

683 views
Skip to first unread message

nakul...@anz.com

unread,
Jul 24, 2018, 9:42:30 PM7/24/18
to cloud-composer-discuss
Not able to run single task in a dag. This is an issue if I want to test one particular task. The notification suggests using clear so task will start automatically. Using clear sometimes, it gives an error around deadlock on threads, suppose its something to do with celery executer and some sort of multi threading going on. 

Can someone assist in easy way of debugging single task rather than running entire dag.


Trevor Edwards

unread,
Jul 25, 2018, 5:52:02 PM7/25/18
to cloud-composer-discuss
The clearing issue is known - https://issues.apache.org/jira/browse/AIRFLOW-351, and someone is currently working on fixing it. It is caused by Airflow attempting to deep-copy DAGs when determining which tasks to clear. Unfortunately this means that there is no way to re-run an individual task that experiences this error without directly modifying the airflow database or swapping in a dummy-dag which is picklable temporarily. Both of these are far from ideal.

Have you tried using the Airflow CLI 'test' command? https://airflow.apache.org/cli.html#test
Reply all
Reply to author
Forward
0 new messages