I'm testing running several bigquery operator tasks (very basic queries writing to tables in this example) with one level of dependency. My DAG (bg_flatten_table_new3, attached) has been running on composer for a few days, mostly working 100%. However, every now and then it fails to execute a task with state "upstream failed" even though upstream executed successfully.
Composer log (attached) for this task is empty and instance details (attached) also doesn't give much info.
Any help would be much appreciated.