Dataflow job successfully completes but Airflow thinks that it is still running.

[Monte Goode]

An airflow instance got itself into a bad state so I redeployed it (composer-1.8.0-airflow-1.9.0) and reinstalled my jars and DAGs. I have a job that runs, then a second one runs after the first one completes, and then a few others run after the first two are complete.

Here’s the problem:

The first job executes in data flow and completes. So then the second job executes in data flow and it completes as well - in Dataflow. It never gets marked as completed in Airflow, so Airflow thinks the job is just perpetually running which gets the DAG stuck in an incomplete perpetually running state.

If I manually mark the job as completed using the UI, then the rest of the jobs fire. But when the DAG runs again the next hour, the same thing happens.

Is there a solution to Airflow not seeing that jobs have successfully completed?

[Guillem Xercavins]

Hi,

Are you using a regional endpoint different than us-central1? If so, it could be related to this: https://stackoverflow.com/questions/58546097/composer-does-not-see-dataflow-job-succeeded/58547282#58547282

Regards,

Guillem

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/8d3a6eba-41ba-4d47-90e8-b032dd23b041%40googlegroups.com.

--

Guillem Xercavins  Big Data Team Lead - Google Cloud Platform Support  Webhelp  guillemx...@google.com