Running in DAG but no running in Recent Tasks

[jasper...@gmail.com]

Hi all,

I have got into a trouble that I have scheduled a dag in daily and the dag can finish with a manually trigger. However, if it is triggered with scheduler, there is only a running label in "DAG Runs", but no tasks in queued or running of recent tasks. 

I have checked the tree view of dag and seen all the tasks are "Operator: undefined" in tooltips of boxes, but for the tasks of manually triggering, their tooltips show "Operator: KubernetesPodOperator".

Do I miss any setting of the cloud compose? Or, Do I need to set anything when I need to run KubernetesPodOpeator in scheulder?

[Feng Lu]

Could you check whether you have set depends_on_past=True in your DAG? If so, a previous failed DAG run may stall tasks in future DAG runs. 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6e0f3bb5-2c45-401c-b471-1553a8ffdc22%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jasper...@gmail.com]

import datetime

from airflow import models
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator

YESTERDAY = datetime.datetime.today() - datetime.timedelta(1)
default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "start_date": YESTERDAY,
    "retries": 1,
    "retry_delay": datetime.timedelta(minutes=1),
    "provide_context": True
}

with models.DAG("parity-dev", 
                default_args=default_args,
                schedule_interval='@daily') as dag:
    ### my dag...

Hi Feng,

I have set "depends_on_past"=False. The above code is my dag configuration. Could you help me to check where is my problem?

Thanks a lot!!!

Jasper

Feng Lu於 2019年3月20日星期三 UTC+8下午3時50分15秒寫道：

Could you check whether you have set depends_on_past=True in your DAG? If so, a previous failed DAG run may stall tasks in future DAG runs. 

On Sun, Mar 17, 2019 at 2:04 AM <jasper...@gmail.com> wrote:

Hi all,

I have got into a trouble that I have scheduled a dag in daily and the dag can finish with a manually trigger. However, if it is triggered with scheduler, there is only a running label in "DAG Runs", but no tasks in queued or running of recent tasks. 

I have checked the tree view of dag and seen all the tasks are "Operator: undefined" in tooltips of boxes, but for the tasks of manually triggering, their tooltips show "Operator: KubernetesPodOperator".

Do I miss any setting of the cloud compose? Or, Do I need to set anything when I need to run KubernetesPodOpeator in scheulder?

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Matt Koson]

Were you ever able to figure out the problem? I have a similar issue.  

[Cedrik Neumann]

Is timedelta(1) actually one day or one second?

There is a nice function, called daysago, in airflow.utils

https://airflow.apache.org/_modules/airflow/utils/dates.html. So yesterday would be days_ago(1)

---

From: cloud-compo...@googlegroups.com <cloud-compo...@googlegroups.com> on behalf of Matt Koson <matt....@cambiahealthplan.com>
Sent: Monday, July 22, 2019 9:25:13 PM
To: cloud-composer-discuss <cloud-compo...@googlegroups.com>
Subject: Re: Running in DAG but no running in Recent Tasks

 

Jasper

Feng Lu於 2019年3月20日星期三 UTC+8下午3時50分15秒寫道：

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6e0f3bb5-2c45-401c-b471-1553a8ffdc22%40googlegroups.com [groups.google.com].
For more options, visit https://groups.google.com/d/optout [groups.google.com].

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/0e070f5f-4b1a-4dcb-9305-4422c56b4414%40googlegroups.com [groups.google.com].