Monthly DAG not running as expected

[Jocelyn]

Hi all,

I have some dags set to run on the 1st of every month at 4pm. Ideally, it should run all data for November on the 1st of December however I noticed it only run for the previous period. That is, my current or latest run is on the 1st November (data for October). I am expecting the latest run to have been on the 1st of December, which should run data for November. I've noticed same for weekly DAGs.

Can anyone please help with this? Maybe I'm getting the logic wrong.

from datetime import timedelta, datetime

import json

from airflow import DAG

from airflow.contrib.operators.bigquery_operator import BigQueryOperator

from airflow.contrib.operators.bigquery_check_operator import BigQueryCheckOperator

from airflow.contrib.operators.bigquery_table_delete_operator import BigQueryTableDeleteOperator

from airflow.contrib.operators.bigquery_to_gcs import BigQueryToCloudStorageOperator

from airflow.contrib.operators.gcs_to_bq import GoogleCloudStorageToBigQueryOperator

from airflow.operators.bash_operator import BashOperator

from airflow.operators import email_operator

from airflow.models import Variable

default_args = {

    'owner': 'airflow',

    'depends_on_past': False,

    #'start_date': seven_days_ago,

    'start_date': datetime(2018, 9, 1, 16, 0, 0),

    'email': ['air...@airflow.com'],

    'email_on_failure': True,

    'email_on_retry': True,

    'retries': 5,

    'retry_delay': timedelta(minutes=5),

}

# Set Schedule: Run pipeline first of every month. 

schedule_interval = "0 16 1 * *"  

# Define DAG: Set ID and assign default args and schedule interval

dag = DAG('test_scripts3', default_args=default_args, schedule_interval=schedule_interval)

Thanks 

Jocelyn

[Imran Hassanali]

+1 I have similar dags that do not get triggered automatically by the schedule (both using cron style and @ method).  Also have others that work as intended but do not see any difference in the config. 

Here is my config:  

default_args = {

'owner': 'airflow',

'depends_on_past': False,

'start_date': datetime.datetime.combine(datetime.datetime.today()

- datetime.timedelta(1), datetime.datetime.min.time()),

'email': ['im...@essential.com'],

'email_on_failure': True,

'email_on_retry': False,

'retries': 0,

}

with models.DAG(

dag_id,

default_args=default_args,

schedule_interval='@weekly',

catchup=False,

On Wed, Dec 5, 2018 at 10:38 AM Imran Hassanali <im...@essential.com> wrote:

+1 I have similar dags that do not get triggered automatically by the schedule (both using cron style and @ method).  Also have others that work as intended but do not see any difference in the config. 

Thanks,

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/ec6d0294-aaab-4ea7-bc96-94ff61dfb198%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Imran Hassanali | im...@essential.com

--

Imran Hassanali | im...@essential.com

[Wilson Lian]

Hard to say without more information, but I'd check the scheduler logs in Stackdriver for more clues.

On Tue, Dec 4, 2018 at 1:05 AM Jocelyn <jocelyn...@swiftcover.com> wrote:

--