bug with catchup=false in composer

[Anthony Brown]

Hi

   I have found a problem with composer and setting catchup=False in options when creating a dag - it is always ignored and lots of catchup runs are created by the scheduler. This is not present in apache airflow code

   I have had a look at the code on the scheduler and it appears that the version of airflow being used is slightly different from the main apache release and one of the changes for composer has introduced this bug.

   In composer (version composer-1.5.0-airflow-1.10.1)  airflow/models.py on line 3287 in the constructor for DAG object, the code is 

self.catchup = catchup or configuration.conf.getboolean('scheduler', 'catchup_by_default')

     If I pass catchup=false when I create a dag, then the value is retrieved from the configuration option - which is set to true by default.

   Can somebody from Google have a look at this please and create a fix for it

Thanks

--

-- 

Anthony Brown

Data Engineer BI Team - John Lewis

Tel : 0787 215 7305

**********************************************************************
This email is confidential and may contain copyright material of the John Lewis Partnership.
If you are not the intended recipient, please notify us immediately and delete all copies of this message.
(Please note that it is your responsibility to scan this message for viruses). Email to and from the
John Lewis Partnership is automatically monitored for operational and lawful business reasons.
**********************************************************************

John Lewis plc
Registered in England 233462
Registered office 171 Victoria Street London SW1E 5NN

Websites: https://www.johnlewis.com
http://www.waitrose.com
https://www.johnlewisfinance.com
http://www.johnlewispartnership.co.uk

**********************************************************************

[Wilson Lian]

Hi Anthony,

Thanks for reporting this issue and sharing the details of your investigation. We've created a patch and will be rolling it out with our next service release (composer-1.5.1-airflow-*). We expect to start rolling that out region-by-region this week. Keep an eye out at https://cloud.google.com/composer/docs/release-notes for details.

best,

Wilson

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAHRXA7FSsa0-FhRG4F5rmf9hHP68vNuZR-dEXkxdQzDXBzfEeg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Tanjin Panna]

This looks like it is still a problem. I am on composer-1.6.1-airflow-1.10.1

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Rafal Biegacz]

Hi,

Composer-1.6.1-airflow-1.10.1 is pretty old.

Did you try out the more recent versions of Composer which are based on Airflow 1.10.3/1/10.2?

Regards, Rafal.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAHRXA7FSsa0-FhRG4F5rmf9hHP68vNuZR-dEXkxdQzDXBzfEeg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

This email and any files transmitted with it are confidential and intended solely for the use of the individual to whom they are addressed. If you have received this e-mail by mistake, please notify the sender immediately and delete the message from your system, without disclosing, copying, distributing or taking any action in reliance on its contents.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/f541e0a4-c461-47a1-9afa-0ee6a4a46517%40googlegroups.com.

[Siva Chaitanya]

I am using composer-1.8.2-airflow-1.10.3 and still facing this issue. Is there any fix to this? 

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAHRXA7FSsa0-FhRG4F5rmf9hHP68vNuZR-dEXkxdQzDXBzfEeg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

This email and any files transmitted with it are confidential and intended solely for the use of the individual to whom they are addressed. If you have received this e-mail by mistake, please notify the sender immediately and delete the message from your system, without disclosing, copying, distributing or taking any action in reliance on its contents.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Imad Youbi Idrissi]

It seems this is still an issue. I am using composer-1.7.5-airflow-1.10.2