Composer version updates

[Terrence Szymanski]

Hi - I am wondering how upgrading to new versions of Composer works. For example if I create a new Composer environment today, it seems to be based off of a different image than a Composer environment I created a few weeks ago. For example "composer-0.5.0-airflow-1.9.0". Is there a way to upgrade the old environment to the new Composer version? Or does this happen automatically, and is this an option I can enable or disable? In another thread it was mentioned that Composer will follow the Airflow release cycle. https://groups.google.com/d/topic/cloud-composer-discuss/KIm_kCUck7s/discussion -- will this apply only to newly created Composer environments or will it be available (or auto updated) on existing environments as well?

Terry

[Trevor Edwards]

In the near future, there will be no option for in-place upgrade of Composer environments; therefore, you will have to create a new environment whenever you wish to upgrade.

Composer is working toward supporting multiple Airflow versions as well as in-place upgrades, but no ETA for this yet.

[Terrence Szymanski]

Cheers, thanks for the explanation. 

[Tobias Kaymak]

What is the recommended way of migrating an environment to a newer one without losing the state of the DAGs?

It might be beneficial for us to upgrade to get bugfixes. Currently, we are rolling out new composer image versions when there is a release, in contrast creating a new environment causes that there is also a new bucket created which then needs additional configuration so that our Container Builder pipeline publishes the code to the new bucket. This way we are missing out updates to the Kubernetes configuration though.

Tobi

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5a01ef70-8df6-4254-9b1d-78b9331fcd0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Feng Lu]

It's on our feature roadmap to support in-place version upgrade. 

Meanwhile, as a temporary workaround, you could update the deployment image manually, i.e., 

kubectl set image deployment/airflow-{scheduler|worker} airflow-{scheduler|worker}='the next composer airflow docker image tag.'. 

Note that pypi dependency installation will reset the airflow image version to the old one when the environment was initially created. 

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAL-8VMU7Ru8_BTbnGjd-kDFdm16pc16dXKerOYx2src1D2B%2Brw%40mail.gmail.com.

[tomas....@unacast.com]

I tried to set the image and the pods starts up, but it didn't run properly so I tried to reset it to the old image and now the container fails on the liveness probe. 

The image I reset to is `gcr.io/cloud-airflow-releaser/gcs-syncd:cloud_composer_service_2018-04-29-RC3`, and the error is:

Liveness probe failed: python: can't open file '/var/local/scheduler_checker.py': [Errno 2] No such file or directory

The error message is quite clear, but I can't understand why the file isn't the container since that image worked before. This is not critical since it is a dev environment, but I would like to know what goes on if we would ever try to upgrade the production environment.

On Tuesday, July 10, 2018 at 11:51:42 PM UTC+2, Feng Lu wrote:

It's on our feature roadmap to support in-place version upgrade. 

Meanwhile, as a temporary workaround, you could update the deployment image manually, i.e., 

kubectl set image deployment/airflow-{scheduler|worker} airflow-{scheduler|worker}='the next composer airflow docker image tag.'. 

Note that pypi dependency installation will reset the airflow image version to the old one when the environment was initially created. 

On Thu, Jul 5, 2018 at 11:53 PM Tobias Kaymak <tobias...@gmail.com> wrote:

What is the recommended way of migrating an environment to a newer one without losing the state of the DAGs?

It might be beneficial for us to upgrade to get bugfixes. Currently, we are rolling out new composer image versions when there is a release, in contrast creating a new environment causes that there is also a new bucket created which then needs additional configuration so that our Container Builder pipeline publishes the code to the new bucket. This way we are missing out updates to the Kubernetes configuration though.

Tobi

On Tue, Jul 3, 2018 at 8:24 AM Terrence Szymanski <terrence....@anz.com> wrote:

Cheers, thanks for the explanation. 

On Tuesday, July 3, 2018 at 3:08:28 AM UTC+10, Trevor Edwards wrote:

In the near future, there will be no option for in-place upgrade of Composer environments; therefore, you will have to create a new environment whenever you wish to upgrade.

Composer is working toward supporting multiple Airflow versions as well as in-place upgrades, but no ETA for this yet.

On Sunday, July 1, 2018 at 9:40:51 PM UTC-7, Terrence Szymanski wrote:

Hi - I am wondering how upgrading to new versions of Composer works. For example if I create a new Composer environment today, it seems to be based off of a different image than a Composer environment I created a few weeks ago. For example "composer-0.5.0-airflow-1.9.0". Is there a way to upgrade the old environment to the new Composer version? Or does this happen automatically, and is this an option I can enable or disable? In another thread it was mentioned that Composer will follow the Airflow release cycle. https://groups.google.com/d/topic/cloud-composer-discuss/KIm_kCUck7s/discussion -- will this apply only to newly created Composer environments or will it be available (or auto updated) on existing environments as well?

Terry

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5a01ef70-8df6-4254-9b1d-78b9331fcd0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Trevor Edwards]

It looks like you reset to the "gcs-syncd" image instead of the "airflow-worker-scheduler" image.