Upgrading machine type of a Cloud Composer cluster

[Ferdinand]

Hello there,

I am using Terraform in order to deploy and save the state of my composer cluster. I would like to upgrade the workers' machine type but if I do so then Terraform will delete and create back the Composer cluster. 

Since the resource is created again, I lost all the logs of my previous cluster. I know that there is a workaround in order to change the machine types of the default-pool but it is not what I want because Terraform will not take that change into account (the configuration state will remain to the previous machine type).

I also know that there was a script in order to "migrate" a composer cluster and while looking at the code I can say that the big picture of the process was : 

1. create a new composer cluster

2. export the airflow database from the current composer cluster to GCS

3.  import the SQL dump from GCS to the new Cloud SQL instance of the composer cluster

4. copy dags/ , logs/ , plugins/ , data/ folder from the current composer cluster bucket to the new composer cluster bucket

I reproduced the process with a DAG deployed in the current cluster, it uses the "CloudSqlInstanceExportOperator" to export the current instance to GCS but I get an exception saying "The client is not authorized to make this request" while trying to run it.

My idea is to make this DAG to export the Airflow DB to GCS, then redeploying my cluster via Terraform to have the good state, and finally to deploy a DAG on the new Composer cluster to import the SQL dump from GCS to the new Airflow DB.

What do I need to have in terms of permissions to make this work? Is the transfer of Airflow DB doable through a DAG or the given script?

Regards,

Ferdinand

[shyam...@gmail.com]

You might need to provide "storage editor/admin" permissions to the service account of the tenant project that hosts cloudsql. Refer the below documentation:

https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/cloud_sql.html#troubleshooting

[Alberto Vicente]

As you mention you are using Composer (GCP Airflow offering), your Cloud SQL engine is running behind Google Tenant side and you have no access to it, please review the following Summit session to get complete picture 

https://www.youtube.com/watch?v=QhnItssm4yU&list=PLGudixcDaxY3RGLSlWoN_cEEXhIT1OPmj&index=11 , also the instance upgrade in place is limited for latest  also create a DAG metadata clean job (https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/composer/workflows/airflow_db_cleanup.py)

Saying that give it a try to Composer Documentation https://cloud.google.com/composer/docs/how-to/authentication and https://cloud.google.com/composer/docs/how-to/managing/upgrading

Regards

Albert

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/08bbe4f5-e32b-4d2d-8715-53bd38623cacn%40googlegroups.com.

--

	Alberto Vicente | Technical Director - Data & Analytics Studio
	GLOBANT AR: +54 11 4109 1700 ext. 16054 | US: +1 877 215 5230 ext. 16054

The information contained in this e-mail may be confidential. It has been sent for the sole use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution or copying of this communication, or any of its contents, is strictly prohibited. If you have received it by mistake please let us know by e-mail immediately and delete it from your system. Many thanks.

 

La información contenida en este mensaje puede ser confidencial. Ha sido enviada para el uso exclusivo del destinatario(s) previsto. Si el lector de este mensaje no fuera el destinatario previsto, por el presente queda Ud. notificado que cualquier lectura, uso, publicación, diseminación, distribución o copiado de esta comunicación o su contenido está estrictamente prohibido. En caso de que Ud. hubiera recibido este mensaje por error le agradeceremos notificarnos por e-mail inmediatamente y eliminarlo de su sistema. Muchas gracias.

[Marcel S. Gongora]

Hi Ferdinand,

You don't need to replace/destroy the whole environment in order to change this value. The reason is that composer API doesn't use this parameter to do further actions, or at least not that I'm aware of. That's basically why their documented way to change the machine type can be done manually. 

Said that: from my point of view, It doesn't worth it to change the whole environment as I said, which can take more than 1h depending on the size of your cluster. What we have done to keep our IaC clean, was to automatize what they documented and you referenced. You may want to take a look to this module: terraform-google-modules/gcloud/google and this resource: google_container_node_pool