Updating of Google Cloud Composer failing

[Fabian Loo]

Hi All,

I am trying to upgrade from composer-1.1.1-airflow-1.9.0 to composer-1.7.1-airflow-1.10.2 by following the instructions here but it is not working. We have already assigned project editor status to our Cloud Composer service account prior to the upgrade. To run the upgrade, we ran the following command:

➜  ~ gcloud beta composer environments update dwh-stable --location us-east1 --image-version composer-1.7.1-airflow-1.10.2

After more than 1 hour, I got the notification that the upgrade has failed. On checking the failed operation, I see the following error message:

➜  ~ gcloud beta composer operations describe --location us-east1 0695de99-b378-4602-b9e7-75bdc6ffd10a
done: true
error:
  code: 3
  message: |
    Http error status code: 400
    Http error message: BAD REQUEST
    Additional errors:
        {"ResourceType":"gcp-types/storage-v1:storage.objects.list","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"errors":[{"domain":"global","message":"<service-acct-no>@cloudservices.gserviceaccount.com does not have storage.objects.list access to composer-unlistable.","reason":"forbidden"}],"message":"<service-acct-no>@cloudservices.gserviceaccount.com does not have storage.objects.list access to composer-unlistable.","statusMessage":"Forbidden","requestPath":"https://www.googleapis.com/storage/v1/b/composer-unlistable/o","httpMethod":"GET","suggestion":"Consider granting permissions to <service-acct-no>@cloudservices.gserviceaccount.com"}}
metadata:
  '@type': type.googleapis.com/google.cloud.orchestration.airflow.service.v1beta1.OperationMetadata
  createTime: '2019-06-06T03:02:41.024Z'
  endTime: '2019-06-06T04:02:47.861Z'
  operationType: UPDATE
  resource: projects/<project-name>/locations/us-east1/environments/dwh-stable
  resourceUuid: a444f6a3-ffd5-4aa0-96a9-4a9d2ac3b675
  state: FAILED
name: projects/<project-name>/locations/us-east1/operations/0695de99-b378-4602-b9e7-75bdc6ffd10a

The image version in Google Cloud Composer UI for the environment is also still showing the old version composer-1.1.1-airflow-1.9.0. We have no luck progressing from here since there doesn't seem to be any error logs from any other places such as workloads to further troubleshoot as well besides the error message above.  

Thank you.

Regards

Fabian

[Sergei Guschin]

Interestingly i start getting same error on upgrading from composer-1.8.0-airflow-1.10.2 to composer 1.8.0-airflow-1.10.3 but the problem is I can't match service account to any of my projects.

[Rick Otten]

On Monday, November 4, 2019 at 1:06:11 PM UTC-5, Sergei Guschin wrote:

Interestingly i start getting same error on upgrading from composer-1.8.0-airflow-1.10.2 to composer 1.8.0-airflow-1.10.3 but the problem is I can't match service account to any of my projects.

I just opened a google support ticket for this on Sunday.  The support team was able to reproduce the issue and has referred it to the composer support team.  I think upgrades to composer-1.8.0-airflow-1.10.3 do not work yet for anyone.

 

[Wai Yin Li]

I have got this error when I install pypl package to the environment, does it mean the error is caused by API services account?

Rick Otten於 2019年11月5日星期二 UTC+8下午8時09分31秒寫道：

[Amol Dabade]

Hi All,

Looks like composer team has solved the issue in recent release. I am trying this out.

[Sergei Guschin]

Yep, i upgraded without any issues so far.

[Hariharan Nagarajan]

Hi Fabian Loo & All,

The issue is not with just Google Composer versions, but possibly due to the pre-step that we might miss while doing the upgrade. And, the pre-step is related to the version that we want to upgrade to.

Considering any composer/ airflow version of the form: x.y.z, where

• x  = Major version - Not allowed for upgrade (https://cloud.google.com/composer/docs/how-to/managing/upgrading#limitations)
• y = Minor version - If there is a minor version upgrade, as a pre-step, we have to remove all existing PYPI packages before doing an upgrade
• z = Revision - Doesn't require (mostly) any pre-steps

The current documentation lacks details today, but the above approach should almost always work.

Thanks,

Hari

[Benjamin GRENIER]

Hi,

My upgrade from composer1.8.3-airflow1.10.3 to composer1.9.0-airflow1.10.6 works but the scheduler is down.

I have the next dialog on Airflow UI :

 

The scheduler does not appear to be running. Last heartbeat was received 21 minutes ago.

The DAGs list may not update, and new tasks will not be scheduled.1

Pod error : 

Back-off 5m0s restarting failed container=airflow-scheduler pod=airflow-scheduler-65d7978dd7-ngcxh_composer-1-9-0-airflow-1-10-6-ea288310(019d922f-548f-11ea-9659-4201ac10000a): CrashLoopBackOff

Show Details

Please find attach logs.

[Hariharan Nagarajan]

Hi Benjamin,

If this was working  in the earlier version, it is most probably failing due to incompatibility of inter-dependent package versions.

Please install pip version 19.0.2, along with your other dependencies, and check other dependencies related to MySQL.

Thanks,

Hari

[Sergei Guschin]

Uninstalling packages helped to complete upgrade (to newer version composer-1.9.1-airflow-1.10.6), but Scheduler failing with SQL Lost connection:

File "/opt/python3.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 208, in __init__ super(Connection, self).__init__(*args, **kwargs2) sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2006, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0")

[GRENIER Benjamin]

Hi Seigei,

Same issue about MySQL connnectio de lost.I opened a support ticket 2 days ago.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5b978227-ac75-4cc4-9841-8b38a80d8c0d%40googlegroups.com.

--------------------------------------------------------------------------------------------

This e-mail transmission (message and any attached files) may contain information that is proprietary, privileged and/or confidential to Veolia Environnement and/or its affiliates and is intended exclusively for the person(s) to whom it is addressed. If you are not the intended recipient, please notify the sender by return e-mail and delete all copies of this e-mail, including all attachments. Unless expressly authorized, any use, disclosure, publication, retransmission or dissemination of this e-mail and/or of its attachments is strictly prohibited. 

Ce message electronique et ses fichiers attaches sont strictement confidentiels et peuvent contenir des elements dont Veolia Environnement et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc destines a l'usage de leurs seuls destinataires. Si vous avez recu ce message par erreur, merci de le retourner a son emetteur et de le detruire ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la publication, la distribution, ou la reproduction non expressement autorisees de ce message et de ses pieces attachees sont interdites.

--------------------------------------------------------------------------------------------

[Raimondas Tijūnaitis]

I managed to fix bricked CC cluster (after performing upgrade from composer-1.7.5-airflow-1.10.2 to composer-1.9.1-airflow-1.10.3) my manually scaling scheduler and worker PODs in backing Kubernetes cluster. To do that go to Kubernetes console, select your cluster and in the workloads pick non-running POD. Then in Actions scale it to required number 

[Sergei Guschin]

What i see on updated cluster that airflow-sqlproxy-service.default is not registered any more and hence failed to resolve. Any ideas how to fix that?

[Sergei Guschin]

Look like latest release (composer-1.9.2-airflow-1.10.6) fixed SQL connection issue on my beaten test project.

But still struggling with updating packages as some failing, for example: pandas-gbq==0.10.0, oauth2client==4.1.3 and google-cloud-bigquery==1.12.1

[Lan Dung]

Hi Sergei,

I'm also using pandas-gbq and had to add the below pypi dependencies to fix it. 

Cloud composer uses some old google-cloud libraries that need to be updated.

pandas-gbq==0.10.0

google-cloud-translate==2.0.1

google-cloud-logging==1.15.0

google-cloud-bigquery==1.23.0

google-cloud-storage==1.20.0

google-cloud-bigtable==1.2.1

google-cloud-spanner==1.14.0

googleapis-common-protos==1.51.0