Re: Composer backend timeout after trying to install Python Dependencies

[Crystal Qian]

Hi Thibault,

Can you verify that your requirements.txt follows the syntax described here: https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies?

It's the correct behavior that there is no mention of your dependency if the PyPI dependency fails to install. However, that error message doesn't seem quite right (the image should fail to read if the PyPI dependency can't be installed.)

I installed beautifulsoup4 on an environment this morning with the following: 

$ echo "beautifulsoup4" > foo.txt

$ gcloud beta composer environments update test-environment --update-pypi-packages-from-file foo.txt

Waiting for [projects/XXX/locations/us-central1/environments/test-environment] to be updated with [projects/XXX/locations/us-central1/operations/2eb315af-7fde-419a-8509-e3e7dd86cf93]

...done.                                                                                                                                                          

Are you consistently getting this error message, also?

Crystal

On Wed, Jul 11, 2018 at 1:32 AM thibault clement <thibau...@gmail.com> wrote:

Hello,

I try to install beautifulsoup4 to my Cloud Composer environment.

Whatever I use the console UI or gcloud (--update-pypi-packages-from-file), I always get the same message after few minute

UI message:

Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING description: "Composer Agent Running. Latest Agent Stage: stage: DEPLOYMENTS_UPDATED\n ." response_timestamp { seconds: 1531296672 nanos: 663000000 } ].

gcloud message:

$ gcloud beta composer environments update XXXXXXX \

→ --update-pypi-packages-from-file airflow/dependencies/requirements.txt \

→ --location europe-west1

Waiting for [projects/XXXXXXX/locations/europe-west1/environments/XXXXXXX] to

be updated with [projects/XXXXXXX/locations/europe-west1/operations/46ec486b-c3f7-4489-a7

8f-250f8eeb06d4]...failed.

ERROR: (gcloud.beta.composer.environments.update) Error updating [projects/XXXXXXX/locations/europe-west1/environments/XXXXXXX]: Operation [projects/XXXXXXX/locations/europe-west1/operations/46ec486b-c3f7-4489-a78f-250f8eeb06d4] failed: Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING

description: "Composer Agent Running. Latest Agent Stage: stage: DEPLOYMENTS_UPDATED\n ."

response_timestamp {

  seconds: 1531293859

  nanos: 661000000

}

].

Then, when I connect to a pod and list the dependencies installed using `pip list` there is no mention of beautifulsoup4.

Do you have any idea of what I'm doing wrong ?

Thanks for your help,

Thibault

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/8236bba3-af61-4fcc-9e44-cb4fea9e07c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[thibault clement]

Thanks for your answer Crystal.

In fact I removed it because the issue wasn't related to python dependencies at all but was trigger by it.

I explain:

My worker was configured to have 20GB. When I add a new python dependency to my environment, my workers pod are recreated.

When I listed my pods using `kubectl get pods` I got some pods with status `ErrImagePull`.

When I describe this error pod, I see this kind of message in the events:

` Warning  Failed                 5m                 kubelet, gke-europe-west1-data-pi-default-pool-3315468f-nmgq  Failed to pull image "gcr.io/XXXXXXXXXX/8472edf9-b666-4027-984f-40462b917927": rpc error: code = Unknown desc = failed to register layer: mkdir /var/lib/docker/overlay/c1b2661600c3b7b30d994bae90c8081fb0599fb60b8339ef7f5ee1e0b67cff8e/tmproot290498040/usr/local/lib/python2.7/site-packages/botocore/data/es: no space left on device`

Looks like configure your worker to have only 20GB is not enough if worker need to be updated.

I recreated an environment and keep the worker disk size to default: 100GB and everything was working fine.

Maybe the minimum 20GB for worker need to be increased to avoid this.

Thibault

--

Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

[Crystal Qian]

Hi Thibault,

Thanks for sharing this; I'll loop in our writer and see if we can explicitly note this to set expectations for users. :)

Crystal