Error notice in composer UI: Composer Backend timed out. Currently running tasks are [stage: TP_DEPLOYMENT_UPDATING response_timestamp { seconds: 1538597221 nanos: 932000000 } ].

[Imran Hassanali]

Hello all, 

I see this error when I click on my cloud composer enviornment:

Composer Backend timed out. Currently running tasks are [stage: TP_DEPLOYMENT_UPDATING response_timestamp { seconds: 1538597221 nanos: 932000000 } ].

I have never seen this before and have not changed any code or configs/IAM access in a couple weeks.  Not sure why I am getting this notice now.  My airflow jobs are still running as expected but worried this notice is going to cause issues down the line. 

Thanks,

--

Imran Hassanali | im...@essential.com

[lc...@lyft.com]

+1

[Feng Lu]

Did someone in your project recently try to update the Composer environment (e.g., install a new pypi package)? 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e40ad63e-36a9-4c09-8685-bc8a55048ae6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Feng Lu]

Resolved offline, in both cases the last environment update operation failed, hence the error message (which should really be a warning message).

In general, for any failed update operation, the running Composer environment is unaffected.

On Wed, Oct 10, 2018 at 3:26 PM lchen via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

[Imran Hassanali]

Thanks Feng appreciate the help.  

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CACZXaYu9gZVwhHKTxogoMSf%3DkBy6H0FZ8WKhsvm_A6ARn33ucw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Imran Hassanali | im...@essential.com

[jake.bi...@infusionsoft.com]

I'm also seeing this, but the Airflow UI is also failing w/a 500. I just installed a single package, `dbt`.  Don't try it on an instance you care about 'cause your webserver and workers will disappear :(

I filed https://issuetracker.google.com/issues/117666270 to track this, and am trying to figure out which of the dependencies are causing the failure. Bisecting dbt's dependency list, I see many of these TP_DEPLOYMENT_UPDATING failure messages, but some of the environment worker/webservers run and some do not.

On Thursday, October 11, 2018 at 1:25:14 PM UTC-7, Imran Hassanali wrote:

Thanks Feng appreciate the help.  

On Thu, Oct 11, 2018 at 1:17 PM 'Feng Lu' via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

Resolved offline, in both cases the last environment update operation failed, hence the error message (which should really be a warning message).

In general, for any failed update operation, the running Composer environment is unaffected.

On Wed, Oct 10, 2018 at 3:26 PM lchen via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

+1

On Tuesday, October 9, 2018 at 2:55:59 PM UTC-7, Imran Hassanali wrote:

Hello all, 

I see this error when I click on my cloud composer enviornment:

Composer Backend timed out. Currently running tasks are [stage: TP_DEPLOYMENT_UPDATING response_timestamp { seconds: 1538597221 nanos: 932000000 } ].

I have never seen this before and have not changed any code or configs/IAM access in a couple weeks.  Not sure why I am getting this notice now.  My airflow jobs are still running as expected but worried this notice is going to cause issues down the line. 

Thanks,

--

Imran Hassanali | im...@essential.com

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e40ad63e-36a9-4c09-8685-bc8a55048ae6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CACZXaYu9gZVwhHKTxogoMSf%3DkBy6H0FZ8WKhsvm_A6ARn33ucw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Imran Hassanali | im...@essential.com

[gaurav...@blackrock.com]

I also see this 

Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING description: "Composer Agent Running. Latest Agent Stage: stage: DEPLOYMENTS_UPDATED\n ." response_timestamp { seconds: 1544002533 nanos: 427000000 } ].

after a PyPi package installation failed. But now I am unable to test any tasks or run a dag, I get the following error:

sh testtask.sh dummy-airflow dag_name taskname 12-02-2018
kubeconfig entry generated for us-east1-dummy-airflow-79200d3a-gke.
error: unable to upgrade connection: container not found ("airflow-worker")

ERROR: (gcloud.composer.environments.run) kubectl returned non-zero status code.

Any ideas on what to do here?

Thanks,

GT

[Jake Biesinger]

I wonder what package you were installing?

Composer has some undeclared dependencies and if they conflict with your pypi packages, this error will come up. I was never able to recover from this error short of deleting the environment and recreating it.

See https://issuetracker.google.com/issues/117666270

The current workaround is to use either a virtualenvoperator or a kubernetes pod operator, so you never actually install your packages in the composer environment.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/a42356f9-3c5c-410a-9d9b-43befa0b38a7%40googlegroups.com.

[gaurav...@blackrock.com]

Thanks a lot, this was super helpful. I installed gcsfs==0.2.0 and got this error. `virtualenvoperator` seems like a decent fix. For what it's worth, I could revive the environment by removing all PyPi dependencies from the Console.

To post to this group, send email to cloud-comp...@googlegroups.com.