Re: Log not visible via Airflow web log

[Wilson Lian]

Hi Jason,

The webserver receives certain Composer-provided Airflow configuration values via environment variables, including one that should prevent this behavior. Please perform a NOOP environment variable (e.g., set FOO_IRRELEVANT=bar) and see if this fixes the issue. If it doesn't, please PM me and fen...@google.com (this week's customer oncall) your Airflow Web UI URL.

thanks,

Wilson

On Tue, May 29, 2018 at 5:39 PM, Jason Chen <chingch...@gmail.com> wrote:

 
 Hi,

 I cannot view tasks' logs via Airflow web log (it was working before).

 The message looks as below and it seems DNS issue (i uses xxxx to replace actual hostname for showing purpose).
 Any ideas ?

Thanks.

*** Log file isn't local.
*** Fetching here: http://airflow-worker-xxxxxxxx-xxxxx:8793/log/test_composer/bigquery_task/2018-05-29T00:00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-xxxxxxxx-xxxxx', port=8793): Max retries exceeded with url: /log/test_composer/bigquery_task/2018-05-29T00:00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f863ff26650>: Failed to establish a new connection: [Errno -2] Name or service not known',))

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e1acff29-e8f4-42da-a354-c2b0049a78b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jason Chen]

 
 Hi Wilson,
  Thanks for your reply. Recently, I removed my testing composer env, because we are planning another composer POC cycle.
   Hence, I do not have airflow running to apply your suggestion.
   I will give a try if I run into the similar issue during next POC cycle.
 
 Thanks.
-Jason

[ar...@armadainteractive.com]

Hello,

I encountered this same issue today. Airflow was working fine before. I set a NOOP environment variable but it didn't help.

Trying to view any task log gives a similar message as below.

Any advice? Thanks.

*** Log file isn't local.
*** Fetching here: http://airflow-worker-5d89d7d5b6-n8f6l:8793/log/pb_daily_metrics_prod1/write_top_unit_sets/2018-10-10T08:00:00/6.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-5d89d7d5b6-n8f6l', port=8793): Max retries exceeded with url: /log/pb_daily_metrics_prod1/write_top_unit_sets/2018-10-10T08:00:00/6.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0d8a57f710>: Failed to establish a new connection: [Errno -2] Name or service not known',))

On Tuesday, 5 June 2018 00:57:02 UTC+3, Wilson Lian wrote:

Hi Jason,

The webserver receives certain Composer-provided Airflow configuration values via environment variables, including one that should prevent this behavior. Please perform a NOOP environment variable (e.g., set FOO_IRRELEVANT=bar) and see if this fixes the issue. If it doesn't, please PM me and fen...@google.com (this week's customer oncall) your Airflow Web UI URL.

thanks,

Wilson

On Tue, May 29, 2018 at 5:39 PM, Jason Chen <chingch...@gmail.com> wrote:

 
 Hi,

 I cannot view tasks' logs via Airflow web log (it was working before).

 The message looks as below and it seems DNS issue (i uses xxxx to replace actual hostname for showing purpose).
 Any ideas ?

Thanks.

*** Log file isn't local.
*** Fetching here: http://airflow-worker-xxxxxxxx-xxxxx:8793/log/test_composer/bigquery_task/2018-05-29T00:00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-xxxxxxxx-xxxxx', port=8793): Max retries exceeded with url: /log/test_composer/bigquery_task/2018-05-29T00:00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f863ff26650>: Failed to establish a new connection: [Errno -2] Name or service not known',))

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

[jake.bi...@infusionsoft.com]

I've seen this one as well. Refreshing and/or rerunning the task seemed to fix it, though that's a lame workaround.

[Feng Lu]

Could you let us know your Composer environment version? In addition, are you using the Airflow webserver provided by Composer or managing your own webserver[1]? 

[1]. https://cloud.google.com/composer/docs/how-to/managing/deploy-webserver

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e1acff29-e8f4-42da-a354-c2b0049a78b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/d7e754bc-6b51-4a62-80d5-9a621a4c8f93%40googlegroups.com.

[Arttu Modig]

I used Composer-provided Airflow, version composer-1.1.1-airflow-1.9.0. I had made some configuration overrides and installed additional PyPi packages.

Re-running tasks didn't help. Seeing the logs was important, so I ended up creating a new Composer environment, now with Python 3 and latest composer-1.2.0-airflow-1.9.0. I'm sorry but I already deleted the old environment.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e1acff29-e8f4-42da-a354-c2b0049a78b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Feng Lu]

No worries, unfortunately it's a bit hard for us to debug the issue without the erroneous environment, might be related to config changes. 

Note that task logs are streamed and available in stackdriver logging as well, details here.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e1acff29-e8f4-42da-a354-c2b0049a78b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/d7e754bc-6b51-4a62-80d5-9a621a4c8f93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5c74f1fe-f369-45c0-9668-8286b4d6ec03%40googlegroups.com.

[Edward Marco Baringer]

i'm currently facing this issue as well, and none of the suggested fixes seem to help. i can provide a link (or whatever info) to the currently running, and broken, composer env.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e1acff29-e8f4-42da-a354-c2b0049a78b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/d7e754bc-6b51-4a62-80d5-9a621a4c8f93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[lsalva...@quilsar.com]

On Friday, November 30, 2018 at 3:42:52 PM UTC+1, Edward Marco Baringer wrote:

i'm currently facing this issue as well, and none of the suggested fixes seem to help. i can provide a link (or whatever info) to the currently running, and broken, composer env.

Same problem here, airflow webserver cannot reach worker logs with the error 'Name or service not known'

None of the suggested fixes helped:

- adding a fake variable

- adding a python package

I've still the composer environment active, someone has already solved this problem?

[Lorenzo Salvadorini]

just to share, the workaround I've found is to deploy a webserver pod inside the compose kubernetes

https://cloud.google.com/composer/docs/how-to/managing/deploy-webserver

--
You received this message because you are subscribed to a topic in the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloud-composer-discuss/wOdZ3Kah5Ls/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/a3132a4c-1583-4c3c-9799-0dfe6693f92d%40googlegroups.com.

[shubhangi agarwal]

I am also facing the same issue. Were you able to find a solution?

[Amol Dabade]

Nope, no solution yet but preliminary cause it seems like composer web agent doesn't have access to bucket its looking for. 

I am using following to look at logs for now.

gsutil cat gs://{composer_bucket}/logs/{dag_id}/{task_id}/{timestamp}/{try_number}.log

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/de887113-451a-4013-8719-926995c5b4e5%40googlegroups.com.

[Rodrigo Chaparro Plata Hernández]

Facing the same issue, but I think I found the error, but not know where to report it to be fixed.

I think that the issue is this:

Airflow tries to reach the log from the http url in the example above:

http://airflow-worker-xxxxxxxx-xxxxx:8793/log/test_composer/bigquery_task/2018-05-29T00:00:00/1.log

 and as you can see (if you open the url in a browser, this return an error) the problem in here is the colons in the time section after the date

the browser think that those are the port, some part of the url or something like that (like the beginning of the url, airflow-worker-xxxxxxxx-xxxxx:8793) so the url must be URL encoded to replace the colons (:) with %3A

here is an example of how Google Cloud Storage generates the URL:

[Álvaro Gomez Covella]

Hello, after upgrading to: composer-1.10.6-airflow-1.10.6, we are also facing this problem in some Environments.
It is a bit annoying reading the logs directly on GCS.

Unable to read remote log from gs://............./2020-07-20T00:00:00+00:00/1.log
*** 404 GET https://storage.googleapis.com/download/storage/v1/b/............check_output_success_avro_lowlq_nobid%2F2020-07-20T00%3A00%3A00%2B00%3A00%2F1.log?alt=media: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

*** Log file does not exist: /home/airflow/gcs/logs/........./2020-07-20T00:00:00+00:00/1.log
*** Fetching from: http://airflow-worker-7bdb7bd989-wwr77:8793/log/......../2020-07-20T00:00:00+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-7bdb7bd989-wwr77', port=8793): Max retries exceeded with url: /log/............./2020-07-20T00:00:00+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efdda4114e0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

On Tue, May 26, 2020 at 10:42 PM Yogi Wahyu Romadon <yo...@paper.id> wrote:

I had these problems too, can anyone solve these problems? or this is a bug?

On Tuesday, June 5, 2018 at 4:57:02 AM UTC+7, Wilson Lian wrote:

Hi Jason,

The webserver receives certain Composer-provided Airflow configuration values via environment variables, including one that should prevent this behavior. Please perform a NOOP environment variable (e.g., set FOO_IRRELEVANT=bar) and see if this fixes the issue. If it doesn't, please PM me and fen...@google.com (this week's customer oncall) your Airflow Web UI URL.

thanks,

Wilson

On Tue, May 29, 2018 at 5:39 PM, Jason Chen <chingch...@gmail.com> wrote:

 
 Hi,

 I cannot view tasks' logs via Airflow web log (it was working before).

 The message looks as below and it seems DNS issue (i uses xxxx to replace actual hostname for showing purpose).
 Any ideas ?

Thanks.

*** Log file isn't local.
*** Fetching here: http://airflow-worker-xxxxxxxx-xxxxx:8793/log/test_composer/bigquery_task/2018-05-29T00:00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-xxxxxxxx-xxxxx', port=8793): Max retries exceeded with url: /log/test_composer/bigquery_task/2018-05-29T00:00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f863ff26650>: Failed to establish a new connection: [Errno -2] Name or service not known',))

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e1acff29-e8f4-42da-a354-c2b0049a78b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e23c88fc-2026-4219-8b66-c97d13a1c2b6%40googlegroups.com.

--

Álvaro Gomez Covella
Cloud Data Engineer

Trakken Web Services
Carrer de Sardenya 229
08013 Barcelona

E-mail: alv...@trakken.es
Web: www.trakken.es

_____________________________________________________________________

HRB 104862, Amtsgericht Hamburg
Managing Directors: Timo Aden, Lennart Paulsen
_____________________________________________________________________

[Álvaro Gomez Covella]

Adding more details:

By checking at the error message, I noticed something weird:

*** Unable to read remote log from gs://XXXXXXXXXXXXX-tenant/.......

However, in my airflow.cfg I have:

remote_base_log_folder = gs://XXXXXXXXXXXXXX-bucket/logs

Why is trying to read to XXXX-tenant when it should end up with XXXX-bucket?

[Rafal Biegacz]

Hi,

We are actively working to resolve this issue. We plan to deliver the fix in August timeframe. Once I know the Composer version that fixes the issue I will circle back to you.

For now, the mitigation action that you can take on your side is to remove the logs that you no longer need from the /logs folder in Composer GCS bucket and archive it in some other place (outside of /logs folder).

In the. current implementation of "Private IP" configuration of Composer, web server reads logs from the dedicated GCS bucket that is located in the tenant project , that's why you might see the error presented below in case logs are missing in this GCS bucket.

*** Unable to read remote log from gs://XXXXXXXXXXXXX-tenant/.......

I'm sorry for this issue. We are working on it with the highest priority.

Regards, Rafal.

---

Cloud Composer Team

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CANWZVfBmi_Py5r2PBXkpcrMda8_%2BtxWkPGnp-V8UjchEVjuqPA%40mail.gmail.com.

[Álvaro Gomez Covella]

Thanks for the update Rafa! I appreciate it!

Cheers

[Benjamin GRENIER]

I have received an update from the Cloud Composer engineering team.
They have two more workarounds to suggest:

1. The team can manually update the web server configuration to read logs directly from the bucket in your project. You would first need to grant viewer roles (like roles/storage.legacyBucketReader and roles/storage.legacyObjectReader) on the bucket to the service account running the web server.
This fix will resolve the delay problem immediately but it will last only until the next PyPI package installation, Airflow config override change, or version upgrade that you would perform on the environment through Composer Cloud Console page, CLI, or API.

2. If you don't have DRS (Domain restricted sharing) enabled which I believe you don't. You can create a new Composer environment, this time through v1 Composer API or without Beta features enabled in Cloud Console. This way Composer will create an environment without the DRS-compliant setup, so without the bucket-to-bucket synchronization.
The problem is that you would need to migrate your DAGs and data to the new environment.

Looking forward to your feedback.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e1acff29-e8f4-42da-a354-c2b0049a78b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CANWZVfBmi_Py5r2PBXkpcrMda8_%2BtxWkPGnp-V8UjchEVjuqPA%40mail.gmail.com.