Failed to open TCP Connection to 169.254.169.254:80
7,307 views
Skip to first unread message
Edward Greathouse
Oct 14, 2019, 2:17:49 PM10/14/19
to Google Stackdriver Discussion Forum
Hello,
I'm using a VM that has Windows Server 2012 R2. I've created a project, a service account key, and installed Stackdriver's Logging Agent on the VM. However, the fluentd.log consistently shows the following error each time it's restarted:
2019-10-10 16:44:06 -0400 [error]: Failed to access metadata service: error_class=Errno::ETIMEDOUT error="Failed to open TCP connection to 169.254.169.254:80 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. - connect(2) for \"169.254.169.254\" port 80)"
This seems like a bug a 169.254.x.x address is one that a computer assigns itself if it can't find a DHCP server. Why would it try to open a TCP connection to that IP address? Has anyone else experienced this problem? ...I'm sure most users are on Linux...
I'm using a VM that has Windows Server 2012 R2. I've created a project, a service account key, and installed Stackdriver's Logging Agent on the VM. However, the fluentd.log consistently shows the following error each time it's restarted:
2019-10-10 16:44:06 -0400 [error]: Failed to access metadata service: error_class=Errno::ETIMEDOUT error="Failed to open TCP connection to 169.254.169.254:80 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. - connect(2) for \"169.254.169.254\" port 80)"
This seems like a bug a 169.254.x.x address is one that a computer assigns itself if it can't find a DHCP server. Why would it try to open a TCP connection to that IP address? Has anyone else experienced this problem? ...I'm sure most users are on Linux...
Igor Peshansky
Oct 14, 2019, 2:21:41 PM10/14/19
to Edward Greathouse, Google Stackdriver Discussion Forum
That's the address of the GCE metadata server. The agent uses the metadata server to detect the platform and extract information about the VM instance. This behavior is the same on both Linux and Windows.
Do you, by chance, have a proxy set up? Have you added the no_proxy variable, as mentioned in https://cloud.google.com/logging/docs/agent/installation#agent-install-windows?
Igor
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/8507cbfa-c70e-46f3-ada3-99377bb65704%40googlegroups.com.
Edward Greathouse
Oct 14, 2019, 3:08:14 PM10/14/19
to Google Stackdriver Discussion Forum
Hi Igor,
There's a chance the VM is setup behind a proxy since the IT department handles creating the VMs. Would I need to request the PROXY_IP and PROXY_PORT from them, or is there a way to find that myself?
I did run the following though:
"setx no_proxy 160.254.169.254 /m"
and I received the response:
"SUCCESS: Specified value was saved."
But after restarting the service, I still received the same error.
Thank you for your help so far!
There's a chance the VM is setup behind a proxy since the IT department handles creating the VMs. Would I need to request the PROXY_IP and PROXY_PORT from them, or is there a way to find that myself?
I did run the following though:
"setx no_proxy 160.254.169.254 /m"
and I received the response:
"SUCCESS: Specified value was saved."
But after restarting the service, I still received the same error.
Thank you for your help so far!
On Monday, October 14, 2019 at 2:21:41 PM UTC-4, Igor Peshansky wrote:
That's the address of the GCE metadata server. The agent uses the metadata server to detect the platform and extract information about the VM instance. This behavior is the same on both Linux and Windows.Do you, by chance, have a proxy set up? Have you added the no_proxy variable, as mentioned in https://cloud.google.com/logging/docs/agent/installation#agent-install-windows?Igor
On Mon, Oct 14, 2019 at 2:17 PM Edward Greathouse <egrea...@ivansinsurance.com> wrote:
Hello,--
I'm using a VM that has Windows Server 2012 R2. I've created a project, a service account key, and installed Stackdriver's Logging Agent on the VM. However, the fluentd.log consistently shows the following error each time it's restarted:
2019-10-10 16:44:06 -0400 [error]: Failed to access metadata service: error_class=Errno::ETIMEDOUT error="Failed to open TCP connection to 169.254.169.254:80 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. - connect(2) for \"169.254.169.254\" port 80)"
This seems like a bug a 169.254.x.x address is one that a computer assigns itself if it can't find a DHCP server. Why would it try to open a TCP connection to that IP address? Has anyone else experienced this problem? ...I'm sure most users are on Linux...
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
Igor Peshansky
Oct 14, 2019, 3:20:54 PM10/14/19
to Edward Greathouse, Google Stackdriver Discussion Forum
Looks like you have a typo there — the IP address of the GCE metadata server is 169.254.169.254, not 160.254.169.254. For commands like this, it's better to copy/paste them from the docs when possible...
You shouldn't need to know the exact proxy settings your IT has set up — the logging agent service should pick up the no_proxy variable and use it with the existing proxy. The idea is that the connection to the metadata server is not really external, and thus should not be routed through the proxy.
If fixing the above typo does not help with this problem, you should try accessing the GCE metadata server from that VM instance when you connect to it directly, e.g., using something like "Invoke-WebRequest -Uri http://169.254.169.254/computeMetadata/v1/instance/id -Headers @{'Metadata-Flavor' = 'Google'}".
Hope this helps.
Igor
On Mon, Oct 14, 2019 at 3:08 PM Edward Greathouse <egrea...@ivansinsurance.com> wrote:
Hi Igor,
There's a chance the VM is setup behind a proxy since the IT department handles creating the VMs. Would I need to request the PROXY_IP and PROXY_PORT from them, or is there a way to find that myself?
I did run the following though:
"setx no_proxy 160.254.169.254 /m"
and I received the response:
"SUCCESS: Specified value was saved."
But after restarting the service, I still received the same error.
Thank you for your help so far!
On Monday, October 14, 2019 at 2:21:41 PM UTC-4, Igor Peshansky wrote:
That's the address of the GCE metadata server. The agent uses the metadata server to detect the platform and extract information about the VM instance. This behavior is the same on both Linux and Windows.Do you, by chance, have a proxy set up? Have you added the no_proxy variable, as mentioned in https://cloud.google.com/logging/docs/agent/installation#agent-install-windows?Igor
On Mon, Oct 14, 2019 at 2:17 PM Edward Greathouse <egrea...@ivansinsurance.com> wrote:
Hello,--
I'm using a VM that has Windows Server 2012 R2. I've created a project, a service account key, and installed Stackdriver's Logging Agent on the VM. However, the fluentd.log consistently shows the following error each time it's restarted:
2019-10-10 16:44:06 -0400 [error]: Failed to access metadata service: error_class=Errno::ETIMEDOUT error="Failed to open TCP connection to 169.254.169.254:80 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. - connect(2) for \"169.254.169.254\" port 80)"
This seems like a bug a 169.254.x.x address is one that a computer assigns itself if it can't find a DHCP server. Why would it try to open a TCP connection to that IP address? Has anyone else experienced this problem? ...I'm sure most users are on Linux...
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/8507cbfa-c70e-46f3-ada3-99377bb65704%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/6c840969-b740-409b-9e14-8445fadab7f7%40googlegroups.com.
Edward Greathouse
Oct 14, 2019, 3:39:45 PM10/14/19
to Google Stackdriver Discussion Forum
Hi Igor,
Ah! That was incredibly lazy of me. I did use the correct ip address of 169.254.169.254. Unfortunately, I still see the error in the fluentd.log file after restarting the service.
Also, I tried invoking the web request from the VM through PowerShell, but I received the following error:
"Invoke-WebRequest : Unable to connect to the remote server at line:1 char: 1 ..." - see the attached screenshot.
I really thank you for the help you've given me so far and I hope you can continue helping me.
Thanks,
Eddie
Ah! That was incredibly lazy of me. I did use the correct ip address of 169.254.169.254. Unfortunately, I still see the error in the fluentd.log file after restarting the service.
Also, I tried invoking the web request from the VM through PowerShell, but I received the following error:
"Invoke-WebRequest : Unable to connect to the remote server at line:1 char: 1 ..." - see the attached screenshot.
I really thank you for the help you've given me so far and I hope you can continue helping me.
Thanks,
Eddie
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/8507cbfa-c70e-46f3-ada3-99377bb65704%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
Igor Peshansky
Oct 14, 2019, 4:44:46 PM10/14/19
to Edward Greathouse, Google Stackdriver Discussion Forum
Heh, interesting. Looks like Invoke-WebRequest actually ignores the proxy by default, unless it's explicitly specified with -Proxy. So the error you're getting is probably unrelated to your proxy settings.
Just some sanity-checks:
- Is your VM on GCE?
- Does your VM use one of the GCE default Windows images (as opposed to a custom Windows image)?
- Do you have a local firewall enabled on your VM that may be preventing outgoing connections? Do you have to add a route to the metadata server? See https://serverfault.com/questions/982517/why-cant-i-access-metadata-server-of-gcp-instance.
Hope this helps.
Igor
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/8507cbfa-c70e-46f3-ada3-99377bb65704%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/6c840969-b740-409b-9e14-8445fadab7f7%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/45afd2cc-9ab3-4a91-9ee6-e88f323e59e7%40googlegroups.com.
Edward Greathouse
Oct 14, 2019, 5:14:34 PM10/14/19
to Google Stackdriver Discussion Forum
Hi Igor,
Well, the VM is not on GCE. It's a server that's hosted here (on-campus). There could be a firewall enabled on the VM, but I'm able to ping various external addresses. Is there a specific address that I need to request access to? I can work with IT to fix any firewall issues.
I viewed the serverfault post link you sent. It looks like the author was using a GCE instance. :(
Fun Fact: I was able to successfully create a bucket through API calls, but I have an application that makes a bunch of syslog calls, so changing them to API calls would be time-consuming. I read that I would be able to package and send syslogs to Stackdriver via the Logging Agent (since it is built from fluentd), so I'm hoping I can get this to work. Honestly, my application currently lacks a decent log aggregator and I just want to fix that.
Well, the VM is not on GCE. It's a server that's hosted here (on-campus). There could be a firewall enabled on the VM, but I'm able to ping various external addresses. Is there a specific address that I need to request access to? I can work with IT to fix any firewall issues.
I viewed the serverfault post link you sent. It looks like the author was using a GCE instance. :(
Fun Fact: I was able to successfully create a bucket through API calls, but I have an application that makes a bunch of syslog calls, so changing them to API calls would be time-consuming. I read that I would be able to package and send syslogs to Stackdriver via the Logging Agent (since it is built from fluentd), so I'm hoping I can get this to work. Honestly, my application currently lacks a decent log aggregator and I just want to fix that.
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/8507cbfa-c70e-46f3-ada3-99377bb65704%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/6c840969-b740-409b-9e14-8445fadab7f7%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
Igor Peshansky
Oct 14, 2019, 5:18:36 PM10/14/19
to Edward Greathouse, Google Stackdriver Discussion Forum
Ah, that explains it. The agents will not work on non-cloud VMs: see here and here. You might want to use the BindPlane product from our partner, Blue Medora.
Igor
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/8507cbfa-c70e-46f3-ada3-99377bb65704%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/6c840969-b740-409b-9e14-8445fadab7f7%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/45afd2cc-9ab3-4a91-9ee6-e88f323e59e7%40googlegroups.com.
--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/95ba5606-b63e-45d1-b445-3deed034d32b%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages