Windows Server 2016 Volume Usage Monitoring

[Nigel Gutzmann]

Hi there,

I have a few Windows Server 2016 VMs that I'd like to monitor disk usage on. It looks like that metric is available only through the agent (https://cloud.google.com/monitoring/api/metrics#agent-disk). But the agent is only compatible with Server 2008 and Server 2012 (https://cloud.google.com/monitoring/agent/#supported_operating_systems). Does anyone know of a way I could do this through stackdriver? If not, does anyone have any other recommendations that would be able to do it?

Thanks,
Nigel G.

[Irina (Google Cloud Support)]

Hi Nigel,

I’ve installed the Stackdriver Monitoring agent on Windows Server 2016 as outlined here, and then was able to see the disk metrics in the Stackdriver console. 

Could you try to perform the same steps and let me know if it works for you?

[Mary Koes]

Hi there,

I'm a product manager with Google Stackdriver.  I confirmed with the engineering team that the agent works with Windows Server 2016 and this is a bug in our documentation, which we'll fix shortly.  Thanks for the heads up on it!

Best,

Mary 

--
© 2017 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/09774a9e-670b-4814-b9b9-d49f0a16dbc7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nigel Gutzmann]

Thanks, I'll give it a try!

On Thursday, 6 July 2017 13:19:54 UTC-7, Mary Koes wrote:

Hi there,

I'm a product manager with Google Stackdriver.  I confirmed with the engineering team that the agent works with Windows Server 2016 and this is a bug in our documentation, which we'll fix shortly.  Thanks for the heads up on it!

Best,

Mary 

On Thu, Jul 6, 2017 at 3:51 PM, 'Irina (Google Cloud Support)' via gce-discussion <gce-dis...@googlegroups.com> wrote:

Hi Nigel,

I’ve installed the Stackdriver Monitoring agent on Windows Server 2016 as outlined here, and then was able to see the disk metrics in the Stackdriver console. 

Could you try to perform the same steps and let me know if it works for you?

On Thursday, July 6, 2017 at 4:03:27 AM UTC-4, Nigel Gutzmann wrote:

Hi there,

I have a few Windows Server 2016 VMs that I'd like to monitor disk usage on. It looks like that metric is available only through the agent (https://cloud.google.com/monitoring/api/metrics#agent-disk). But the agent is only compatible with Server 2008 and Server 2012 (https://cloud.google.com/monitoring/agent/#supported_operating_systems). Does anyone know of a way I could do this through stackdriver? If not, does anyone have any other recommendations that would be able to do it?

Thanks,
Nigel G.

--
© 2017 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

[Nigel Gutzmann]

Hi Mary,

I installed the Monitoring Agent on a Windows Server 2016 instance and had no problems with it. The metrics started to be reported in the stackdriver dashboard right away. However, now I am trying to replicate that by adding the stackdriver installation into an image that I am using in an instance template for an instance group. I used the instance that was reporting the metrics as the base image, so I'm sure there isn't anything wrong with the image.

However, my instances in the instance group are not reporting any metrics from the Stackdriver Agent. After the instances start up, I can verify that the Stackdriver Agent is running by executing the tasklist command remotely (StackdriverAgent.exe is running as a Service). But for that instance on the Stackdriver dashboard, no metrics are being reported.

Are there maybe some ports that need to be opened in the firewall? My instances in the instance group do not have an external IP, maybe that's an issue? Or could it be something else?

Thanks,

Nigel G.

On Thursday, 6 July 2017 13:19:54 UTC-7, Mary Koes wrote:

Hi there,

I'm a product manager with Google Stackdriver.  I confirmed with the engineering team that the agent works with Windows Server 2016 and this is a bug in our documentation, which we'll fix shortly.  Thanks for the heads up on it!

Best,

Mary 

On Thu, Jul 6, 2017 at 3:51 PM, 'Irina (Google Cloud Support)' via gce-discussion <gce-dis...@googlegroups.com> wrote:

Hi Nigel,

I’ve installed the Stackdriver Monitoring agent on Windows Server 2016 as outlined here, and then was able to see the disk metrics in the Stackdriver console. 

Could you try to perform the same steps and let me know if it works for you?

On Thursday, July 6, 2017 at 4:03:27 AM UTC-4, Nigel Gutzmann wrote:

Hi there,

I have a few Windows Server 2016 VMs that I'd like to monitor disk usage on. It looks like that metric is available only through the agent (https://cloud.google.com/monitoring/api/metrics#agent-disk). But the agent is only compatible with Server 2008 and Server 2012 (https://cloud.google.com/monitoring/agent/#supported_operating_systems). Does anyone know of a way I could do this through stackdriver? If not, does anyone have any other recommendations that would be able to do it?

Thanks,
Nigel G.

--
© 2017 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

[Mary Koes]

Adding Igor, an engineer on the Stackdriver agents team

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/92336e6e-9691-4cdb-b2a7-baacbf9a573c%40googlegroups.com.

[Igor Peshansky]

Hi, Nigel,

As I understand, the agent service is running and does not report any errors or warnings in the Windows event log, correct?

How are you authenticating the agent? Do you use the default GCE credentials, or are you supplying a credentials file?

Have you run Sysprep before creating the image (https://cloud.google.com/compute/docs/instances/windows/creating-windows-os-image)?

Do you see the same behavior with a clean Windows instance image that only has the Monitoring agent installed?

If everything is configured correctly and you're still seeing no metrics, we'll need some information to help us debug. Can you please send me (in a personal email off-list) the project number and instance ids of the affected instances? Would you, by chance, be willing to share the affected image, or does it have private information on it?

        Igor
-- sent from a mobile device, please excuse tyops and omissns

[Nigel Gutzmann]

Hi Igor,

Whoops, I forgot to run Sysprep! I will give that a try and let you know if that fixes the issue.

Cheers,

Nigel G.

[Igor Peshansky]

Nigel,

Thanks for sending me the info off-thread. I see agent metrics for two instances in your project, neither of which seems to be from your instance group. Was the information you sent gathered after you used Sysprep to create the image?

My first guess is that the instances in the instance group don't have the correct scopes (https://cloud.google.com/monitoring/agent/install-agent#verify-creds), but that should have resulted in 403's from the API, which would have appeared in the Event Log. We'll need to debug this further. I'll follow-up off-thread for more details.

        Igor

[Igor Peshansky]

Forgot to ask: were you able to reproduce this on an image created from a clean Windows instance image that only has the Monitoring agent installed?

        Igor

[Nigel Gutzmann]

Hi Igor,

Sorry about the delay getting back to you on this. I just realized that the instances actually didn't have the correct credentials, but I've added them now and it hasn't solved the problem. Now on the Google Compute Dashboard the instances all say that they've got the following Cloud API permissions:

Stackdriver Logging API- Write Only
Stackdriver Monitoring API - Write Only

All other permissions are disabled. The VMs are also currently using a service account which is a project owner (although we are going to remove that soon).

Still seeing no agent metrics in the dashboard though. 

Sysprep was run in creating the initial image. I then installed the monitoring agent and created a second image (the currently used image) off of the initial image without running sysprep again. Is that a problem? If so, I can try that next week.

I can also try to recreate this on a clean Windows instance next week.

Cheers,

Nigel G.

[Igor Peshansky]

Thanks for responding, Nigel.

Yes, getting a repro on a clean VM image that you'll be able to share with us should help us debug this.

Since you're using the GCE authorization token, you could also try running some commands on the affected instance to verify that it is sufficiently authorized to talk to the API (will send the exact commands on Monday). I'd also ask you to double-check that no errors from the Stackdriver agent appear in the Event Viewer.

I'm not fully familiar with creating Windows images, but the instructions do state that creating an image is a destructive operation which requires running Sysprep (every time, as I interpret them). Wouldn't hurt to try.

        Igor
-- sent from a mobile device, please excuse tyops and omissns

[Nigel Gutzmann]

Hi Igor,

I was digging into the event log and found the following messages on the instance not reporting metrics:

The StackdriverAgent service has started.

Got exception: [Error 2] The system cannot find the file specified

Using default endpoint: https://monitoring.googleapis.com/v3/projects/%s/collectdTimeSeries

Can't get w3svc data, disabling iis collection: winmgmts:.Win32_PerfFormattedData_W3SVC_WebService

Can't get sql general statistics, disabling sql server collection: winmgmts:.Win32_PerfFormattedData_MSSQLSERVER_SQLServerGeneralStatistics

The StackdriverAgent service has started.

Got exception: [Error 2] The system cannot find the file specified

Using default endpoint: https://monitoring.googleapis.com/v3/projects/%s/collectdTimeSeries

Can't get w3svc data, disabling iis collection: winmgmts:.Win32_PerfFormattedData_W3SVC_WebService

Can't get sql general statistics, disabling sql server collection: winmgmts:.Win32_PerfFormattedData_MSSQLSERVER_SQLServerGeneralStatistics

Exception sending metrics to Stackdriver: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Exception sending metrics to Stackdriver: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Exception sending metrics to Stackdriver: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Exception sending metrics to Stackdriver: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Exception sending metrics to Stackdriver: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Then the Exception sending metrics to Stackdriver is repeated many times. Any idea why that would be happening? On the instances that are correctly reporting metrics in the dashboard that exception is not happening (obviously that would be true, but I also double checked it by creating a different instance manually).

Cheers,

Nigel G.

[Igor Peshansky]

Hi, Nigel,

Looks like the agent is having trouble connecting to the API server. If you ping monitoring.googleapis.com from the affected machine, does it resolve the address? Does it resolve it to the same IP as on the unaffected machines? Are you able to send requests using the API explorer (on the right side of https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/create) from the affected machine?

It's possible that by creating the image you've ended up with a corrupted HOSTS or LMHOSTS file...

        Igor

[Nigel Gutzmann]

Hi Igor,

I finally figured it out, thanks to your suggestion to check the ping! Turns out the ping failed, so I checked pings to other websites and they all failed. The VMs did not have an external IP (which I did say in my original message, but maybe didn't put enough emphasis on), so they couldn't access the internet. After adding ephemeral external IPs to the instances, the metrics are being reported in the dashboard. 

Thanks for your help!
Nigel G.

[Igor Peshansky]

Hi, Nigel,

Ah, that would explain it. By default, only instances with external IPs can talk to Google services. You might want to look into https://cloud.google.com/compute/docs/private-google-access/configure-private-google-access.

        Igor
-- sent from a mobile device, please excuse tyops and omissns

[Nigel Gutzmann]

Thanks Igor!