Exception sending metrics to Stackdriver EventID 255

[Ricky Thacker]

I am in the process of configuring Stackdriver logging and monitoring on one of my GCP Compute instances.  The Agent install went smoothly and I am seeing event logs being reported in stackdriver. 

The problem is that since installing the agent my event logs has been filling with StackdriverMonitoring Warnings, about 1 per minute.  I have been unable to find what might be causing this.

I have uninstalled and reinstalled the logging agent, but the results is the same.

Can anyone give me an idea what what to try next to get this resolved?

On the server I've ran 

      Get-Service -Name StackdriverLogging  and Get-Service -Name Stackdrivermonitoring  and both services ARE running.

I've also been able to write an event log and it does show up both on the local server and in stackdriver logging.

New-EventLog   -LogName Application -Source "Test Source"

Write-EventLog -LogName Application -Source "Test Source" -EntryType Information -EventID 1 -Message "Testing 123 Testing."

I've also verified the access scopes are set correctly by doing the following under the VM instance detail page and confirmed the Stackdriver Logging API and Stackdriver Monitoring API both have "Write only" access.

Any help or direction would be greatly appreciated!

Here is the information that I pulled from one of the  (rapidly growing) number of warnings in the application event log.

Exception sending metrics to Stackdriver: error(0, 'RegisterEventSource/ReportEvent', 'No error message is available')

Event ID 255

System

Provider [ Name] StackdriverMonitoring

EventID 255 [ Qualifiers] 16384

Level 3

Task 0

Keywords 0x80000000000000

TimeCreated [ SystemTime] 2019-09-19T16:28:21.718971400Z

EventRecordID 80960

Channel Application

Computer <server name was here>

Security

EventData

Exception sending metrics to Stackdriver: error(0, 'RegisterEventSource/ReportEvent', 'No error message is available')

[Igor Peshansky]

Hi, Ricky,

This is a known minor issue with the Windows Monitoring agent. Did you follow the troubleshooting steps to verify that the agent is sending metrics? If so, the messages are harmless and can be ignored. If you'd rather not have them ingested into Stackdriver Logging, you can set up a log exclusion.

If the agent is not sending metrics, please let us know, and we can investigate further.

        Igor

P.S. FYI, this forum is moderated, and your messages only show up after moderator approval.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/db19d139-2217-4f31-a093-96b511ccc3d3%40googlegroups.com.

[Tiaan Swemmer]

Hi all,

I'd like to add to Ricky's post; I'm also experiencing this problem on one of our Windows VMs - Event ID 255. We have a limited number of Windows instances, and this is the only VM with this error.

The same troubleshooting/re-installation/API Access scopes checks were followed... it seems this Windows VM has trouble with permissions, although all instances are in the same organization and were configured for monitoring with weeks of each other.

* I tested with "timeseries.list method" after Igor's email and get a 200 response, but with empty results; {}.

Note the event 255 info is different:

- System 

  - Provider 

   [ Name]  StackdriverMonitoring 

 - EventID 255 

   [ Qualifiers]  16384 

   Level 3 

   Task 0 

   Keywords 0x80000000000000 

 - TimeCreated 

   [ SystemTime]  2019-09-20T11:46:33.187292900Z 

   EventRecordID 407530 

   Channel Application 

   Computer [server-name-here]

   Security 

- EventData 

   Error sending metrics to Stackdriver. Gateway API replied with a 403: { "error": { "code": 403, "message": "Permission denied (or the resource may not exist).", "status": "PERMISSION_DENIED" } }  

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[Ricky Thacker]

Igor,

Thanks so much for the suggestions.  That troubleshooting page you linked was different than the one I used but man of the steps were the same, but I had not attempted the Timeseries.list page tests.  I went though the steps and just as Tiaan also posted I receive "200" response but with no information.  

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[Ricky Thacker]

I set up a log exclusion in GCP as Igor suggested to ignore the "no error message available" messages I receive in my server event viewer.  It's still annoying that my log is full of these, but as long as I look at the log via stackdriver with the exclusion in place I don't have to see them.  I'm still not sure why I receive the 200 response with no information on my TimeSeries page test, but it appears my instance is reporting other things as expected. 

[Ricky Thacker]

Any assistance appreciated

I have set up stackdriver logging agent on an additional instance and am receiving a different Event 255 message this time around, now it's Unable to find the server at metadata.google.internal.

 I've been through all the same troubleshooting steps as before but have not been able to figure this one out.  The main difference with this instance is that it was not built in Compute Engine, rather it was virtualized from a physical server using the Cloud Endure process.

I initially suspected it may be permissions since this box was not built in GCP and wouldn't have the default service account configured.  However on my server when I open the Google Cloud SDK shell  and run "gcloud auth 

list"  the instance shows to be associated with one of the main accounts set up in IAM which has many more permissions assigned to it than the default service account does. 

I've also saw reference to needing to check the private-key credentials, I'm guessing this is a different set of permissions than the compute Engine credentials?  but I do not know how to check/verify/fix this.  The only steps I found was for running linux commands, but i'm not sure what I need to check for on windows, I began the process of creating a new service account, but later backtracked as I wasn't sure it was necessary or how to fully implement.    (linux commands were   sudo cat $GOOGLE_APPLICATION_CREDENTIALS and sudo cat /etc/google/auth/application_default_credentials.json)

Just as FYI the monitoring & logging agent installs on the instance with no issues, but no logs whatsoever show up in Stackdriver.  

I have confirmed the stackdriver monitoring and logging agents are running via powershell by running    Get-Service -Name StackdriverLogging    and   Get-Service -Name StackdriverMonitoring

I have performed the TimeSeries test as before, and get a "green" 200 message but no data except a couple brackets.

The application event log on my instance shows a lot of StackdriverMonitoring warnings stating the warning below.

-

System

- Provider [ Name] StackdriverMonitoring

- EventID 255 [ Qualifiers] 16384

Level 3

Task 0

Keywords 0x80000000000000

- TimeCreated [ SystemTime] 2019-09-23T22:50:58.000000000Z

EventRecordID 1842195

Channel Application

Computer <the instance name was here>

Security

-

EventData

Exception sending metrics to Stackdriver: ServerNotFoundError('Unable to find the server at metadata.google.internal',)

[Mary Koes]

+Joe Lynch as well...

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/b2ffcdd4-e30c-4b9b-bcca-e6756082579c%40googlegroups.com.

[Ricky Thacker]

Also an addendum to my last comment.  This Google cloud instance does not show any Scope or service account info under the instance settings whereas on my other insance (that was built in GCP and not imported, it does show these settings).  I do not see a way to edit/change/add anything here.

[Ricky Thacker]

Another update, i've been going through all the logs I can find as related to stackdriver on the instance.  I have found the the C:\Program Files (x86)\Stackdriver\LoggingAgent\fludentd.log contains the following message adding credence to my suspicion the my issue is something permission related.  an excerpt of the log stats :"AuthorizationError: Error code 404 trying to get security access token\nfrom Compute Engine metadata for the default service account. This\nmay be because the virtual machine instance does not have permission\nscopes specified"  

As mentioned prior my instance details does not show any information concerning a "Service Account" nor does it show the "Cloud API access scopes" even after stopping the instance.

per this link (https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances ) I should be able to stop the instance and edit the service account and scope details, but non are available to me to edit.

There is a section under "GCLOUD" on that page that show a command for setting a service.  Would this break anything if I attempted using those commands to set the service account and scopes?  and would it be okay to set it to the "default" service account, or should I create a new one?

The gcloud command referenced are:

gcloud compute instances set-service-account [INSTANCE_NAME] \
   [--service-account [SERVICE_ACCOUNT_EMAIL] | --no-service-account] \
   [--no-scopes | --scopes [SCOPES,...]]

where:

• [SERVICE_ACCOUNT_EMAIL] is the service account email you want to use. For example: my-s...@my-project-123.iam.gserviceaccount.com.
• [INSTANCE_NAME] is the name of the instance.
• [SCOPES] is a comma-separated list of full scope URIs, or scope aliases provided in the description for the --scopes flag. If you want to remove all scopes for the instance, use the --no-scopes flag instead.

Below i've attached the messaged found in my C:\Program Files (x86)\Stackdriver\LoggingAgent\fludentd.log logfile... The messages below are repeated over & over.

2019-09-24 14:25:15 -0500 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=3 records=364 error_class=GRPC::Unavailable error="14:Deadline Exceeded"

  2019-09-24 14:25:15 -0500 [error]: #0 suppressed same stacktrace

2019-09-24 14:25:34 -0500 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-09-24 14:25:35 -0500 chunk="59351381ad52fe59fae3a809027adc5e" error_class=GRPC::Unavailable error="14:Getting metadata from plugin failed with error: #<Signet::AuthorizationError: Error code 404 trying to get security access token\nfrom Compute Engine metadata for the default service account. This\nmay be because the virtual machine instance does not have permission\nscopes specified.\n>"

  2019-09-24 14:25:34 -0500 [warn]: #0 suppressed same stacktrace

2019-09-24 14:26:18 -0500 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-09-24 14:25:35 -0500 chunk="59350f2e9c6445ee5bb74170f46203f5" error_class=GRPC::Unavailable error="14:Deadline Exceeded"

  2019-09-24 14:26:18 -0500 [warn]: #0 suppressed same stacktrace

[Ricky Thacker]

Okay, I used the command to manually set the service account via the gcloud command line, the command made the "service account" and "scope" settings finally show up under the instance details (they were previously missing), for some reason I still had to manually edit them again in the instance deatuks (perhaps I had mistyped something in my command). Once this was completed (and another reboot later) All my server logs began showing up in StackDriver.  

Unfortunately, even though logging is now working, I am still getting the Exception sending metrics to Stackdriver: ServerNotFoundError('Unable to find the server at metadata.google.internal') error messages in my Instance event log and these are ALSO showing up in my stackdriver logs.  Is this another one of those "known issues" that Igor Peshansky reference?  Are these safe to ignore and/or filter out via an exception?  Better yet, is there a way to resolve this error so It's not filling up my logs?

Thanks!

gcloud compute instances set-service-account [INSTANCE_NAME] -service-account [SERVICE_ACCOUNT_EMAIL] --scopes [SCOPES,...]

On Tuesday, September 24, 2019 at 3:01:05 PM UTC-5, Ricky Thacker wrote:

Another update, i've been going through all the logs I can find as related to stackdriver on the instance.  I have found the the C:\Program Files (x86)\Stackdriver\LoggingAgent\fludentd.log contains the following message adding credence to my suspicion the my issue is something permission related.  an excerpt of the log stats :"AuthorizationError: Error code 404 trying to get security access token\nfrom Compute Engine metadata for the default service account. This\nmay be because the virtual machine instance does not have permission\nscopes specified"  

As mentioned prior my instance details does not show any information concerning a "Service Account" nor does it show the "Cloud API access scopes" even after stopping the instance.

per this link (https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances ) I should be able to stop the instance and edit the service account and scope details, but non are available to me to edit.

There is a section under "GCLOUD" on that page that show a command for setting a service.  Would this break anything if I attempted using those commands to set the service account and scopes?  and would it be okay to set it to the "default" service account, or should I create a new one?

The gcloud command referenced are:

gcloud compute instances set-service-account [INSTANCE_NAME] \
   [--service-account [SERVICE_ACCOUNT_EMAIL] | --no-service-account] \
   [--no-scopes | --scopes [SCOPES,...]]

where:

• [SERVICE_ACCOUNT_EMAIL] is the service account email you want to use. For example: my-sa-123@my-project-123.iam.gserviceaccount.com.