UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte

[Sujeet Gupta]

Hi All

I am creating route and trying to process the event in Cloud Function. I am constantly getting json parsing error as below. Looking forward for your quick response on it.

Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 383, in run_background_function _function_handler.invoke_user_function(event_object) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 217, in invoke_user_function return call_user_function(request_or_event) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 214, in call_user_function event_context.Context(**request_or_event.context)) File "/user_code/main.py", line 17, in run event = json.loads(base64.b64decode(event['data']).decode('utf-8')) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte

payload:

Event: {'insertId': '-naoi1tdc862', 'logName': 'projects/bold-hallway-258919/logs/cloudaudit.googleapis.com%2Factivity', 'operation': {'id': 'operation-1578614395820-59bbdce4cdb98-efcc6fb2-cd3fdc8c', 'last': True, 'producer': 'compute.googleapis.com'}, 'protoPayload': {'@type': 'type.googleapis.com/google.cloud.audit.AuditLog', 'authenticationInfo': {'principalEmail': 'xxx.x@gmail.com'}, 'methodName': 'v1.compute.routes.insert', 'request': {'@type': 'type.googleapis.com/compute.routes.insert'}, 'requestMetadata': {'callerIp': '', 'callerSuppliedUserAgent': 'google-cloud-sdk gcloud/273.0.0 command/gcloud.compute.routes.create invocation-id/f794c079d1a64dd69f017bcb58b06f5d environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 18.7.0),gzip(gfe)'}, 'resourceName': 'projects/bold-hallway-258919/global/routes/myroutetencommandline', 'serviceName': 'compute.googleapis.com'}, 'receiveTimestamp': '2020-01-10T00:00:00.383651222Z', 'resource': {'labels': {'project_id': 'bold-hallway-258919', 'route_id': '123658669026574483'}, 'type': 'gce_route'}, 'severity': 'NOTICE', 'timestamp': '2020-01-09T23:59:59.675Z'}

Thank you,

Sujeet

4252864465

[Ahmad P - Cloud Platform Support]

Hi Sujeet,

This is related to Python. 

You can find different solutions in this stackoverflow case.

[kapilt]

its actually specific to python cloud functions on gcp api audit logs via pub/sub. there seems to be some variability with how gcp is sending events, in most cases base64 json bits,  in some cases sending non utf8 blobs, perhaps some protobuf binary.

[Ahmad P - Cloud Platform Support]

As per the log provided, this is an issue with the SDK. 

You use SDK 273 on Mac OS and I do not have access to a mac machine, but it could be that the issue is affecting all OS's. I can advise to upgrade to the latest SDK as it may resolve the issue.

As Google Groups is reserved for general product discussions, it would be best to open a Public Issue tracker include all reproduction steps as to reproduce in our environment such as the exact command used, then the proper support channel can assist you further.

[kapilt]

another user was able to point out that this was a user error in that they were getting unrelated messages into their pub/sub topic that were in a different format (gzip'd) than the stack driver audit log messages,

Thanks,

Kapil

[Ahmad P - Cloud Platform Support]

Unfortunately, it needs more details to find out the root cause. I opened a case on Public Issue tracker to investigate more.

Please follow this Public Issue tracker to resolve your issue.