Intermittent errors from ESP server

[Akshay Bapat]

Hi Cloud Endpoints folks,

I am not sure if this is the right forum to raise this (please point to one if not) but here goes.

I am in the Nest Machine Perception team - we have a grpc service called snappy running in GCP. For some of the calls in this service, we also have configuration that enables these to be called using cloud endpoints' http calls.

On each box that runs this service, we also run an esp server that can serve the http requests.

The problem is that every few requests or so, when I make a request such as:

response = requests.get('http://localhost:8080/v1/general_motion_markers/666fd49cb2e34d0b9691c5c53a3243e3?start_timestamp_us=1515786743000000'),

the esp service throws a response with content saying 'Json message was not parsed'.

Here is a code block elaborating this:

>>> for i in range(0, 100):

...   print i

...   response = requests.get('http://localhost:8080/v1/general_motion_markers/666fd49cb2e34d0b9691c5c53a3243e3?start_timestamp_us=1515786743000000')

...   if 'entirety' in response.content:

...     print response.content

...     break

... 

0

1

{

 "code": 3,

 "message": "Nested protocol message not parsed in its entirety.",

 "details": [

  {

   "@type": "type.googleapis.com/google.rpc.DebugInfo",

   "stackEntries": [],

   "detail": "internal"

  }

 ]

}

Note how the request succeeded on the first iteration and failed on the second one.

I have checked that the grpc service (snappy) had a consistent response during all of the requests so I suspect this has something to do with the protobuf to json transcoding but I don't really have a good grasp of what the problem might be.

The esp server logs have this message on the failed request:

127.0.0.1 - - [16/Jan/2018:23:24:33 +0000] "GET /v1/general_motion_markers/666fd49cb2e34d0b9691c5c53a3243e3?start_timestamp_us=1515786743000000 HTTP/1.1" 200 405058 "-" "python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-514.10.2.el7.x86_64"

[libprotobuf WARNING external/protobuf_git/src/google/protobuf/util/internal/json_objectwriter.cc:53] JsonObjectWriter was not fully closed.

Would appreciate any help in resolving this.

Thanks,

Akshay