Why does unmarshalling this API response return an unexpected EOF?

[ivanche...@gmail.com]

Hi all,

I'm creating a microservice in Go, using protocol buffers and gRPC. It interacts with a third-party API (Snooth) and I'm trying to unmarshal the JSON response into a protobuf struct I've created, using the proto package.

Unmarshalling returns an unexpected EOF error.

I've summed up the details in full on this question at Stack Overflow: https://stackoverflow.com/questions/50314476/why-does-unmarshalling-this-api-response-return-an-unexpected-eof

In addition to the information posted there, I've also tried using strconv.Unquote before unmarshalling in line with the idea that the escaped characters in the API response are causing a double-encode. But this didn't work either.

Any help would be greatly appreciated as I've been stuck on this for a while now and feel there's something simple I'm missing.

Thanks.

[Tamás Gulácsi]

It DOES parse with then encoding/json package: https://play.golang.org/p/IQzMm2tDI7w

The protoc-generated code's Unmarshal parses a Protocol Buffers encoded byte stream, NOT JSON!

[Luke IC]

I've tried that, but I need to decode the JSON into a struct generated from a .proto. Updating the code to use json.Unmarshal results in the wines array from the response not decoding correctly, here is what I see when printing the struct that I have unmarshalled into:

meta:<results:1489449 returned:15 status:1 > wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<> wines:<>

Note the repeated values are empty. The relevant source code can be viewed here: https://github.com/lukeic/vinogo/tree/feature/snooth/snooth. Help figuring this out would be much appreciated as I've spent ages on it now.

[alex....@gmail.com]

You need to use the go protobuf json decoder 

https://github.com/golang/protobuf/tree/master/jsonpb

[Tamás Gulácsi]

Please create a test case (snooth_test.go) which illustrates your problem: has the input as const string, tries to Unmarshal into your chosen struct, probably fails with error.

All without making HTTP requests!

[Luke IC]

Thanks a lot, I went back and gave jsonpb a go and finally got it unmarshalling as desired.

To do so though I had to unmarshal and marshal the response with the regular json library first, in order to get around some escaped values in the response (I was receiving a 'bad value in Struct' error:

    resJson, err := ioutil.ReadAll(res.Body)

    j := make(map[string]interface{})

    jbytes, err := json.Marshal(j)

    result := &pb.Response{}
    r := strings.NewReader(string(jbytes))
    if err := jsonpb.Unmarshal(r, result); err != nil {
        panic(err)
    }

Is this fine to do, or is it super inefficient and will cause problems?

[alex....@gmail.com]

In the long term I would investigate what's wrong with the json and if possible fix it at the source.

Also use this as a reference on what gets mapped to what https://developers.google.com/protocol-buffers/docs/proto3#json

[Luke IC]

Thanks again for the help. Unfortunately it's a third-party API, so I won't be able to fix it myself. The full error message is as follows:

bad value in StructValue for key "image": unrecognized type for Value "\"https:\\/\\/ei.isnooth.com\\/multimedia\\/0\\/2\\/8\\/image_787698_square.jpeg\""

I assume the cause for this error is the escaped quotes?

[alex....@gmail.com]

I do not see escaped quotes on your stackoverflow post, can you post the raw json exactly as received?

[Luke IC]

Here's a gist: https://gist.github.com/lukeic/7350d07cac28b2809e9930802251844f

This is the JSON I see output from (errors removed for brevity):

body, _ := ioutil.ReadAll(res.Body)
fmt.Println(string(body))

[Tamás Gulácsi]

2018. május 14., hétfő 23:03:46 UTC+2 időpontban Luke IC a következőt írta:

Thanks a lot, I went back and gave jsonpb a go and finally got it unmarshalling as desired.

To do so though I had to unmarshal and marshal the response with the regular json library first, in order to get around some escaped values in the response (I was receiving a 'bad value in Struct' error:

This seems an error of jsonpb!

Till they fix it (after you've reported it), just replace `\/` with `/`: https://play.golang.org/p/oFwsW8eX8aJ

[Luke IC]

Cool, thanks. I've created a new issue on the `jsonpb` repo as well.