Embedding arbitrary JSON in a message

[John Lilley]

Disclaimer: I am totally new to protobuf, engaged in an exploratory POC.  Please forgive dumb questions :-)

We are looking at migrating an existing, JSON-based protocol in which hand-coded C++ is written to perform serdes between objects and JSON. We want to replace the hand-coding with an automated approach that can be shared between C++ and Java.  However, a stumbling block I see is that some messages have an arbitrary field full of JSON like:

{

   "name":"john", 

   "address":"123 main st",

   "attributes":{ any JSON can go here }

}

While I realize that we could stringify the JSON, this breaks our published API.  Is there any way I can use protobuf to perform serdes between message like this and some struct like:

{

   string name;

   string address;

   json attributes;

}

I'm even OK if the internal data is stringified JSON:

{

   string name;

   string address;

   string attributes;

}

So long as the exchanged JSON isn't stringified.  In other words, this is bad:

{

   "name":"john", 

   "address":"123 main st",

   "attributes":"{ \"attr1\":\"value1\", \"attr2\":[\"elem1\", \"elem2\"] }"

}

It needs to be exchanged like

{

   "name":"john", 

   "address":"123 main st",

   "attributes":{ "attr1":"value1", "attr2":["elem1", "elem2"] }"

}

Is this possible?

Thanks

john

[Josh Humphries]

Hi, John,

Take a look at the well-known type google.protobuf.Struct. It is basically a JSON value, modeled as a proto. It's JSON representation is exactly what you want, too:

https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L52

----
Josh Humphries
jh...@bluegosling.com

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Josh Humphries]

Oops, I meant to point you to google.protobuf.Value: https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L63

It can represent any kind of JSON value. The Struct type is what is used to represent JSON objects (there is also ListValue, for arrays, as well as support for JS primitive types and null).

----
Josh Humphries
jh...@bluegosling.com

[Siddharth Kherada]

Can I please get an example of how to do this is any language with proto file?

I desperately need help with this. I am stuck for a week now on this issue.

Thanks,

Sid

[Jimit Modi]

Hello @Siddharth, @John,

Did you guys found any solution around it. We are also stuck at very same thing.

----
Josh Humphries
jh...@bluegosling.com

----
Josh Humphries
jh...@bluegosling.com

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

[Josh Humphries]

If you are using JavaScript, there may not be a good solution. Ironically, the JavaScript implementation of protos does not support the canonical JSON representation for protobuf types (which would be needed to convert arbitrary JSON into a google.protobuf.Value).

Most other languages, though, should work fine. Here's a sketch in Go:

example.proto

syntax = "proto3";

import "google/protobuf/struct.proto";

message SomeMessage {

    google.protobuf.Value arbitrary_json = 1;

}

example.go

package example

import "github.com/golang/protobuf/ptypes/struct"

import "github.com/golang/protobuf/jsonpb"

func example(json string) (*SomeMessage, error) {

    msg := &SomeMessage{ArbitraryJson: &structpb.Value{}}

    jsm := jsonpb.Marshaler{}

    if err := jsm.Unmarshal(json, msg.ArbitraryJson); err != nil {

        return nil, err

    }

    return msg, nil

}

You could also use the "encoding/json" package to then convert any kind of Go value into a google.protobuf.Value: you just have to marshal to JSON first, and from there into the protobuf. While using JSON as intermediate format is not efficient, you do get the benefit that a google.protobuf.Value created this way is guaranteed to be well-formed, whereas just using a string or bytes field could be garbage that is not proper JSON.

----
Josh Humphries
jh...@bluegosling.com