How to represent variable JSON object data with Proto definition

[Bojan D]

Newbie question... If I have the following sample "User" JSON data:

{

  firstName: 'Bob'

  lastName: 'Smith',

  email: 'b...@gmail.com',

  metadata: {

    // a plain JS object that 

    // - will always exist and be at least an empty {} object

    // - could potentially contain any number of properties and values, depending on specific "user"

  }

}

How do I represent the metadata property within proto definition? I've tried

message User {

  message Metadata {}

  string email = 1;

  string firstName = 2;

  string lastName = 3;

  Metadata metadata = 4;

}

or using `Any`

message User {

  string email = 1;

  string firstName = 2;

  string lastName = 3;

  google.protobuf.Any details = 4;

}

but without success. Working within Node.js if relevant. Any help is much appreciated. Thanks!

[Tim Kientzle]

On Nov 15, 2016, at 9:15 AM, Bojan D <dbo...@gmail.com> wrote:

Newbie question... If I have the following sample "User" JSON data:

{

  firstName: 'Bob'

  lastName: 'Smith',

  email: 'b...@gmail.com',

  metadata: {

    // a plain JS object that 

    // - will always exist and be at least an empty {} object

    // - could potentially contain any number of properties and values, depending on specific "user"

  }

}

How do I represent the metadata property within proto definition?

You want to use the well-known types “Struct” or “Value”, which are specifically designed to support ad hoc JSON parsing.  “Struct” supports parsing any valid JSON object structure, “Value” can parse any valid JSON:

message User {

  string email = 1;

  string firstName = 2;

  string lastName = 3;

  google.protobuf.Struct metadata = 4;

}

Tim

[Bojan D]

Thanks for the answer. Somehow I missed the Protocol Buffers Well-Known Types page in the docs / protobuf website.

I still have issue creating an instance of the message. For example if I have a plain object:

{

  firstName: 'Bob'

  lastName: 'Smith',

  email: 'b...@gmail.com',

  metadata: {

    foo: 'bar',

    active: true

  }

}

And I try to create an instance of the message (to be sent via grpc) I get error

.google.protobuf.Struct#foo is not a field: undefined

Same thing if I use google.protobuf.Value. I've tried numerous ways of doing this but can't seem to accomplish it. I must be missing something. Anyone have any ideas.

Thanks for the help again.

Bojan

[Tim Kientzle]

I think we need more details:

   Are you using C++ or Java or Python or Ruby or C# or some other language?

   What version of protoc?

   Can you show the code you’re using to create the message?

Tim

-- 
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Bojan D]

Sure sorry...

I am working with node.js and trying to use grpc. Reproducible steps:

Install node 6.9.1

mkdir grpctest

cd grpctest

npm install grpc

user.proto:

import "google/protobuf/struct.proto";

syntax = "proto3";

package user;

message User {

  string email = 1;

  string firstName = 2;

  string lastName = 3;

  google.protobuf.Value metadata = 4;

}

message GetUserRequest {

  string email = 1;

}

service UserService {

  rpc GetUser(GetUserRequest) returns (User);

}

server.js:

const path = require('path')

const grpc = require('grpc')

const PROTO_PATH = path.resolve(__dirname, './user.proto')

const HOSTPORT = '0.0.0.0:50051'

const UserService = grpc.load(PROTO_PATH).user.UserService

const data = {

  "b...@gmail.com": {

    "email": "b...@gmail.com",

    "firstName": "Bob",

    "lastName": "Smith",

    "metadata": {

      "foo": "bar",

      "active": true

    }

  },

  "ja...@gmail.com": {

    "email": "ja...@gmail.com",

    "firstName": "Jane",

    "lastName": "Smith"

  }

}

function getUser(call, callback) {

  const user = data[call.request.email]

  if (!user) {

    return callback(new Error('User Not Found'))

  }

  return callback(null, user)

}

function main() {

  const server = new grpc.Server()

  server.addProtoService(UserService.service, { getUser })

  server.bind(HOSTPORT, grpc.ServerCredentials.createInsecure())

  server.start()

  console.log(`User service running @ ${HOSTPORT}`)

}

main()

client.js

const path = require('path')

const grpc = require('grpc')

const PROTO_PATH = path.resolve(__dirname, './user.proto')

const HOSTPORT = '0.0.0.0:50051'

const UserService = grpc.load(PROTO_PATH).user.UserService

const client = new UserService(HOSTPORT, grpc.credentials.createInsecure())

client.getUser({ email: 'ja...@gmail.com' }, (err, user) => {

  console.log(user)

  process.exit()

})

Run the server using command:

node server.js

User service running @ 0.0.0.0:50051

Run the client:

node client.js

{ email: 'ja...@gmail.com',

  firstName: 'Jane',

  lastName: 'Smith',

  metadata: null }

change the email in client request to 'b...@gmail.com'

/Users/bojand/dev/nodejs/grpctest/node_modules/protobufjs/dist/protobuf.js:2472

                            throw Error(this+"#"+keyOrObj+" is not a field: undefined");

                            ^

Error: .google.protobuf.Value#foo is not a field: undefined

    at Error (native)

...

Within the server part once we load proto we do have a constructor for the User class, and I've tried playing around with trying to create an instance of that in different ways and it crashes when trying to create that instance. I've also tried to encode from stringified JSON and still fails. When there is "metadata" it crashes. I think I am doing something dumb but I am not sure what.

Thanks.

[Bojan D]

Here's a simpler server.js that doesn't require a client.js or deals with any of the grpc stuff really.

const path = require('path')

const grpc = require('grpc')

const PROTO_PATH = path.resolve(__dirname, './user.proto')

const HOSTPORT = '0.0.0.0:50051'

const UserProto = grpc.load(PROTO_PATH);

const User = UserProto.User

const data = {

  "b...@gmail.com": {

    "email": "b...@gmail.com",

    "firstName": "Bob",

    "lastName": "Smith",

    "metadata": {

      "foo": "bar",

      "active": true

    }

  },

  "ja...@gmail.com": {

    "email": "ja...@gmail.com",

    "firstName": "Jane",

    "lastName": "Smith"

  }

}

function main() {

  const u = new User(data['ja...@gmail.com'])

  console.log(u)

}

main()

change the email to 'b...@gmail.com' and it would crash.

[Adam Cozzette]

I think the problem you're running into is that unfortunately we don't currently support proto3 JSON for Javascript. Proto3 JSON is the JSON-based format that understands the special representations for well-known types such as Struct and Any.

Actually gRPC's dynamic codegen uses protobuf.js, which is a totally separate implementation from the main one we maintain at https://github.com/google/protobuf. But neither implementation has support for proto3 JSON, so if you're using Node.js then Struct, Value, and Any won't be of much help for now.

Could you describe a bit more about what the metadata field needs to contain? Does it really need to support completely arbitrary JSON or would it possibly be something where there are just a handful of different basic schemas you expect? If the field can be arbitrary JSON then perhaps the best solution would be to just use a bytes field and store the raw JSON blob inside it.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.

[Bojan D]

Thanks for the answer. I thought I was doing something wrong. I was actually looking into protobuf.js thinking I could use that alongside grpc (it was able to handle Struct/Value/Any types); so it's good to know that grpc indeed uses that. It seems there are a few similar issues on GH already, and the library might be getting a rewrite so maybe it will be solved at some point in the future. I'll look further into it. 

This isn't really a real world scenario for me at the moment... I am just doing some experimenting with grpc and wanted to try to handle variable JSON data. It's not really a stretch if for example we are working with schemaless / document based databases and depending on app context where you may want to store some variable metadata for objects. I agree the workround is to do manual encode / decode .

Thanks for the help, it's appreciated!

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.