Capnp <-> Json codec

[Derrick Johnson]

I noticed this in the Roadmap, has anyone started on this yet?  If so, do you want some help?

If not, I'm wondering where the best place to put this would be in the codebase.

Perhaps a new file capnp/serialize-json.h:

class JsonMessageReader : public InputStreamMessageReader

{};

void writeJsonMessage(kj::OutputStream &output, MessageBuilder &builder)

{}

[Kenton Varda]

Hi Derrick,

I'm not aware of anyone working on this currently. It would be great if you wanted to work on this!

The interface should probably be something like:

    DynamicValue::Builder parseJson(kj::StringPtr text, Type type, Orphanage orphanage);

    // Parse `text` as a JSON representation of the given type, allocating objects into the

    // MessageBuilder backing `orphanage`.

    kj::StringTree serializeJson(DynamicValue::Reader reader);

Extending MessageReader only works when the input is actually Cap'n Proto format, and using MessageBuilder as a parameter to the serializer doesn't give you the type information needed.

You could still use InputStream/OutputStream instead of strings but I generally feel that this adds a lot of complication with no significant advantage.

We'll want to make this a separate library so that it doesn't add bloat for people who don't want it. It could make sense for it to be a separate project as well, particularly if you want to use some existing JSON parsing library. OTOH, if you wrote a parser using kj/parse then I could imagine this living in the Cap'n Proto repo. I would just call it `capnp/json.h`, and build a library `capnp-json.so`.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.

[JF Gauvin]

With Pycapnp it's easy to do this:

# capnp to JSON
json.dumps(obj=capnpData.to_dict(ordered=True))

# JSON to capnp
dict = json.loads(jsonData)   
return schema.new_message(**dict)

[Kenton Varda]

Ah, yes. Similarly, node-capnp can pretty trivially be composed with JSON.parse / JSON.stringify, since it already has facilities to convert to/from Javascript native objects.

-Kenton

[phyto...@gmail.com]

I'm new to capnp - sorry if this is basic. So with thrift their code generation solution seems to be - have users tie into the c++ infrastructure and the generators are built into one big project. From the capnp *Other Languages* page it sounds like the plugin approach is quite different - you pass any *capnp capable process* the desired schema to generate code for. 

Is it correct to say: the format of that data conforms to a *schema.capnp* instance?

If so, this is a great improvement over thrift - since C++ with a large/heavy setup requirement is not necessary to write generators. Plus it is kind of dog-fooding.

But if you bumped the priority of transcoding capnp to json, couldn't you add an additional flag to the compile step to *optionally* transcode to json first and pass that to the plugin. If this were done then there is no dependency on capnp generators for capnp. Of course they would need to understand it to generate code for it. But with this approach couldn't people wanting to start in on support for their own language do it without delay (assuming language supports json) and without the bootstrap issues?

[Kenton Varda]

On Thu, May 14, 2015 at 5:41 AM, <phyto...@gmail.com> wrote:

I'm new to capnp - sorry if this is basic. So with thrift their code generation solution seems to be - have users tie into the c++ infrastructure and the generators are built into one big project. From the capnp *Other Languages* page it sounds like the plugin approach is quite different - you pass any *capnp capable process* the desired schema to generate code for. 

Is it correct to say: the format of that data conforms to a *schema.capnp* instance?

If so, this is a great improvement over thrift - since C++ with a large/heavy setup requirement is not necessary to write generators. Plus it is kind of dog-fooding.

Yep. This is an idea I originally implemented in the protobuf compiler, and then brought over to Cap'n Proto. I'm not sure why Thrift hasn't copied it yet. :)

But if you bumped the priority of transcoding capnp to json, couldn't you add an additional flag to the compile step to *optionally* transcode to json first and pass that to the plugin. If this were done then there is no dependency on capnp generators for capnp. Of course they would need to understand it to generate code for it. But with this approach couldn't people wanting to start in on support for their own language do it without delay (assuming language supports json) and without the bootstrap issues?

Maybe. How would a plugin advertise that it wants JSON input? I'd hate for this to be a flag that application developers need to know about.

-Kenton

[Farz Hemmati]

To advertise expected input format, a naive way would be to output "json\n" onto stdout, but since current plugins might abuse stdout in various ways (capnpc outputs its result to stdout), this can run into a variety of problems.

Instead, I propose writing the JSON into a Data field of the expected CodeGeneratorRequest along with a magic 64 bit number and a size field before. This would allow a capnp-understanding plugin to simply ignore the JSON as it's just some field it doesn't care about. It would also allow a json-only plugin to skip stdin until it sees the magic number, then read the expected number of bytes.

The result for a json-plugin would be like so:

|-- ignore --| {MAGIC NUMBER} {8 byte size} {size bytes of JSON} |-- ignore --|

For a capnp-plugin, it would be a single Data field that it would 'parse' by reading the tag and length bytes and skipping the length to reach the rest of the message after the JSON/Magic number.

This would keep it backwards and forward compatible. The only downside is that very large schemas now take up more memory as the JSON encoding has to be created and sent to each plugin.

What do you think?

[Daniel Davidson]

Farz,

If I understood Kenton, the question is how does the plugin say it wants json, as opposed to how does the compiler tell the plugin what it is getting. Sounds like the concern is to keep the tool simple for users. To keep it simple, assume a plugin supports one input encoding. Initially capnp input (status quo) or json (new) with potential for others in future. No need to assume/require one plugin supports multiple encodings. 

Current:
capnp compile -ocapnp ./c++/src/capnp/schema.capnp

So, no change to existing stuff. But, add a new flag -p<plugin>[:<dir>] which is more general and informative than -o<lang>[:<dir>]. Define a capnp plugin naming convention - if your plugin starts with json_to, then you get json. The compiler just encodes the shcema to json before emitting and passing to the plugin.

capnp compile -pjson_to_dart schema.capnp

capnp compile -pjson_to_swift schema.capnp

capnp compile -pcapnp schema.capnp

The last one is then equivalent to the original with no special encoding specified - so it is capnp. Then from an application developer's perspective they are just choosing their favorite plugin for their language. There could even be competing implementations for a target language: -pjson_to_dart_google... -pjson_to_dart_zippy.

Also you might want to think about allowing the compiler to support pass through args to the plugin:

capnp compile -pjson_to_dart --plugin-flag=include-mongo-stubs schema.capnp

Thanks

Dan

[Kenton Varda]

Hmm, I think both of these are problematic:

- Embedding the JSON in every CodeGeneratorRequest seems wasteful when most plugins don't need the JSON. I don't like saying "well, who cares, it's just build time" as this waste tends to add up and iteration speed maters when developing.

- Requiring a different user-visible command-line is exposing implementation details and means that it will be hard for a plugin to switch to capnp format later on.

Here's a different idea: we build it as an adapter. The capnp tool itself remains unchanged, but we offer another binary called capnp-json-adapter. This binary reads in a CodeGeneratorRequest in capnp format, converts it to JSON, then execs some other binary and writes the JSON to the new program's stdin. You use the adapter by symlinking it as your plugin. So if you are making a plugin for, say, Scala, you symlink capnpc-scala to capnp-json-adapter. The adapter looks at its argv[0] to find out what plugin you were trying to execute, and then executes capnp-json-<name>, e.g. capnp-json-scala in this case.

This whole thing -- including the json codec -- can in theory be implemented as a separate project, avoiding bloat and simplifying project management (no need to wait for me to review changes).

Thoughts?

-Kenton

[Yaron Minsky]

That seems nice. If nothing else, it would make bootstrapping a capnp
compiler easier and more pleasant.

y

[marko....@gmail.com]

Hello everybody,

I looked in the docs, but couldn't find any information: was json de/encoding implemented? If not, could I help with the implementation?

Cheers,
Marko

[Kenton Varda]

Hi Marko,

It has been implemented, but the code hasn't been in a release yet. See:

https://github.com/sandstorm-io/capnproto/blob/master/c%2B%2B/src/capnp/compat/json.h

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.