Beginner q: Capn' Proto schema --> JSON object construction & validation?

[aseem....@gmail.com]

Hi there,

I'm new to Capn' Proto (impressive work!), and binary protocols generally, coming from the world of web dev and JS.

We define various models in our system with Capn' Proto schemas. We'd like to keep these nicely typed definitions, but one of our use cases now calls for working with simple JSON objects instead of Capn' Proto messages.

E.g. for the Person example on <https://capnproto.org/language.html>, the equivalent JSON object might be:

{
  "name": "Alice Smith",
  "birthdate": {"year": 1980, "month": 6, "day": 15},
  "email": "al...@example.com",
  "phones": [{"number": "123-456-7890", "type": "mobile"}]
    // not sure yet exactly how we'll translate enums like Type
}

We ideally wouldn't duplicate all the schema definitions in e.g. JSON Schema, and we'd ideally not write the code by hand to generate (from capnp schemas) and validate (against capnp schemas) JSON objects. (In either case, we'd have to manually keep something in sync with the capnp schemas.)

So ideally, there would be e.g. codegen tools for doing those things... but I don't think I'm finding any! E.g. the JS libs seem to be only about reading and writing capnp *messages*.

I'd guess this use case isn't new, so I'm very likely missing something obvious as a beginner. Please feel free to point me in any direction. =)

Thank you very much!

Aseem

[Kenton Varda]

Hi Aseem,

Some implementations of Cap'n Proto provide JSON transcoding. In particular:

- capnp-node actually converts everything to Javascript objects which can of course be JSON-ified.

- pycapnp I think has functions to convert to/from dicts which are easy to convert to/from JSON.

- The C++ library has JsonFormat in capnp/compat/json.h. Currently only capnp -> JSON is implemented, but https://github.com/sandstorm-io/capnproto/pull/258 adds JSON -> capnp. Note that this class hasn't made it into a release yet, so you'd have to work from github master, which normally I don't recommend unless you are Linux-exclusive.

I'm not sure about the other per-language implementations but if they don't have a JSON translator then I think it's reasonable to file as a feature request. :)

FWIW, enums are represented as string names. 64-bit integers are represented as decimal strings since Javascript numbers cannot represent 64-bit ints without precision loss. Voids become "null". I think everything else is encoded in "the obvious way".

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.

[aseem....@gmail.com]

Hi Kenton,

Thanks for the quick reply!

My understanding of all of those options are that they work with Cap'n Proto *messages*. Am I misunderstanding?

In this case, we aren't passing around capnp messages; we'd like to pass around JSON objects. So we don't need code that can convert JSON objects to/from capnp messages. We'd like code that can *construct* JSON objects, from capnp *schemas*, and validate them against those schemas.

Does that make sense? Please let me know if I'm unclear or missing something fundamental. =)

And thanks for the info on enums, 64-bit ints, and voids!

Aseem

[Aseem Kishore]

Possibly another avenue: are there any tools that can convert a Cap'n Proto schema to e.g. JSON Schema? (Then, we could use existing tools to construct & validate JSON objects from/against JSON schemas.)

I didn't find anything from a quick search here either, but I again may be overlooking something simple.

Thanks!

[Kenton Varda]

Hi Aseem,

Personally, even when I'm working with pure JSON data which will never be transmitted as Cap'n Proto, my preference is to build the message as a Cap'n Proto message object which I convert to JSON, and to parse the object into another Cap'n Proto object on the receiving end, because:

1) This has the side effect of validating the message against the schema.

2) It's much more convenient to access the message structure using the type-safe Cap'n Proto generated code than it is to access a JSON AST.

I actually have never encountered anyone using JSON schemas. I knew a JSON schema format exists as a standard, but the one time I looked at it, it appeared ridiculously verbose. It seems like most people prefer to use schema-by-convention when working with JSON.

With that said, I'm sure it would not be too hard to write some code which takes a Cap'n Proto schema representation (per schema.capnp) and converts it to JSON schema format. But no, I don't know of any such tools.

-Kenton

[JF Gauvin]

Hi,

I was looking at this some months ago and there was no tool. I was thinking to do it in Python. With the recent work related to JSON in Cap'n Proto, it's maybe easier to do this in C++.
I think that having a JSON schema can be useful, because there are libraries automatically generating the web user interface.

It could be simple to do a basic cap'n proto schema to JSON schema tool. But, if you want to do more advanced things, like using minimum/maximum, title, reference, units, format, etc, it's harder because cap'n proto schema don't have all those things. You need to add annotations. I was thinking to have a schema to handle them. For example:

# String formats
const datetime :Text = "date-time"; # RFC 3339, section 5.6
const email    :Text = "email";     # RFC 5322, section 3.4.1
const hostname :Text = "hostname";  # RFC 1034, section 3.1
const ipv4     :Text = "ipv4";      # RFC 2673, section 3.2
const ipv6     :Text = "ipv6";      # RFC 2373, section 2.2
const uri      :Text = "uri";       # RFC 3986


# "units":
annotation units @0x9caa3354d2f1c633 (field) :Text;
const unitsAnnotationId :UInt64 = 0x9caa3354d2f1c633;

# "hidden": true
#annotation hidden (field) :Void;
#const hiddenAnnotationId :UInt64 = ;

#annotation rfc3339 @0xb6d6d8943d576768 (*) :Void;
#const rfc3339AnnotationId :UInt64 = 0xb6d6d8943d576768;

# readOnly=true
annotation readOnly @0xd437766f1b485772 (field, struct) :Void;
const readOnlyAnnotationId :UInt64 = 0xd437766f1b485772;

# "uniqueItems": true
annotation uniqueItems @0x9347cdc70f0ca3ac (field) :Void;
const uniqueItemsAnnotationId :UInt64 = 0x9347cdc70f0ca3ac;

# "pattern":
annotation pattern @0x80e125fe157241b8 (field) :Text;
const patternAnnotationId :UInt64 = 0x80e125fe157241b8;

# "format": "date-time", "email", "hostname", "ipv4", "ipv6", "uri"
annotation format @0xa6b3c8b9eaa3b416 (field) :Text;
const formatAnnotationId :UInt64 = 0xa6b3c8b9eaa3b416;

# "minLength":
annotation minLengthUint16 @0xe22fc2a6bab37f0c (field) :UInt16;
const minLengthUint16AnnotationId :UInt64 = 0xe22fc2a6bab37f0c;

# "maxLength:
annotation maxLengthUint16 @0xe47fc96550c8baa7 (field) :UInt16;
const maxLengthUint16AnnotationId :UInt64 = 0xe47fc96550c8baa7;

# "multipleOfFloat":
annotation multipleOfInt32 @0xdd886092a32b871f (field) :Float32;
const multipleOfInt32AnnotationId :UInt64 = 0xdd886092a32b871f;

#annotation multipleOfFloat32  (field) :Float32;
#const multipleOfFloat32AnnotationId :UInt64 = ;

annotation multipleOfFloat64 @0x857d1a06a23c76f9 (field) :Float64;
const multipleOfFloat64AnnotationId :UInt64 = 0x857d1a06a23c76f9;

# "exclusiveMinimum": true
annotation exclusiveMinimum @0x91e6230ba7bf0bf6 (field): Void;
const exclusiveMinimumAnnotationId :UInt64 = 0x91e6230ba7bf0bf6;

# "exclusiveMaximum": true
annotation exclusiveMaximum @0xfe1569266c25cd23 (field): Void;
const exclusiveMaximumAnnotationId :UInt64 = 0xfe1569266c25cd23;

# "minimum":
#annotation minimumInt16 (field) :Int16;
#const minimumInt16AnnotationId :UInt64 = ;

annotation minimumUint16 @0xf3a306fd9cb560cc (field) :UInt16;
const minimumUint16AnnotationId :UInt64 = 0xf3a306fd9cb560cc;

annotation minimumInt32 @0xe3a4c77f928f11f1 (field) :Int32;
const minimumInt32AnnotationId :UInt64 = 0xe3a4c77f928f11f1;

#annotation minimumUint32  (field) :UInt32;
#const minimumUint32AnnotationId :UInt64 = ;

#annotation minimumFloat32  (field) :Float32;
#const minimumFloat32AnnotationId :UInt64 = ;

annotation minimumFloat64 @0xc31db5c1eea61fa2 (field) :Float64;
const minimumFloat64AnnotationId :UInt64 = 0xc31db5c1eea61fa2;

# "maximum":
#annotation maximumInt16  (field) :Int16;
#const maximumInt16AnnotationId :UInt64 = ;

annotation maximumUint16 @0x8cc82e85ae657500 (field) :UInt16;
const maximumUint16AnnotationId :UInt64 = 0x8cc82e85ae657500;

annotation maximumInt32 @0x81f5d672443fc146 (field) :Int32;
const maximumInt32AnnotationId :UInt64 = 0x81f5d672443fc146;

#annotation maximumUint32 (field) :UInt32;
#const maximumUint32AnnotationId :UInt64 = ;

#annotation maximumFloat32 (field) :Float32;
#const maximumFloat32AnnotationId :UInt64 = ;

annotation maximumFloat64 @0xfedd17bd33458fb8 (field) :Float64;
const maximumFloat64AnnotationId :UInt64 = 0xfedd17bd33458fb8;


Then, a structure would be written like this in another cap'n proto schema:

struct Params
{
    param1 @0 :Float64 = 50 $AttribJson.units("%") $AttribJson.minimumFloat64(0.0) $AttribJson.maximumFloat64(100.0) $AttribJson.multipleOfFloat64(0.1);
    param2 @1 :UInt16 = 0 $AttribJson.units("ms") $AttribJson.minimumUint16(0) $AttribJson.maximumUint16(999);
}


Then, in the cap'n proto to JSON schema tool, you need to read the annotations.


Jean-François Gauvin

[Kenton Varda]

FWIW, I think this is a great use of annotations -- in fact I've seen a lot of systems that use Protobuf annotations (aka "custom options") like this.

I would maybe choose shorter identifier names, though. :)

-Kenton

[ad...@binarycannon.com]

During code generation, capnp already turns your .capnp files into a datastructure that it represents as a capnp message.  You can convert that capnp message to json, maybe with a 'capnp decode' step if you prefer to operate on the text version of the message.  This also has the benefit that if you turn the json into a capnp message, you can run the code generators on it.