JSON-schema converters

[Andrei Neculau]

A shot in the dark here, but has anyone bumped into the need of converting JSON-schema to/from ProtoBuf, Thrift, Avro..?

[Francis Galiegue]

Well, with my implementation, this is a possibility. Tatu Saloranta of
Jackson has even hinted me that the next version of my API could be
used to do that. Especially since Jackson has Avro support.

All you need to do is write an appropriate processor. See this page,
session "processors":

https://github.com/fge/json-schema-validator/wiki/Roadmap

--
Francis Galiegue, fgal...@gmail.com
Try out your JSON Schemas: http://json-schema-validator.herokuapp.com

[Julien Silland]

I have worked on something to configure a two-way transform between protobuf and JSON, and which is also able to produce the corresponding JSON schema. This is still my employer's IP but I'm trying to get it open-sourced as we speak. I'll update this thread when this happens.

-jsi

On Mon, Feb 11, 2013 at 12:47 PM, Andrei Neculau <andrei....@gmail.com> wrote:

A shot in the dark here, but has anyone bumped into the need of converting JSON-schema to/from ProtoBuf, Thrift, Avro..?

--
You received this message because you are subscribed to the Google Groups "JSON Schema" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-schema...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Andrei Neculau]

between JSON and ProtoBuf,

but not between JSON Schema and ProtoBuf IDL? or did I miss smth?

--

andreineculau.com -- Emails should be five sentenc.es or less.

The time spent on any item of the agenda will be in inverse proportion to the sum involved. *C. Northcote Parkinson* http://en.wikipedia.org/wiki/Parkinson's_Law_of_Triviality

[Julien Silland]

The protobuf IDL is represented at runtime by classes such as Descriptors.{Descriptor,FieldDescriptor, etc...} and all that data is available statically from the generated code. The library works with a message descriptor to create the equivalent JSON Schema, modulo some transforms that are configurable by the user. Parsing and serializing from and to JSON are also on the feature list. The use case we had was to expose internally-defined proto messages as JSON.

Generating the IDL from a JSON schema seems hardly doable as the feature set of the JSON Schema spec is vastly greater than that of the protobuf IDL.  For instance, I don't see how additionalItems could be represented in the protobuf IDL.

-jsi

[Francis Galiegue]

On Thu, Feb 14, 2013 at 12:30 AM, Julien Silland <jsil...@gmail.com> wrote:
> The protobuf IDL is represented at runtime by classes such as
> Descriptors.{Descriptor,FieldDescriptor, etc...} and all that data is
> available statically from the generated code. The library works with a
> message descriptor to create the equivalent JSON Schema, modulo some
> transforms that are configurable by the user. Parsing and serializing from
> and to JSON are also on the feature list. The use case we had was to expose
> internally-defined proto messages as JSON.
>
> Generating the IDL from a JSON schema seems hardly doable as the feature set
> of the JSON Schema spec is vastly greater than that of the protobuf IDL.
> For instance, I don't see how additionalItems could be represented in the
> protobuf IDL.
>

When your library is "out there", let me know: I'll write a processor
using it which means a protobuf can be taken as an input, generate a
JSON Schema and be used directly in the validation chain...

In the same vein, the first processor I'll write will be a
JJSchema-backed processor allowing to take a _class_ as an argument
and plug it directly in the validation chain.

[Julien Silland]

I made rather vain attempts at using POJOs to back my schemas and decided to just give up after realizing I either had to sacrifice type safety or ease of use. I went the PB route because they retain a lot more metadata about their types after compilation that Java classes do. I'd be interested to see what you come up with.

Once I've open sourced the lib, I'll likely spend some amount of personal time adding supporting for Thrift objects.

Cheers,

-jsi

[Andrei Neculau]

Generating the IDL from a JSON schema seems hardly doable as the feature set of the JSON Schema spec is vastly greater than that of the protobuf IDL.  For instance, I don't see how additionalItems could be represented in the protobuf IDL.

Indeed, but I didn't say it has to be a 1:1 match. You may define a structure via JSON Schema, with validation and description information, and then just want to use a transport serialization for it.

I might be off-target here (lacking experience), but the types in ProtoBuf&co are mainly checkmarks for (de)serializing the binary form, rather than validators, so the semantics themselves of an IDL do not match those of a JSON Schema.

[Julien Silland]

On Wed, Feb 20, 2013 at 2:31 PM, Andrei Neculau <andrei....@gmail.com> wrote:

Indeed, but I didn't say it has to be a 1:1 match. You may define a structure via JSON Schema, with validation and description information, and then just want to use a transport serialization for it.

I see your point, but it seems to assume your project has a blank slate where you can start from a JSON Schema and then work your way up the system, up to the point where you need a better-performing serialization format. I think most projects work the other way around and have a stronger need to surface their internal model (whatever its form: PB, Thift, Some relational DB) to the rest of the world — that was certainly our use case.

I can see how it'd be appealing to first define API payloads as validating schemas and from there derive the internal business logic and model. That certainly appears to ba a healthier approach but not something a lot of folks can afford to do, time-wise.

 

I might be off-target here (lacking experience), but the types in ProtoBuf&co are mainly checkmarks for (de)serializing the binary form, rather than validators, so the semantics themselves of an IDL do not match those of a JSON Schema.

Apart from the required/optional declaration for each field, Protobuf have actually very little added semantic value: protobufs are really only a graph of typed properties. But I find the structures allowed by JSON to simply be a superset what the Proto IDL can represent, e.g. heterogeneous arrays.

Simply looking at the use cases people on this list ask questions about sometimes make my head hurts when I try to model how I'd achieve something similar in PB-land and yet JSON-Schema appears largely able to answer most of their needs. I suspect there would be a lot of friction at the interface between the two worlds, and the resulting developer experience would be frustrating. I'd be happy to be proven wrong :-)

Cheers,

-jsi

[Francis Galiegue]

On Thu, Feb 14, 2013 at 2:15 AM, Julien Silland <jsil...@gmail.com> wrote:
> I made rather vain attempts at using POJOs to back my schemas and decided to
> just give up after realizing I either had to sacrifice type safety or ease
> of use. I went the PB route because they retain a lot more metadata about
> their types after compilation that Java classes do. I'd be interested to see
> what you come up with.
>

Well, I have an example already ;)

Based on JJSchema and using my core library:

https://github.com/fge/json-schema-processor-examples/blob/master/src/main/java/com/github/fge/jjschema/JJSchemaProcessor.java

As the output is compatible with my whole validation chain, it means I
can validate JSON instances using a Java class :p

[Andrei Neculau]

On Thursday, February 21, 2013 10:21:50 PM UTC+1, Julien Silland wrote:

On Wed, Feb 20, 2013 at 2:31 PM, Andrei Neculau <andrei....@gmail.com> wrote:

Indeed, but I didn't say it has to be a 1:1 match. You may define a structure via JSON Schema, with validation and description information, and then just want to use a transport serialization for it.

I see your point, but it seems to assume your project has a blank slate where you can start from a JSON Schema and then work your way up the system, up to the point where you need a better-performing serialization format. I think most projects work the other way around and have a stronger need to surface their internal model (whatever its form: PB, Thift, Some relational DB) to the rest of the world — that was certainly our use case.

I can see how it'd be appealing to first define API payloads as validating schemas and from there derive the internal business logic and model. That certainly appears to ba a healthier approach but not something a lot of folks can afford to do, time-wise.

I'm not assuming it, I'm living it :) And yes, it also has the affordancel :)

I fully agree with you though on what the common flow is.