Rules for conversion of XDI JSON serialization to X3 serialization

[Drummond Reed]

It struck me that the rules are so straightforward for conversion of the JSON serialization format to X3 Standard format that I could just list them in an email. Here we go:

1) For the JSON root context:
  a) convert each {"!": {"predicate": "literal"}} statement into [$[predicate["literal"]]] SEE NOTE 1
  b) convert each {"*": {"predicate": [reference]} statement into [$[predicate[reference]]]
  c) convert each {"$": {"predicate": {subcontext}} statement into [subject] <-- yes, the predicate of the JSON $ statement is the subject in X3 -->

2) For each first-level JSON subcontext (where the X3 already has [subject])
  a) convert each {"!": {"predicate": "literal"}} statement into [subject[predicate["literal"]]]  SEE NOTE 1
  b) convert each {"*": {"predicate": [reference]} statement into [subject[predicate[reference]]]
  c) convert each {"$": {"predicate": {subcontext}} statement into [subject[predicate[[]]] ==>

3) For each second-level JSON subcontext (where the X3 already has [subject[predicate[[]]])
  a) convert each {"!": {"predicate": "literal"}} statement into [subject[predicate[[$[predicate["literal"]]]]]  SEE NOTE 1
  b) convert each {"*": {"predicate": [reference]} statement into [subject[predicate[[$[predicate[reference]]]]]]
  c) convert each {"$": {"predicate": {subcontext}} statement into [subject[predicate[[subject]]]] <-- again, the predicate of the JSON $ statement is the subject in X3 -->

4) Repeat steps 2 and 3 for each lower level of subcontext, recursed inside the previous level, until done.

NOTE 1: This step will require converting the value of the JSON serialization (which may be any native JSON datatype) into a string.

=Drummond

On Thu, Oct 7, 2010 at 11:54 PM, Drummond Reed <drummo...@gmail.com> wrote:

This suggests our work item for the second part of tomorrow's call, which is mapping the new JSON serialization to X3 statements. If we can work out that map, and it's lossless, then there's no question that we can build a new JSON serialization module for XDI4J, and we'd be able to rock 'n roll forward as Mike would like.

If I could stay awake further here tonight, I'd work on it now, but I stayed up too late last night, so we'll do it live tomorrow (if we have time after the OAuth stuff).

'night,

=Drummond

On Thu, Oct 7, 2010 at 11:16 PM, Markus Sabadello <markus.s...@gmail.com> wrote:

Joseph,

Mike is trying to send the following X3J message to the PDS:

{"=me": {"$get": {"$": null}}}

This should be equivalent to the following in X3 Simple:

=me
    $get
        /
            $

(To test this, you could go to the XDI Converter, enter the X3 Simple and convert it to X3J).

When sending this message to a PDS, it should ALWAYS return some data (the context self-descriptor).
But when Mike sends it in X3J, he gets no response.

I did some research and think it's a bug in your X3JReader.
It seems your code actually converts the above JSON to the following X3 Simple equivalent:

=me
    $get
        "{\"$\":null}"

Could you take a look and work with Mike to fix this..?

Mike, as a quick workaround, you could send your messages in one of the other supported formats, e.g. X3 Standard, which would look like this:
[=me[$get[[$]]]]

Markus

[Joseph Boyle]

On Oct 8, 2010, at 12:32 AM, Drummond Reed wrote:

It struck me that the rules are so straightforward for conversion of the JSON serialization format to X3 Standard format that I could just list them in an email. Here we go:

1) For the JSON root context:
  a) convert each {"!": {"predicate": "literal"  }} statement into [$[predicate["literal"]]] SEE NOTE 1
  b) convert each {"*": {"predicate": [reference ]} statement into [$[predicate[reference]]]
  c) convert each {"$": {"predicate": {subcontext}} statement into [subject]         <-- yes, the predicate of the JSON $ statement is the subject in X3 -->

If the ! * $ contexts are good, why not simply retain them in the X3 serialization?

2) For each first-level JSON subcontext (where the X3 already has [subject])
  a) convert each {"!": {"predicate": "literal"  }} statement into [subject[predicate["literal"]]]  SEE NOTE 1

  b) convert each {"*": {"predicate": [reference ]} statement into [subject[predicate[reference]]]

  c) convert each {"$": {"predicate": {subcontext}} statement into [subject[predicate[[]]]    ==>

What does the ==> mean?

3) For each second-level JSON subcontext (where the X3 already has [subject[predicate[[]]])
  a) convert each {"!": {"predicate": "literal"  }} statement into [subject[predicate[[$[predicate["literal"]]]]]  SEE NOTE 1

  b) convert each {"*": {"predicate": [reference ]} statement into [subject[predicate[[$[predicate[reference]]]]]]

  c) convert each {"$": {"predicate": {subcontext}} statement into [subject[predicate[[subject]]]]    <-- again, the predicate of the JSON $ statement is the subject in X3 -->

4) Repeat steps 2 and 3 for each lower level of subcontext, recursed inside the previous level, until done.

NOTE 1: This step will require converting the value of the JSON serialization (which may be any native JSON datatype) into a string.

Are we abandoning efforts to support JSON datatype semantics? I thought we agreed this would be easy for scalar types and for many JSON object { } structures, with the main problem being the JSON array type.

It is also still worth thinking about JSON serialization as JSON arrays, i.e. square brackets in X3 simply become square brackets in JSON with no rearrangement necessary. Besides visual straightforwardness, this allows Javascript code to straightforwardly use array element access by position instead of having to find the set of keys of a { } JSON object. The only remaining point is distinguishing references from literals and there are several possible easy ways of doing that. (Isn't there already XRI syntax for a literal, perhaps as a cross-reference?)

A former advantage of using { } structures for serialization was that we only needed 2 levels of { } vs. 3 levels of [ ] , but in the "new view" I think we need 3 levels of { } or possibly even 4 - could we clarify this with examples?

[Drummond Reed]

On Fri, Oct 8, 2010 at 8:25 AM, Joseph Boyle <boyle....@gmail.com> wrote:

On Oct 8, 2010, at 12:32 AM, Drummond Reed wrote:

It struck me that the rules are so straightforward for conversion of the JSON serialization format to X3 Standard format that I could just list them in an email. Here we go:

1) For the JSON root context:
  a) convert each {"!": {"predicate": "literal"  }} statement into [$[predicate["literal"]]] SEE NOTE 1
  b) convert each {"*": {"predicate": [reference ]} statement into [$[predicate[reference]]]
  c) convert each {"$": {"predicate": {subcontext}} statement into [subject]         <-- yes, the predicate of the JSON $ statement is the subject in X3 -->

If the ! * $ contexts are good, why not simply retain them in the X3 serialization?

What I defined here was just a mapping from JSON to X3 Standard. I didn't address how we could add the richer semantics of the JSON format to X3. You are right that we can start to use the ! and * contexts in X3 if we want.

2) For each first-level JSON subcontext (where the X3 already has [subject])
  a) convert each {"!": {"predicate": "literal"  }} statement into [subject[predicate["literal"]]]  SEE NOTE 1

  b) convert each {"*": {"predicate": [reference ]} statement into [subject[predicate[reference]]]

  c) convert each {"$": {"predicate": {subcontext}} statement into [subject[predicate[[]]]    ==>

What does the ==> mean?

Nothing - it was left over from a copy and paste. Ignore it.

3) For each second-level JSON subcontext (where the X3 already has [subject[predicate[[]]])
  a) convert each {"!": {"predicate": "literal"  }} statement into [subject[predicate[[$[predicate["literal"]]]]]  SEE NOTE 1

  b) convert each {"*": {"predicate": [reference ]} statement into [subject[predicate[[$[predicate[reference]]]]]]

  c) convert each {"$": {"predicate": {subcontext}} statement into [subject[predicate[[subject]]]]    <-- again, the predicate of the JSON $ statement is the subject in X3 -->

4) Repeat steps 2 and 3 for each lower level of subcontext, recursed inside the previous level, until done.

NOTE 1: This step will require converting the value of the JSON serialization (which may be any native JSON datatype) into a string.

Are we abandoning efforts to support JSON datatype semantics? I thought we agreed this would be easy for scalar types and for many JSON object { } structures, with the main problem being the JSON array type.

Again, all I was doing here was defining the conversion of JSON into the existing X3 Standard. To keep the semantics of the different JSON datatypes, we'd need to figure out how to add that metadata to X3 Standard (I glossed over that part in NOTE 1).

It is also still worth thinking about JSON serialization as JSON arrays, i.e. square brackets in X3 simply become square brackets in JSON with no rearrangement necessary. Besides visual straightforwardness, this allows Javascript code to straightforwardly use array element access by position instead of having to find the set of keys of a { } JSON object. The only remaining point is distinguishing references from literals and there are several possible easy ways of doing that. (Isn't there already XRI syntax for a literal, perhaps as a cross-reference?)

When we tried it before, we ran into some problem - I can't remember what it was. But I agree we should try again. I'd like to determine once and for all why we should or should not use JSON arrays for serialization.

A former advantage of using { } structures for serialization was that we only needed 2 levels of { } vs. 3 levels of [ ] , but in the "new view" I think we need 3 levels of { } or possibly even 4 - could we clarify this with examples?

Let's do that in the exercise. I have to go run an errand, but I'll start playing with it when I get back.

=Drummond 

[Joseph Boyle]

On Oct 8, 2010, at 9:07 AM, Drummond Reed wrote:

On Fri, Oct 8, 2010 at 8:25 AM, Joseph Boyle <boyle....@gmail.com> wrote:

On Oct 8, 2010, at 12:32 AM, Drummond Reed wrote:

It struck me that the rules are so straightforward for conversion of the JSON serialization format to X3 Standard format that I could just list them in an email. Here we go:

1) For the JSON root context:
  a) convert each {"!": {"predicate": "literal"  }} statement into [$[predicate["literal"]]] SEE NOTE 1
  b) convert each {"*": {"predicate": [reference ]} statement into [$[predicate[reference]]]
  c) convert each {"$": {"predicate": {subcontext}} statement into [subject]         <-- yes, the predicate of the JSON $ statement is the subject in X3 -->

If the ! * $ contexts are good, why not simply retain them in the X3 serialization?

What I defined here was just a mapping from JSON to X3 Standard. I didn't address how we could add the richer semantics of the JSON format to X3. You are right that we can start to use the ! and * contexts in X3 if we want.

On Wednesday Markus gave his opinion that we are actually changing the graph model, not just the serialization - I think I agree.

Our existing X3 serialization <---> Java data structures <---> JSON (X3J) serialization conversion would handle the "new view" ! * $ format and simply reproduce it in X3 as well as the Java-object graph.

Are we abandoning efforts to support JSON datatype semantics? I thought we agreed this would be easy for scalar types and for many JSON object { } structures, with the main problem being the JSON array type.

Again, all I was doing here was defining the conversion of JSON into the existing X3 Standard. To keep the semantics of the different JSON datatypes, we'd need to figure out how to add that metadata to X3 Standard (I glossed over that part in NOTE 1).

Using metadata is encapsulating JSON data without actually allowing its semantics to be used in XDI... we've also talked about native support.

It is also still worth thinking about JSON serialization as JSON arrays, i.e. square brackets in X3 simply become square brackets in JSON with no rearrangement necessary. Besides visual straightforwardness, this allows Javascript code to straightforwardly use array element access by position instead of having to find the set of keys of a { } JSON object. The only remaining point is distinguishing references from literals and there are several possible easy ways of doing that. (Isn't there already XRI syntax for a literal, perhaps as a cross-reference?)

When we tried it before, we ran into some problem - I can't remember what it was. But I agree we should try again. I'd like to determine once and for all why we should or should not use JSON arrays for serialization.

I don't think there is any problem other than distinguishing literals and references.

[Drummond Reed]

On Fri, Oct 8, 2010 at 10:16 AM, Joseph Boyle <boyle....@gmail.com> wrote:

On Oct 8, 2010, at 9:07 AM, Drummond Reed wrote:

On Fri, Oct 8, 2010 at 8:25 AM, Joseph Boyle <boyle....@gmail.com> wrote:

It is also still worth thinking about JSON serialization as JSON arrays, i.e. square brackets in X3 simply become square brackets in JSON with no rearrangement necessary. Besides visual straightforwardness, this allows Javascript code to straightforwardly use array element access by position instead of having to find the set of keys of a { } JSON object. The only remaining point is distinguishing references from literals and there are several possible easy ways of doing that. (Isn't there already XRI syntax for a literal, perhaps as a cross-reference?)

When we tried it before, we ran into some problem - I can't remember what it was. But I agree we should try again. I'd like to determine once and for all why we should or should not use JSON arrays for serialization.

I don't think there is any problem other than distinguishing literals and references.

After working it through, I agree. I gave up before because I couldn't figure out a simple solution to that. But ironically the solution for the "JSON array format" is identical to what we figured out for what I'll now call the "JSON object format". Here's the basic pattern, shown in condensed format:

[

["!",["predicate",["any-form-of-json-datatype-as-a-literal"]]],

["*",["predicate",["reference", "reference", "reference"]]],

["$",["predicate",[["start-recursion-of-!-*-$-statements-here"]]]]

]

In other words, the !, *, and $ statements are used to declare what to expect as the contents of the JSON array that is the object of the ! subject. For ! statements, it can be any JSON datatype. For * statements, it must be an array. For $ statements, it must be a nested array, which starts the whole recusion over again.

Here's the pattern shown after you put the above into JSON Lint:

[
    [
        "!",
        [
            "predicate",
            [
                "any-form-of-json-datatype-as-a-literal"
            ]
        ]
    ],
    [
        "*",
        [
            "predicate",
            [
                "reference",
                "reference",
                "reference"
            ]
        ]
    ],
    [
        "$",
        [
            "predicate",
            [
                [
                    "start-recursion-of-!-*-$-statements-here"
                ]
            ]
        ]
    ]
]

Joseph, do you think this is significantly more efficient to process? It does look much closer to X3 Standard -- the only real addition is the commas and the use of the metagraph symbols to solve the datatyping problem.

=Drummond

[Joseph Boyle]

Another way would be just to serialize some marker as part of the literal string, for example another pair of quotes around the string.

Actually if the keys are known to be only "!" "*" and "$", that is well suited to a JSON { } object to represent that level - you don't have to fetch the set of keys and iterate over it, you can just use ."!" or ["!"] to retrieve the children of a given class directly. Continuing to use arrays for the rest of the triple structure, that would give:

{

"!":["predicate",["any-form-of-json-datatype-as-a-literal"]],

"*":["predicate",["reference", "reference", "reference"]],

"$":["predicate",[["start-recursion-of-!-*-$-statements-here"]]]

}

and:

{

    "!": [

        "predicate",

        [

            "any-form-of-json-datatype-as-a-literal"

        ]

    ],

    "*": [

        "predicate",

        [

            "reference",

            "reference",

            "reference"

        ]

    ],

    "$": [

        "predicate",

        [

            [

                "start-recursion-of-!-*-$-statements-here"

            ]

        ]

    ]

}

Digressing from the X3 Standard compatibility for a minute, the above could be further simplified by removing brackets:

{

"!":["predicate","any-form-of-json-datatype-as-a-literal"],

"*":["predicate","reference", "reference", "reference"],

"$":["predicate","start-recursion-of-!-*-$-statements-here"]

}

That doesn't account for multiple predicates of the same type under one subject, though - how were you doing it in your example? Presumably as additional array elements in the lists headed by "!" "*" "$"? The additional brackets would be needed for disambiguation only in the case of references where multiple references are allowed.

Have we ever considered dropping multiple references per predicate in favor of repetition of the predicate for each reference? Spacewise, this would not be that big an expansion, and the savings in structure needed might be worth it.