Call for Input! JSON Serialization Design Doc

[Beka Westberg]

Hello everyone!

As I said in the core Q3 work post, this quarter I will be working on a project related to serialization. Specifically, I will be building a new JSON serializer :D 

This will give developers more flexibility around how they interact with serialized state. That includes making it easier to save to other formats (including binary!), and making it easier to programmatically deal with state (eg when creating dynamic toolbox categories).

Our current plan is to create a separate JSON serializer, and to icebox the XML one. This means the XML serializer will still exist, but it will not get any new features, eg plugin serialization. For more details, please see the full design doc (~15 minute read, not including appendices). 

If you're short on time but still want to participate, here are a few questions that I would love input on:

  1) If you are currently using the XML serializer, what backwards compatibility features would be most useful to you? (~2 min read)

    * Would you use an XML to JSON format converter?

    * Would you appreciate JSON wrappers for existing hooks (eg mutationToDom)?

  2) What structure do you think the JSON format should have? (~2 min read)

    * Should connections point to arrays or objects to deal with shadows?

    * Should value inputs and statement inputs be bundled, or separated?

    * Should stacks be sequential, or nested?

If you have any other thoughts, feel free to leave them below, in the document, or message me directly!

    

This was something that got requested a lot at the 2021 summit, so I'm excited to be working on it! I can't wait to hear your feedback =)

Thank you for your time,

--Beka

[Jonathan de Halleux]

This is a fantastic news.

[Beka Westberg]

> This is a fantastic news.

I'm glad you think so! I'm excited about it to :D

[Uwe K]

Hello Beka,

this looks very interesting, I'm looking forward to having more formats to choose from. If we would move away from XML, then we'd need a converter for our existing projects. So if you write one, that would make the decision easier. I don't have an opinion on your other questions, except stacks should be arrays!

On Monday, July 12, 2021 at 10:09:44 PM UTC+2 bwes...@google.com wrote:

[Beka Westberg]

Thank you for the feedback :D I can't wait for there to be more formats too!

> If we would move away from XML, then we'd need a converter for our existing projects.

The "poor mans converter" is to deseriaize your XML into a headless worspace, and then reserialize it to JSO/whatever. So even if I don't get to something more specialized this summer, there's still a solution! But it's good to know that there's a demand for a converter :D

> I don't have an opinion on your other questions, except stacks should be arrays!

Hehe thank you for your feedback! But currently I'm actually leaning toward keeping them nested :/ I've attempted to implement the stack-based format, and there were two big problems:

1) Shadows complicate the stack-based format, and rely on hidden invariants.

For example, if you have a stack like:

```

[

  {

    'type': 'text_print',

  },

  {

    'type': 'text_print',

    'kind': 'shadow',

  },

  {

    'type': 'text_print',

    'kind': 'shadow',

  },

  {

    'type': 'text_print',

  },

]

```

This is actualy the serialization of a stack that looks like this:

It only works because there is an invariant of Blockly that shadow blocks may not have non-shadow children, which isn't very intuitive or clear.

2) Stacks don't work well with the type system.

For example, in a stacks-as-arrays format the top level structure would most likely look like this:

```

{

  "blocks": [  // Array of arrays (stacks)

    [  // Array of blocks (a stack)

      {  // Actual block

        "type": "text_print"

      }

    ]

  ]

}

```

To keep the types consistent, that would mean that a group of blocks like below would also need to be wrapped in an array (even though there will never be a following block).

This especially becomes an issue when creating JSOs programmatically, such as dynamic toolbox categories.

But the core team is pretty split on how this should work, so if anyone has opinions we would really appreciate them :D

Best wishes and thank you again for the feedback!

--Beka

[Beka Westberg]

Hello again =)

I've updated the design document with a section about deserialization order. This is useful when you want to minimize the amount of duplicate state you have to save. For example, in some cases you will want to store the state of a field in your extraState/mutator in addition to where it is normally stored, but in other cases you don't need to do that. The doc goes over the situations in more detail =)

I've also updated the doc to have more info about alternative structures we considered for the JSO, and reasons why we decided not to pick them. We're still open to opinions on this one though!

As always if anyone has comments, questions, or suggestions I really appreciate them! Feel free to leave them here, in the doc, or the message me directly =)

Best wishes,

--Beka