How about just one more built-in generic type for Teleport?

[Y Sharp]

Hi,

I am thinking of:

x :: t({"Nullable": p}) if and only if

*either*

1. x is null

or

2. x :: t(p) where p is a string (i.e., p is a concrete type instance) or any of

{"Array": q}, {"Map": q}, {"Struct": q}, or {"Nullable": q} for some q

(i.e., p is a generic type instance parameterized by q).

Thus, as a direct consequence of the above definition, for any JSON value x and for any (well-formed) Teleport type instance p:

x :: t(p) => (implies) x :: t({"Nullable": p})

Therefore, in addition to:

null :: t("JSON")

(which we already knew) we have also:

null :: t({"Nullable": "JSON"})

null :: t({"Nullable": "Schema"})

null :: t({"Nullable": "Integer"})

null :: t({"Nullable": "Float"})

null :: t({"Nullable": "String"})

null :: t({"Nullable": "Boolean"})

null :: t({"Nullable": "DateTime"})

and, by very definition:

null :: t({"Nullable": p}) for any user-defined concrete or generic type instance p.

Unsurprisingly, the main rationale for introducing "Nullable" is to lessen the *decision burden* of the application w.r.t. type checking / schema validation in the use cases of possible "maybe / nullable" value semantics precisely *when* the application coincidentally *does* encounter null values in the JSON input it has to process.

For instance, this way, with an incoming null read from the input, and the data, of say, only t(p) - *instead of* t({"Nullable": p}) - for a given target entity of the host language in which the application was written, the latter will have at least the opportunity to benefit from the Teleport type checking implementation (which will reject null in that example).

Thoughts?

Cyril

[Alexei Boronine]

Hi Cyril,

There are two parts to this response. Firstly, let's talk about adding types to the spec.

I'm trying to have my cake and eat it too: to keep Teleport minimal but to encourage extensibility. We both know the trade-offs involved in adding functionality and in the case of Teleport, I lean heavily towards minimalism (which IMO is a big contributing factor to the success of JSON). Because of this, I emphasize that type-checking existing JSON documents is *not* a design goal of Teleport (for that purpose, there is http://json-schema.org/). Instead, Teleport provides a minimal set of building blocks, which should *not* include null. The inconsistency created by null greatly outweighs its legitimate use cases, which are few. I explain this in the second part of the email.

Having said that, an extended version of Teleport that *does* make type-checking existing documents its goal would be quite useful, despite stepping into JSON Schema's territory. For this project, Nullable is an excellent type: elegant and composable that covers just about every real-world use of null in JSON. If a type is useful but not fundamental enough to be in the spec, where should it be defined? This is an open question, and an important one for Teleport as a project.

The formal definition for Nullable can be written as simply as this:

    x :: t({"Nullable": p}) if and only if x = null or x :: t(p)

The Python implementation is about 10 LOC: https://github.com/cosmic-api/teleport/blob/py-master/python/teleport/examples.py#L16-L25

Now I explain my decision against null.

First of all, there are two distinct concepts of null:

1. In static type systems (SQL, Java etc.), null represents the absence of a value. It is the Nothing in Haskell's Maybe.

2. In dynamic type systems, null is a singleton value which is used as a placeholder for a missing value.

I don't have a problem with either of these definitions, but I have a problem with the way #2 is usually implemented, including in JSON. The dynamic null is great, but its semantics are weakened by the fact that most dynamic languages provide *multiple* ways to represent missing data. Python has None, but it also has IndexError, KeyError, AttributeError etc. JavaScript has null, but it also has undefined [1].

In JSON (a dynamic type system), you have, broadly speaking, three ways to represent missing data:

a. By putting null in place of a value (at document root [2], in arrays, in objects)

b. By omitting the value along with its name (in objects)

c. By omitting the entire document [3]

For document root and objects respectively, (b) and (c) are obviously preferable [4]. As such, we are left with just one justifiable place for null: JSON arrays. I can think of two use cases for null array elements: naive sparse arrays and tuples, and I don't think either of them deserves to be in the Teleport core. Sparse arrays are obscure and tuples (w/ Nullable) are functionally equivalent to Structs, so adding Tuple into the core will force people to make a choice between Tuple and Struct whenever they need to group some values. This is a completely inconsequential design decision that can be made once and never bothered with again.

Overall, the justifiable use cases for null are so rare that I'm convinced that including it in JSON in the first place was a mistake. Frankly, I would like to force null out of Teleport entirely by making Teleport operate on a null-less subset of JSON, but this would break the wildcard type and the ability to make an extension of Teleport as described in the first part of the email, so I'm keeping it.

[1] I'm not sure what undefined really is, but it behaves like a singleton value, same as null

[2] Not allowed by the JSON spec, probably as a result of a similar discussion

[3] For example, an HTTP response with an empty body where a JSON body was expected

[4] In some cases, there are two semantically distinct ways that a value can be missing in an object. In these cases using (a) and (b) to distinguish between them may be an elegant approach. IMHO, for every legitimate use case there will be dozens of ill-conceived ones. I think most people working with JSON have encountered errors or were forced to write defensive code as a result of this.

[Y Sharp]

Hi,

Thanks again for the elaborated reply.

Yes, I am not sure either as to why exactly null has been allowed to appear in JSON instances vs. say, undefined.

I do suspect it's precisely the latter's inconvenience which motivated to keep null, e.g., to give explicit in-band (within payloads) hints to the application that this or that particular target member value isn't known and that this absence of information needs to be acknowledged as such (as opposed to doing silent fallback onto defaults of the target host language).

As you pointed out, I do find Nullable interesting for Teleport for its composability and precisely giving the opportunity to Teleport schema authors to move away from a host language/implementation-dependent-only handling of nulls in the JSON.

This way, if Nullable isn't used, the application layer can stay in charge of interpreting conflicts or ambiguities with the host language target type system; but if it is used, then the Teleport schema has its word to say on what it is acceptable to consider as "nullable" (or not) because of the coincidental presence of nulls in the JSON input.

Thanks also for the clarification on the Teleport design goals (agnosticism) regarding its relation with host language type checking in general; duly noted.

Cheers,

Cyril