Missing dimensions

[Igor]

Hi.

Let assume we have following data:

I.e. sex dimension has 2 values: female and mail, route -- oral and intra, var -- AUC6 and AUC12. But not all possible strata (i.e. combination of sex, route and var)  were filled/measured/interested. 

Is it possible to represent such data with json-stat without introducing formal strata with nulls in value?

Regards,

Igor.

[Simon Speich]

The JSON-stat schema allows for an "extension" property, where you can add whatever structure you want:  https://json-stat.org/format/#extension

We used this to define a property to list all the indexes where the values are systematically null and then excluded them from the value array.

Just an idea.

Simon

[Simon Speich]

Sorry, not quite correct, we didn't exclude them from the value array, but used the information from the extension property to hide them when rendering and provided a show/hide function to the user.

[Xavier Badosa]

If you have a lot of nulls due to missing data or impossible combinations of variables use an object instead an array of values:

Instead of:

value: [3, null, null, null, null, null, 25]

use

value: {"0": 3, "6": 25}

Xavier

--
You received this message because you are subscribed to the Google Groups "json-stat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-stat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/json-stat/f99ad4f3-bce9-4391-9838-cfe16a5f5b8en%40googlegroups.com.

[Igor]

Is it something like below?
{ 

  ...

  "dimension": {

  ...

  }, 
  "extension": {
    "id": "partial data",
    "stratas": {
      "exclude": [
        {"route": "intra","sex": "female"},
        {"route": "oral","sex": "male"},
        {"route": "intra","sex": "male","var": "AUC12"},
        {"route": "oral","sex": "male","var": "AUC12"}
      ]
    },
    "version": "0.1"
  },
  "value": [...]

}
That will need further processing/interpretation? 

If "Yes", this will mean that standard tools need to be extended in include additional logic.

Regards,

Igor.

[Igor]

Got is, Xavier.

 

Data that I assume to store in json-stat format may not (or even  often will not) contain all combinations of dimension values. And number of "facts"/measurements may be quite big (experimental data and their derivatives).  NULLs will be ok but if not too many. And storing data as an object at a first glance looks too heavy for big datasets. As for now extensions with include/exclude definition of strata looks more promising. 

Regards,

Igor.

[Xavier Badosa]

Igor,

You can use the structure you want inside an "extension" (as far as the extensions are in the allowed places -dataset-level extension, dimension-level extension). https://json-stat.org/format/#extension

JSON-stat libraries should provide a way to retrieve extension information but cannot process it as they can't know in advance its contents. So yes, you need to include the additional logic.

Xavier

To view this discussion on the web visit https://groups.google.com/d/msgid/json-stat/3b82e880-25d1-4b07-9577-a76a2ca92f4an%40googlegroups.com.

[Igor]

Hi  Xavier.

Can I summarize the answer on the topic as following?

1. json-stat suggest that data for all dimensions are presented.

2. If some data are not available, nulls may be used inplace in an array or json object for scarce data.

3. If some combinations of dimensions are missing or not used, an extension may be used. But it is a responsibility of user to write an appropriate validator/reader.

Ragards,

Igor.

[Xavier Badosa]

That is sort of correct,

Xavier

To view this discussion on the web visit https://groups.google.com/d/msgid/json-stat/f06c315c-7e8d-4280-b2ce-706c957a5091n%40googlegroups.com.