New properties discussion: "default", "required"

[Xavier Badosa]

In my work developing JSON-stat visualizations and viewers like the Table Browser (https://json-stat.org/format/browser/), I stumbled upon the convenience of providing more hints on how to first display the data. For instance, it seems useful to allow producers to tag a dimension category as the "default", meaning that, if an interface requires to pre-select one and only one value (for example, in a pull-down menu), the "default" category should be the one to be picked. With no indication of a "default" category, the interface is forced to arbitraryly select the first (or the last) category, or one with a particular id (like "0") or label (like "total").

Long time ago I also discussed with Trygve Falch (Statistics Norway) the use of JSON-stat as a metadata format, replacing the PC-Axis metadata format. I was warned by Trygve that this is currently not a 100% possible because JSON-stat has no way to express the PC-Axis concept of "elimination". Lars Knudsen (Statistics Denmark) expressed a similar concern more recently.

Considering JSON-stat adoption has been partly tied to PX-Web, it makes a lot of sense to be very attentive to the needs of PC-Axis-based producers, as they are most of the JSON-stat providers. That said, I try to keep the standard as simple as possible and only add features that can be considered general needs. That's why I do not plan to propose to add an "elimination" property, which would match perfectly PC-Axis needs: if the meaning of the term is too narrow it won't address a problem as general as possible (but of course I might be wrong trying to solve too many cases at the same time).

"elimination" is used in PC-Axis metadata responses to provide information on how a dataset can be customized (what dimensions can be dropped). It can take two forms:

ELIMINATION("sex")=YES;

ELIMINATION("region")="All Denmark";

My standardization proposal is based on the general properties "default" (already mentioned) and "required", as a simple way to provide more metadata on dimensions. As I said earlier, "default" only means some category in a dimension is to be picked if the context requires to pick a single one. "required" (by default, true) only means that, when false, some dimension might be removed from a dataset ("how" is up to the provider).

To express ELIMINATION("sex")=YES PC-Axis providers would be able to use:

"sex" : {

   "label" : ...

   "category" : {

      "index": ...

      "label": ...

   },

   "required": false

}

To express ELIMINATION("region")="All Denmark" PC-Axis providers would be able to use:

"region" : {

   "label" : ...

   "category" : {

      "index": ...

      "label": ...

      "default": "000" //Category id for "All Denmark"

   },

   "required": false

}

("default" applies to categories; "required" to dimensions. Both are non-mandatory.)

In a non-PC-Axis context, a provider that does not offer a mechanism to remove dimensions would probably not use "required" (always true) but would still benefit from "default" to inform of the existence of a main category in a dimension. (Or maybe it could use "required": false in data responses to suggest a simplified view of a dataset.)

Should "required" and "default" be added to the standard? Is there a general need for them? (Otherwise they are better kept inside "extension") Is their meaning too open? Are they trying to solve too many (different) cases at the same time?

If added, do you plan to use any of these new properties? How?

All feedback is welcome and appreciated.

[Xavier Badosa]

If no feedback is received, I will interpret there's no real interest in adding "default" and "required" and, for simplicity's sake, they won't be added to the standard.

X. 

[Jan Bruusgaard]

This was discussed at the PX user meeting last week, and it would be nice if these were added. It makes it possible to use JSON-stat for PxWeb-metadata in the future. We could then have the same json-format for metadata and for data. 

Today's PxWebApi has a less efficient json-format for metadata, and SCB now want to develop a version2 of the API.

Jan Bruusgaard

Statistics Norway, Communications department

.  

[Xavier Badosa]

Dear Jan,

Not having received any feedback in a year and a half I assumed there was no real interest for this. If there is (some) interest, it must be decided, pondering how strong and general that interest is, how these properties are supported (inside "extension" or not).

As you know, I was an invited speaker in the last Workshop of the Expert Group on SDMX where I spoke about JSON-stat in the sea of standards. One of the ideas that inspired my presentation were standardized extension patterns: instead of making richer (but less simple) the JSON-stat spec (adding new properties), specific needs related to other standards can be addressed using "extension" in a standardized (that is, somehow "officialized") way.

I explained this in the context of the SDMX-JSON/JSON-stat conversion. JSON-stat can be as rich as SDMX-JSON (using "extension") and, because of this, it could be offered in SDMX infrastructures as an additional dissemination format (so that users could have more options to choose from, taking into account the features and the available tools for each format). But this should be done following a documented standardized extension pattern, so that conversion tools like fromSDMX(), sdmx2jsonstat or mapDataSetsToJsonStat() use compatible mappings. This is already happening somehow at a very small scale (fromSDMX() and sdmx2jsonstat are following Eurostat's extension pattern to keep status label information in JSON-stat). My proposal to the Expert Group was (big scale) to identify the most important pieces of SDMX missing in JSON-stat and agree on a way to preserve them inside "extension" in JSON-stat.

Something similar could be done in the context of the JSON-stat/PX-JSON relationship.

Is "elimination" the only missing concept? Are "default" and "required" the right way to address "elimination"? Are "default" and "required" relevant concepts beyond the JSON-stat/PX-JSON conversion? Is there a general need for them as to deserve to be included as new optional properties or should they be kept inside "extension" (that is, not part of the JSON-stat standard but part of the standardized extension patterns)?

I'd appreciate any feedback on these particular questions,

X.