Periodicity and isFrequency

[Bob Jolliffe]

Hi

I've been been looking at the time_period dimension of observations in terms of (i) discussions we have had about it and (ii) what is required by sdmx standard and (iii) what is recommended.

The initial text we have in the draft standard doc from april 2009 is:

• Not misunderstand some concepts used as metadata for an indicator definition that might seem similar to concepts used as dimensions in the data set. For example,

  • frequency defines the recommended frequency of data collection: the concept is used as indicator definition metadata;

  • periodicity is the actual frequency of compilation of the data: the concept is used as a metadata attribute of the time series;

So looking at some sample SDMX-HD DSDs out there we typically have:
 <Attribute conceptRef="PERIODICITY" conceptSchemeRef="CS_COMMON" conceptVersion="1.0" conceptSchemeAgency="SDMX-HD" codelist="CL_PERIODICITY" codelistVersion="1.0" codelistAgency="SDMX-HD" attachmentLevel="Series" assignmentStatus="Mandatory" />


This might be OK except that that the SDMX markup spec (part 3 of the standard, page 42) says that:
"Any key family which uses the time dimension must also declare a frequency dimension, conventionally the first dimension in the key (the set of ordered non-time dimensions)."

I think this suggests that instead of the above, we actually should have:
 <Dimension isFrequency="true" conceptRef="PERIODICITY" conceptSchemeRef="CS_COMMON" conceptVersion="1.0" conceptSchemeAgency="SDMX-HD" codelist="CL_PERIODICITY" codelistVersion="1.0" codelistAgency="SDMX-HD" assignmentStatus="Mandatory" />

Cheers
Bob

[Bob Jolliffe]

I think this suggests that instead of the above, we actually should have:
 <Dimension isFrequency="true" conceptRef="PERIODICITY" conceptSchemeRef="CS_COMMON" conceptVersion="1.0" conceptSchemeAgency="SDMX-HD" codelist="CL_PERIODICITY" codelistVersion="1.0" codelistAgency="SDMX-HD" assignmentStatus="Mandatory" />

Sorry - slight error in above.  In fact the attribute should be isFrequencyDimension="true"

What would we do without validating editors?

Bob.

[Bob Jolliffe]

2010/1/22 Bob Jolliffe <bobjo...@gmail.com>

And one other small error - there is no assignmentStatus attribute on a Dimension.  So it would be:

<Dimension isFrequencyDimension="true" conceptRef="PERIODICITY" conceptSchemeRef="CS_COMMON" conceptVersion="1.0" conceptSchemeAgency="SDMX-HD" codelist="CL_PERIODICITY" codelistVersion="1.0" codelistAgency="SDMX-HD" />
 
I've checked on a few DSD's from other domains and this indeed does seem to be the correct convention.  I'm not too sure how this fits with our disaggregation hierarchical codelist - I am sure we don't want to to add the list of frequency types into the tree of allowable dimension options.  I suppose that we have to say that this dimension (the one with isFrequencyDimension="true"), like the time dimension itself, should be treated as special in this regard.

Cheers
Bob

[Gary Patchen]

Sorry, replied to Bob instead of whole group.

 

FYI - see email below.

 

gary patchen

lead consultant

direct + 41.(0)58.307.7094

mobile +41.(0)79.333.1339

gary.p...@b-i.com
www.b-i.com

blue-infinity headquarters
t +41(0)58.307.7000
f +41(0)58.307.7001
INTERNATIONAL
t +800.307.70.000
f +800.307.70.001

b-i  branding.technology.integration.

The information in this e-mail, and those ensuing, is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, please destroy this message and notify us immediately.
 

Please think of the environment before printing this email

 

---

From: Bob Jolliffe [mailto:bobjo...@gmail.com]
Sent: vendredi, 22. janvier 2010 12:02
To: Gary Patchen
Subject: Re: Periodicity and isFrequency

Hi Gary

2010/1/22 Gary Patchen <gary.p...@b-i.com>

Hi Bob,

 

Didn't we say that we do not need PERIODICITY at all as each observation will have a time period and to the actual frequency of compilation of the data can be deduced from this?

 

    e.g CL_PERIODICITY code = 4 is "Quarterly"

 

   <ns:Series PERIODICITY="4"   >
    <ns:Obs OBS_VALUE="88" TIME_PERIOD="Q1-2009" />
    <ns:Obs OBS_VALUE="76" TIME_PERIOD="Q2-2009" />

    <ns:Obs OBS_VALUE="678.4" TIME_PERIOD="Q3-2009" />
   </ns:Series>

 

I know we said that we didn't need both Frequency and Periodicity.  And that we needed to better define the concepts.  It was through trying to do this and referring back to the SDMX standard that I have realized that there must be a frequency dimension.
 

    Its obvious that the data is quarterly from the time_period attribute? Also if PERIODICITY is needed and the observation are monthly yet PERIODICITY is Quarterly should this be an SDMX-HD validation error?

 

   <ns:Series PERIODICITY="4"   >
    <ns:Obs OBS_VALUE="88" TIME_PERIOD="M1-2009" />
    <ns:Obs OBS_VALUE="76" TIME_PERIOD="M2-2009" />

    <ns:Obs OBS_VALUE="678.4" TIME_PERIOD="M3-2009" />
   </ns:Series>

I think it should certainly be an error though a difficult one to catch with the xsd.  It can be done with schematron assertions but we haven't got around to that yet :-)  Once we have all our "business rules" firmed up we can start looking at this sort of validation.
 

 

    maybe:

 

<Dimension isFrequencyDimension="true" conceptRef="PERIODICITY" conceptSchemeRef="CS_COMMON" conceptVersion="1.0" conceptSchemeAgency="SDMX-HD" codelist="CL_PERIODICITY" codelistVersion="1.0" codelistAgency="SDMX-HD"  />
   

    should be:

 

<Dimension isFrequencyDimension="true" conceptRef="FREQUENCY" conceptSchemeRef="CS_COMMON" conceptVersion="1.0" conceptSchemeAgency="SDMX-HD" codelist="CL_FREQUENCY" codelistVersion="1.0" codelistAgency="SDMX-HD" />

regards

Yes I do agree.  If we have to have a frequency dimension it is certainly better to call it FREQUENCY than to call it PERIODICITY :-)

Cheers
Bob

PS.  I see you didn't cc to the group.  I don't know if you intended to reply to just me, but if not, please cc the others for input.

 

gary patchen

lead consultant

direct + 41.(0)58.307.7094

mobile +41.(0)79.333.1339

gary.p...@b-i.com
www.b-i.com

blue-infinity headquarters
t +41(0)58.307.7000
f +41(0)58.307.7001
INTERNATIONAL
t +800.307.70.000
f +800.307.70.001

b-i  branding.technology.integration.

The information in this e-mail, and those ensuing, is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, please destroy this message and notify us immediately.
 

Please think of the environment before printing this email

 

---

From: sdm...@googlegroups.com [mailto:sdm...@googlegroups.com] On Behalf Of Bob Jolliffe
Sent: vendredi, 22. janvier 2010 11:21
To: sdm...@googlegroups.com
Subject: Periodicity and isFrequency

--
You received this message because you are subscribed to the Google Groups "SDMX-HD (Health Domain)" group.
To post to this group, send email to sdm...@googlegroups.com.
To unsubscribe from this group, send email to sdmx_hd+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sdmx_hd?hl=en.

[Bob Jolliffe]

2010/1/22 Gary Patchen <gary.p...@b-i.com>

Mind you we do already have a CL_PERIODICITY codelist which was referred to both by the PERIODICITY attribute in DSD and the FREQUENCY attribute in MSD.  So if I understand things correctly we must:
(i) remove the PERIODICITY attribute from DSD;
(ii) include instead a FREQUENCY dimension in DSD - where this dimension has isFrequency="true";
(iii) rename the common CL_PERIODICITY codelist to CL_FREQUENCY;
(iv) retain the FREQUENCY metadata attribute in the MSD but change its codelist reference to CL_FREQUENCY.

Its worth noting that the frequency dimension of a dataset is quite a different beast to the frequency attribute in the MSD.  In the former case it refers to the period type in which actual data has been reported - eg on a monthly basis or quarterly basis.  You are right that this is kind of redundant if we have a time dimension like 2010-Q1, or 2011, from which this can be inferred but I guess it makes it explicit and anyway it seems we don't have too much choice in terms of SDMX compliance.

The frequency attribute in the latter case refers to metadata for the indicator - the MSD specifically provides structure for metadata for *indicators* after all, not data.  So the presence of this attribute for an indicator would indicate that data for this particular indicator *should* be reported monthly, quarterly or what have you.  I don't think there is ambiguity in both being called FREQUENCY.

Is this clear?

Cheers
Bob

PS.  I think there is some further confusion to be sorted in the MSD regarding "frequency of collection" and "frequency of dissemination" but I'm not going there now :-(