Disaggregated results data

[Herb Caudill]

One other question on reporting results data via IATI, this one probably a bit thornier than the issue of indicator identifiers. 

Typically, implementers are asked to disaggregate reported results data in various ways. The most common attributes are demographic (gender, age, ethnicity) but there's no end to the possibilities. Just a few examples:

• Yields, disaggregated by crop type (soybeans, cassava, groundnuts, etc)
• Training, disaggregated by organization type (NGO, government agency, private firm, etc)
• Individuals reached by a public information campaign, disaggregated by media type (TV, radio, print, etc)
  
• Individuals treated, disaggregated by HIV status (positive, negative, unknown)
• Individuals treated, disaggregated by treatment regimen (sdNVP, AZT+sdNVP, ART)

It doesn't seem at all realistic to come up with any sort of universal codelist since the possibilities are literally endless; we're often forced to just match on text, with the aid of an internal thesaurus for synonyms (e.g. M = Man = Men = Male = Homme etc.)

In many legacy systems that we've dealt with, disaggregations are treated as separate indicators (e.g. "1.1a Men trained" vs
"1.1b Women trained"). This is a very messy approach and I'd hope we wouldn't have to re-introduce it here. 

Further complicating things, when data is disaggregated by more than one attribute, you have two possible approaches, both about equally common in the real world:

1. Parallel disaggregation 

Male	Female	Total	Child	Adult	Total
20	30	50	10	40	50

 

2. Cross-disaggregation (preferable because it captures more information)

Male		Female		Total
Child	Adult	Child	Adult
10	20	5	15	50

Has there been any discussion on how to incorporate disaggregation into the IATI indicator element? If not, we'd be happy to draft a proposal, since this is a problem that we've already had to work through in our own software product. 

Cheers

Herb Caudill
Founder | Chief Technology Officer
202.294.7901

DevResults: A better way to manage international aid. | www.devresults.com

[Bill Anderson]

Herb

We haven't looked at this and would welcome your input.

Bill

Sent from my iPhone

--
You received this message because you are subscribed to the
"IATI Technical" discussion list. Find out more at http://www.aidtransparency.net/governance/tag
 
To post to this group, send email to iati-te...@googlegroups.com
 
To unsubscribe from this group, send email to
iati-technica...@googlegroups.com
 
For more options, including the option to switch to a digest subscription, visit this group at http://groups.google.com/group/iati-technical
 
Tickets for the IATI technical secretariat can be posted to http://support.iatistandard.org
---
You received this message because you are subscribed to the Google Groups "IATI Technical Advisory Group (TAG) technical discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iati-technica...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

[David Megginson]

Good points, Herb.  While there is a semi-infinite number of types of disaggregation possible, I've found that there are five core disaggregation facets (called the "5W") that are pretty-much universal across different types of aid data, countries, and sectors, and those would provide good starting points for any standardisation effort:

• Who: the agent of the thing described in the data (e.g. donor, NGO, local ministry, community group, etc.)
• What: sector/subsector categorisation of what's being described in the data
• Where: geographical location/s of what's being described in the data
• When: period of time when the data applicable
• For Whom: demographic classification of the subject or beneficiary (especially SADD - sex-and-age disaggregated data)

IATI already supports expressing all of these in the standard, though we still need standards for the actual values (there is some help already in the IATI code lists).  The beneficiary classification -- especially SADD -- will be difficult to standardise globally, because it often depends on local census data, and different countries have different age-group categories, but we could at least push for in-country standards.

Cheers, David

[Catherine Marschner]

All - MCC would also like to look at publishing our results data in a greater disaggregated format, particularly by gender.  Our own systems can handle multiple disaggregation types but we just post the rolled up data to IATI. Would love to have some conversations about ways to do this!

[Bill Anderson]

Catherine (and belatedly Herb)

 

One of the main reasons for publishing  to the IATI standard is for your data to be comparable with other publishers' data. This applies as much to results as it does to geography, sectors, etc. Without commonly used indicator vocabularies reporting results through IATI has limited value. The M&E world in general appears not to appreciate the need for global standards which would allow for better comparison of results data being produced from different sources.

 

One good example of what this should look like is the World Health Organisation's Indicator and Measurement Registry which is "a central source of metadata of health-related indicators used by WHO and other organizations. It includes indicator definitions, data sources, methods of estimation and other information that allow users to get better understanding of their indicators of interest. It facilitates complete and well-structured indicator metadata, harmonization and management of indicator definitions and code lists, internet access to indicator definitions, and consistency with other statistical domains."

--

You received this message because you are subscribed to the
"IATI Technical" discussion list. Find out more at http://www.aidtransparency.net/governance/tag
 
To post to this group, send email to iati-te...@googlegroups.com
 
To unsubscribe from this group, send email to
iati-technica...@googlegroups.com
 
For more options, including the option to switch to a digest subscription, visit this group at http://groups.google.com/group/iati-technical
 
Tickets for the IATI technical secretariat can be posted to http://support.iatistandard.org
---
You received this message because you are subscribed to the Google Groups "IATI Technical Advisory Group (TAG) technical discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iati-technica...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

[Brian]

Sorry to come late to this discussion. I entirely agree with Bill’s point about the need for common standards for indicators of results. Indeed it is imperative that M&E practitioners first examine existing indicators that are well defined and often routinely collected by developing country statistical systems. This supports sustainability and comparability across countries. Projects that seek to design their own specific indicators add to data collection burdens, often ‘reinvent the wheel’ and are inherently unsustainable.

 

In addition to the WHO source that Bill quotes below, another good source of metadata is that developed for the Millennium Development Goals by an inter-agency approach encompassing all UN agencies and other international organisations. Many of the indicators are disaggregated by gender and or urban/rural – which goes a little way to address the disaggregation points made in this discussion. The MDG metadata are at: http://mdgs.un.org/unsd/mdg/Metadata.aspx . It is reasonable to assume that these indicators will continue to be used to monitor the Sustainable Development Goals that will replace the MDGs this year – and be expanded further to cover the additional goals and targets.

 

Brian Hammond

Former IATI TAG Chair and former Chair of the UN Inter-agency working group on MDG indicators

[Laia Grino]

Hi everyone,

Thanks for this interesting discussion. I am likely to be in the minority here, but I just wanted to offer a word of caution in terms of pursuing adoption of common indicators. Over the past two years, InterAction's Evaluation and Program Effectiveness Working Group has organized several sessions on agency level measurement systems. Though these systems can take various forms, one type of system entails establishing common indicators for use across an organization's programs so that they can then be aggregated and used to assess effectiveness beyond the project level. For various reasons, several organizations have ended up abandoning such systems. One major issue is that having data on indicators without greater understanding of the context - why something happened - isn't that useful. There is also a real tension between greater standardization and ownership (including ownership beyond the government level).

None of this is too say that organizations shouldn't share information on their results. It is more a point that we should be careful about how we design the results part of the IATI standard.

Best,

Laia

[Herb Caudill]

Laia's points are all well taken. I just want to make it clear that what I'm proposing isn't that IATI choose standard indicators. That would not only be completely out of IATI's remit but would be utterly unfeasible in practice. 

What we do need is (1) way to uniquely identify existing indicators in use by organizations, and (2) a schema for defining what they represent. (It would be up to any given organization to decide whether they want to standardize at the organizational level, or just catalog project-level indicators.) Either way, without those two things, the whole idea of using the IATI format to report results is a non-starter. 

FWIW my proposed schema changes are here:

• Performance reporting in IATI http://blog.devresults.com/iati-results-standard/
• Towards an IATI indicator definition schema http://blog.devresults.com/proposed-indicator-definition-schema/

Cheers

Herb

[David Megginson]

I agree with Laia that we're looking at a balancing act (as with all data standards): organisations' current indicators are optimised for the work they need to do, and we cannot expect them to throw them out simply for the convenience of data scientists. On the other hand, where we can show real and persuasive value, we might be able to get people to move to some common indicators in areas where the differences are mostly trivial and unnecessary.

Here are two good existing catalogues of common humanitarian indicators, the first developed with wide participation of the NGO sector, and the second developed by the NGO sector (via Sphere):

• Indicators Registry
  
• Sphere for Assessments

Cheers, David

On Thu, Mar 26, 2015 at 10:46 AM, Laia Grino <lgr...@interaction.org> wrote: