Dealing with "the data problem"

10 views
Skip to first unread message

MERL Tech Forum

unread,
Mar 28, 2017, 5:36:05 AM3/28/17
to MERL Tech Forum
Reading Emily Tomkys' blogpost:  http://merltech.org/we-have-a-data-problem/ 

Oxfam is trying to "design a data hub where no matter what the type of data (unstructured, semi-structured or structured) and no matter how we collect the data (mobile data collection tools or on paper), our data can integrate into a database. This isn’t about creating new tools – rather it’s about focusing on the interoperability and smooth transition between tools and storage options.  We plan to set this up so data can be pulled through into a reporting layer which may have a mixture of options for quantitative analysis, qualitative analysis and GIS mapping. We also know we need to give our micro-programme data a home and put everything in one place regardless of its source or format and make it easy to pull it through for analysis."

Curious how other organizations are dealing with the "data problem." Thoughts anyone?


Robert Worthington

unread,
Mar 28, 2017, 7:39:52 AM3/28/17
to MERL Tech Forum
Hi Emily,

I assume you’ve looked at the work being done on http://frictionlessdata.io/ ?  This is a very light-weight approach that uses a standardised ‘container format’ to describe and package data.  I’ve not followed this project for a while, so cannot add much more.

Our work at Kwantu has taken us in a similar direction.  We work with NGOs and governments to build systems to manage both operational and performance data.  The interoperability challenge you mention is a challenge that comes up again and again - either in the context of multiple gov departments (all of which need to share data) or the network context in which many NGOs now work to deliver programmes.

Over the last five or six years we’ve been developing and testing the following approach:

Data objects

Similar to the container format used by Frictionless data, this goes a bit further.  Our data objects are made up of the following components:

- Standard data envelope - this ensures that all data is curated in the same way.  This includes geo-coding, date/time and so on.
- Data model - this is unique, but the data model is built up using a standard set of elements
- Schema - this validates the data model and ensures that the business rules associated with it are incorporated
- View and edit model - this ensures that the data is de-linked from the application that created it, making it portable

We’ve build a domain specific language that enables us to create new data objects using an XML configuration file.  At some point this will be replaced by a visual editor.  Since the data object is defined pragmatically it can be transformed into any relevant file format.  We use JSON as that enables data objects (and their data) to be stored locally and hence work offline.

This approach means that data in other systems can be mapped to a data object and transformed into the data object format.  Hence they act either as a way of standardising how new data is collected and/or to harmonise data across existing systems.

Data governance

We have more work to do on the data governance side, but can share some of our thinking.

We’ve built a prototype Data Registry that enables different organisations to create an App.  This is a container for data objects and workflow.  They may then upload data objects, which the Data Registry will transform into JSON.  Data objects may be shared across apps, encouraging standardisation where possible.  There is then a build and release process for apps, which are then accessible to use by end users via a browser in a self-contained web-app.

Once released and in use an app may choose to share data with others using the Data Registry (or not).  This allows for the possibility of aggregating data across different organisations using either the same App or different Apps that share the same data objects.  End to end encryption is part of our planning, but not there yet.  Once data has been aggregated like this it can then be pushed into a data warehouse for reporting.

In our work with varied NGOs and government departments we also see many similarities at a low, operational level in the types of data objects used.  For example, data objects related to projects, people, groups, meetings, outcomes, action plans etc.  This raises the interesting question of how governments and NGO networks working in a specific sector could facilitate the bottom-up development of data standards.

We’re exploring this question at the moment with World Vision and CARE around work on social accountability, with a view to enabling both organisations to share data from similar programmes.

This is a far more intensive approach than Frictionless data, so may not be appropriate for what you are planning.  Either way I’m happy to share lessons and thinking if it’s helpful.

These presentations may also be useful background (although need updating now):


Best,

Rob



--
The MERL Tech Forum is an online space for discussion, learning, and sharing experiences and challenges related to digital data and technology in Monitoring, Evaluation, Research and Learning (MERL) in the social impact & international development and humanitarian fields. We focus on five key areas (organizational, methodological, technological, operational and ethical) that require attention when using digital technology in MERL and when data is digital. Please contact Linda Raftree with any queries at lindar...@gmail.com
---
You received this message because you are subscribed to the Google Groups "MERL Tech Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to MERLTech_Foru...@googlegroups.com.
To post to this group, send email to MERLTec...@googlegroups.com.
Visit this group at https://groups.google.com/group/MERLTech_Forum.
To view this discussion on the web visit https://groups.google.com/d/msgid/MERLTech_Forum/01d7fdb5-dd3b-49b3-a768-fe9d574aedd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Courtney Roberts

unread,
Mar 28, 2017, 7:39:52 AM3/28/17
to MERL Tech Forum
I would recommend applying machine learning tools to help with classification. Moonshot's Global (www.moonshotglobal.com) helps firms do this, and saves time and debate on taxonomy development.

Contact me if you'd like more information.

Courtney Roberts 
Principal
Moonshot Global
Cour...@moonshotglobal.com

Sent from my iPhone

Joshua Powell

unread,
Mar 28, 2017, 5:19:33 PM3/28/17
to Courtney Roberts, MERL Tech Forum
In addition to these great resources, I know that Akvo is doing some work on this through their Lumen project (formerly DASH). It may be worth getting in touch with Henry Jewell to learn more, as well.

Best,
Josh


On Tue, Mar 28, 2017 at 6:11 AM, Courtney Roberts <cour...@moonshotglobal.com> wrote:
I would recommend applying machine learning tools to help with classification. Moonshot's Global (www.moonshotglobal.com) helps firms do this, and saves time and debate on taxonomy development.

Contact me if you'd like more information.

Courtney Roberts 
Principal
Moonshot Global
Cour...@moonshotglobal.com

Sent from my iPhone

On Mar 28, 2017, at 5:36 AM, MERL Tech Forum <MERLTech_Forum@googlegroups.com> wrote:

Reading Emily Tomkys' blogpost:  http://merltech.org/we-have-a-data-problem/ 

Oxfam is trying to "design a data hub where no matter what the type of data (unstructured, semi-structured or structured) and no matter how we collect the data (mobile data collection tools or on paper), our data can integrate into a database. This isn’t about creating new tools – rather it’s about focusing on the interoperability and smooth transition between tools and storage options.  We plan to set this up so data can be pulled through into a reporting layer which may have a mixture of options for quantitative analysis, qualitative analysis and GIS mapping. We also know we need to give our micro-programme data a home and put everything in one place regardless of its source or format and make it easy to pull it through for analysis."

Curious how other organizations are dealing with the "data problem." Thoughts anyone?


--
The MERL Tech Forum is an online space for discussion, learning, and sharing experiences and challenges related to digital data and technology in Monitoring, Evaluation, Research and Learning (MERL) in the social impact & international development and humanitarian fields. We focus on five key areas (organizational, methodological, technological, operational and ethical) that require attention when using digital technology in MERL and when data is digital. Please contact Linda Raftree with any queries at lindar...@gmail.com
---
You received this message because you are subscribed to the Google Groups "MERL Tech Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to MERLTech_Forum+unsubscribe@googlegroups.com.
To post to this group, send email to MERLTech_Forum@googlegroups.com.

--
The MERL Tech Forum is an online space for discussion, learning, and sharing experiences and challenges related to digital data and technology in Monitoring, Evaluation, Research and Learning (MERL) in the social impact & international development and humanitarian fields. We focus on five key areas (organizational, methodological, technological, operational and ethical) that require attention when using digital technology in MERL and when data is digital. Please contact Linda Raftree with any queries at lindar...@gmail.com
---
You received this message because you are subscribed to the Google Groups "MERL Tech Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to MERLTech_Forum+unsubscribe@googlegroups.com.
To post to this group, send email to MERLTech_Forum@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/MERLTech_Forum/8A43FAA7-AD73-4C74-85EA-C5103C400694%40moonshotglobal.com.

Emily Tomkys

unread,
Apr 11, 2017, 10:49:35 AM4/11/17
to Joshua Powell, Courtney Roberts, MERL Tech Forum
Thanks all for your input - I will keep you posted on how we develop our ideas over coming months.

Emily
_________________________________________
Emily Tomkys | ICT in Programme Officer - PMEAL
Oxfam GB | Oxford | +44 (0) 1865 47 3720 | skype: emilytomkys | Twitter: @emilytomkys






From:        Joshua Powell <jpo...@developmentgateway.org>
To:        Courtney Roberts <cour...@moonshotglobal.com>
Cc:        MERL Tech Forum <MERLTec...@googlegroups.com>
Date:        28/03/2017 22:19
Subject:        Re: [MERL Tech] Dealing with "the data problem"
Sent by:        merltec...@googlegroups.com



In addition to these great resources, I know that Akvo is doing some work on this through their Lumen project (formerly DASH). It may be worth getting in touch with Henry Jewell to learn more, as well.

Best,
Josh
Joshua Powell
 Chief Strategy Officer

jpo...@developmentgateway.org 
Cell: +1 410-960-5423 
1110 Vermont Avenue NW, Suite 500
Washington, DC 20005
www.developmentgateway.org

On Tue, Mar 28, 2017 at 6:11 AM, Courtney Roberts <cour...@moonshotglobal.com> wrote:
I would recommend applying machine learning tools to help with classification. Moonshot's Global (www.moonshotglobal.com) helps firms do this, and saves time and debate on taxonomy development.

Contact me if you'd like more information.

Courtney Roberts 
Principal
Moonshot Global
301-256-4559
Cour...@moonshotglobal.com

Sent from my iPhone

On Mar 28, 2017, at 5:36 AM, MERL Tech Forum <MERLTec...@googlegroups.com> wrote:
Reading Emily Tomkys' blogpost:  http://merltech.org/we-have-a-data-problem/ 

Oxfam is trying to "design a data hub where no matter what the type of data (unstructured, semi-structured or structured) and no matter how we collect the data (mobile data collection tools or on paper), our data can integrate into a database. This isn’t about creating new tools – rather it’s about focusing on the interoperability and smooth transition between tools and storage options.  We plan to set this up so data can be pulled through into a reporting layer which may have a mixture of options for quantitative analysis, qualitative analysis and GIS mapping. We also know we need to give our micro-programme data a home and put everything in one place regardless of its source or format and make it easy to pull it through for analysis."

Curious how other organizations are dealing with the "data problem." Thoughts anyone?

--
The MERL Tech Forum is an online space for discussion, learning, and sharing experiences and challenges related to digital data and technology in Monitoring, Evaluation, Research and Learning (MERL) in the social impact & international development and humanitarian fields. We focus on five key areas (organizational, methodological, technological, operational and ethical) that require attention when using digital technology in MERL and when data is digital. Please contact Linda Raftree with any queries at lindar...@gmail.com


---
You received this message because you are subscribed to the Google Groups "MERL Tech Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to MERLTech_Foru...@googlegroups.com.
To post to this group, send email to MERLTec...@googlegroups.com.
Visit this group at https://groups.google.com/group/MERLTech_Forum.


To view this discussion on the web visit

https://groups.google.com/d/msgid/MERLTech_Forum/01d7fdb5-dd3b-49b3-a768-fe9d574aedd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
The MERL Tech Forum is an online space for discussion, learning, and sharing experiences and challenges related to digital data and technology in Monitoring, Evaluation, Research and Learning (MERL) in the social impact & international development and humanitarian fields. We focus on five key areas (organizational, methodological, technological, operational and ethical) that require attention when using digital technology in MERL and when data is digital. Please contact Linda Raftree with any queries at lindar...@gmail.com


---
You received this message because you are subscribed to the Google Groups "MERL Tech Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to MERLTech_Foru...@googlegroups.com.
To post to this group, send email to MERLTec...@googlegroups.com.
Visit this group at https://groups.google.com/group/MERLTech_Forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/MERLTech_Forum/8A43FAA7-AD73-4C74-85EA-C5103C400694%40moonshotglobal.com.

For more options, visit https://groups.google.com/d/optout.

--
The MERL Tech Forum is an online space for discussion, learning, and sharing experiences and challenges related to digital data and technology in Monitoring, Evaluation, Research and Learning (MERL) in the social impact & international development and humanitarian fields. We focus on five key areas (organizational, methodological, technological, operational and ethical) that require attention when using digital technology in MERL and when data is digital. Please contact Linda Raftree with any queries at lindar...@gmail.com
---
You received this message because you are subscribed to the Google Groups "MERL Tech Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to MERLTech_Foru...@googlegroups.com.
To post to this group, send email to MERLTec...@googlegroups.com.
Visit this group at https://groups.google.com/group/MERLTech_Forum.
To view this discussion on the web visit https://groups.google.com/d/msgid/MERLTech_Forum/CA%2Bxx5Ctu0YESxXwHSh7DyJr4vCObEo1jr7mmo84odVYTTYET4A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.





Oxfam works with others to overcome poverty and suffering

Oxfam GB is a member of Oxfam International and a company limited by guarantee registered in England No. 612172.
Registered office: Oxfam House, John Smith Drive, Cowley, Oxford, OX4 2JY.
A registered charity in England and Wales (no 202918) and Scotland (SC 039042)

Reply all
Reply to author
Forward
0 new messages