IATI approaching the Big Data Chasm? (was Re: [IATI Tech] Data Size Notice)

[David Megginson]

Thanks for this posting, Ben. This is an interesting problem: as IATI becomes more successful and the data pipeline gets fatter, we might be approaching the edge of what I'll call the big data chasm, a sudden discontinuity where traditional approaches (like centralised relational databases) can't keep up. This is good — it means IATI's succeeding — but it also means we'll have to rethink some of our technical infrastructure, including how we use software like CKAN and its DataStore.

Please keep us informed.

Cheers, David

On Wed, Jun 25, 2014 at 12:20 PM, Ben Webb <bjwe...@googlemail.com> wrote:

We've become aware that the data published by the United States ( http://iatiregistry.org/publisher/unitedstates )  has increased in size recently from about 1GB to over 3GB - this brings the total size of the IATI dataset to over 4GB.

We just wanted to share this information so that anyone consuming data on an overnight, or automatic, process is forewarned. This may or may not affect your processes.

Some of the new files are too big for the Registry to process (e.g. http://iatiregistry.org/dataset/unitedstates-ht which is 112MB, whilst the registry has a hard limit of 50MB). Tools that rely on the Registry's update metadata (such as the Datastore), will not yet have updated the activities in such files.

Additionally, the Dashboard's automatic data download process failed today, since the server ran out of disk space, so the Dashboard website is a day out of date.

Best regards,

IATI Technical Team

--
You received this message because you are subscribed to the
"IATI Technical" discussion list. Find out more at http://www.aidtransparency.net/governance/tag
 
To post to this group, send email to iati-te...@googlegroups.com
 
To unsubscribe from this group, send email to
iati-technica...@googlegroups.com
 
For more options, including the option to switch to a digest subscription, visit this group at http://groups.google.com/group/iati-technical
 
Tickets for the IATI technical secretariat can be posted to http://support.iatistandard.org
---
You received this message because you are subscribed to the Google Groups "IATI Technical Advisory Group (TAG) technical discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iati-technica...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.