@last-updated-datetime : Are there really these massive updates all the time?

[ckreutz]

Hello!

I worked with the data store the past store. Great work! What a joy to get JSON data. ;-)

However, I noticed one thing: There are incredible many updates most of the time I wonder whether these are "real" updates. 

In the standard it say:

@last-updated-datetime

The last date/time that the data for this specific activity was updated. This date must change whenever the value of any field changes.

When I query the API for example with http://datastore.iatistandard.org/api/1/access/activity?last-updated-datetime__gt=2015-02-23&last-updated-datetime__lt=2015-02-25&limit=500

I get way more than 500 updates within two days. I checked similar time ranges and often as many updates happen. 

Can someone tell me if theses are really changes to the data or timestamp changes for all data if only one activity is updated?

Thanks in advance!

Best Christian 

[david C]

Hi Christian

I'm sorry you haven't had an answer to your question. I was thinking someone else might jump in...

AFAIK the last-updated-datetime attribute was included to allow publishers to notify the rest of us that an activity has changed. How this is implemented is not necessarily consistent across the data.

Much of it has to do with how publishers generate their activity data. Some have them connected to live systems, so that when people update a record on the system it triggers updates in the IATI output. Others hand craft their IATI files once a year.

I'm not surprised at 500 updates in a two day period across the entire dataset. (approx 350,000)

You might also find our Dashboard useful:

http://dashboard.iatistandard.org/

as a way of getting a handle on the state of the data.

Hope that is of some use.

All the best

David Carpenter

--
You received this message because you are subscribed to the
"IATI Technical" discussion list. Find out more at http://www.aidtransparency.net/governance/tag
 
To post to this group, send email to iati-te...@googlegroups.com
 
To unsubscribe from this group, send email to
iati-technica...@googlegroups.com
 
For more options, including the option to switch to a digest subscription, visit this group at http://groups.google.com/group/iati-technical
 
Tickets for the IATI technical secretariat can be posted to http://support.iatistandard.org
---
You received this message because you are subscribed to the Google Groups "IATI Technical Advisory Group (TAG) technical discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iati-technica...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Kreutz]

Hi David,

thanks a lot for your reply. That explains a lot. 

Dashboard is fantastic. Use it all the time to make sense of data. :-)

Best Christian 

 

 

 

 

 

On 23.03.2015 16:15:17, david C <capr...@gmail.com> wrote:

---

[Steven Flower]

Hi Christian

I wonder if the timeliness tab on the dashboard can help too: http://dashboard.iatistandard.org/timeliness.html.  This looks for an updated transaction date - we can see that Global Giving, for example, have an update most days.  NB: this is a count of the number of days that a change was observed, not the number of files or activities or transactions.  However, it does give some indication of the velocity of updates

Thanks

Steven

[Christian Kreutz]

Hi Steven,

that is really great. Have not seen it before and explains it now even better. :-)

I have another question in this regard and someone might have an answer? 

We intent to import all activities in our database, but then update only changes. 

But we wonder how we can identify new transactions in particular? 

Transactions have no identifier, and only some a „ref“. Most have a date, but there could be more than one transaction per day. 

If we do not want to import each month the complete dump and incrementally update only new contributions, I wonder how can that be possible in regards to transactions etc.? 

So far I can only see this working if I store the number of transactions and update if there is a new one, but then I miss changed amounts of previous transactions. 

I really hope there is a better solution than to import the whole dump each month. Thanks a lot in advance for any hints!

Best Christian 

[David Megginson]

It might be useful to track two dates:

• Last updated -- the last time the data provider asserts that the data was valid.
• Last changed -- the last time the data actually changed.

The first date comes from the provider; the second date, we could calculate ourselves, by comparing the updated data to the previous data and noting when it has actually changed.

Cheers, David

[Steven Flower]

It's probably also worth asking "what's an update?"?

As mentioned, that dashboard page looks for updated transaction dates, in order to register an update: http://dashboard.iatistandard.org/timeliness.html.  This particular use case would therefore not log new activities (with no transactions); changes to budgets; results or new documents - for example.  

The broad thinking might be that a typical publisher activity would be set up some core/stable elements (activity ID; title; description, country; participating-org, sector etc) that might not be subject to change.  Other aspects might be reflective of the point in time being reported, but could change - activity-status; activity-date.  And then - one could consider elements such as budget, transaction, document-link and results as (potentially) being more volatile - either through updates or additional elements.

Finally, there are then the situations when a publisher may (exceptionally) reconfigured their publishing approach and so wider changes occur - more/less activities/files; new URLs, etc. These are also valid changelog entries.

If you didn't know, the IATI Registry runs a script every night to look for a "change" in a file.  If it spots one, it logs this in the publisher and file history, via the "IATI Harvester Script" - eg: http://iatiregistry.org/publisher/activity/unicef.  However, this change isn't descriptive, or appreciative of any of the above considerations.

So - changes in IATI are interesting and complex, but a good sign of a data standard maturity...

Best wishes

Steven