API Standardisation - discussion starter

[Tim Davies]

Hello all,

In the discussion part of the Open Aid Flows session at OKFest in Helsinki (video of presentations here http://bambuser.com/v/2993098) there was some discussion of API Standardisation. Building on that, and discussion in Cookham and hosted by the UK Government Digital Service recently, I've put together some draft working notes on approaches to agree a standard pattern for REST APIs onto IATI data.

The goal of this would be to have a common approach, so that front-end applications can be build agnostic about the backend that supplies them with data. This is becoming increasingly important as the amount of IATI data grows, and it may become necessary to use a range of backend tools to manage search and retrieval of activity information; aggregation and so-on. 

In the draft notes at http://wiki.iatistandard.org/roadmap/api_standardisation I've set out possible approaches. This is very much a first draft, and in need of comments and input. Please do either add to the wiki directly, or share thoughts here.

Some of the main issues to work out are:

• Approaches to determining parameters that a RESTFul API would use. E.g. to fetch a list of activities which have commitment transactions would you use URIs like 
  /access/activities?condition=transactions/transaction/@type='C'
  or
  /access/activities?transactions_transaction__type=C
  
  
• Approaches to serialising IATI data as JSON

All input welcome,

All the best

Tim

-- 

http://www.timdavies.org.uk

07834 856 303.

@timdavies

Co-director of Practical Participation: http://www.practicalparticipation.co.uk

--------------------------

Practical Participation Ltd is a registered company in England and Wales - #5381958. 

[Bill Anderson]

Hi Tim

 

Excellent stuff. I agree fully with the concept, architecture and design. Will leave it to the more technically literate to engage on the details.

 

Convenience elements mentioned in approach 3 will make everything a lot more usable – a must for me...

 

Bill

--
You received this message because you are subscribed to the
"IATI Technical" discussion list. Find out more at http://wiki.iatistandard.org/community/mailing_list
 
To post to this group, send email to iati-te...@googlegroups.com
 
To unsubscribe from this group, send email to
iati-technica...@googlegroups.com
 
For more options, including the option to switch to a digest subscription, visit this group at http://groups.google.com/group/iati-technical
 
Tickets for the IATI technical secretariat can be posted to http://support.iatistandard.org
 
 

[Dan Mihaila]

Hello Tim,
This is quite interesting and as you said it is already a must since Cookham meeting and I admit that a common standard (IATI) must have a common interface (IATI API) in order to be accessible or to provide information in a widely accepted format no matter what is the IT system behind.
The starting draft looks very good, but I think it is missing the scenarios: 7-10 queries (samples or better used queries by different organizations) about IATI data stored in a system. I think we can collect this queries (with different grades of difficulty) quite easy and fast and then we can suggest what is the best approach regarding the "REST" style to be used.What do you think?

Cheers,
 Dan

Dan Mihaila, IT Consultant
Development Gateway International
49 Rue de Treves (Box 7) • 1040 Brussels, BELGIUM
(M) +40 722 502 304 • (GTalk) dan.m...@gmail.com • (Skype) carcotelul
• (MSN) dan.m...@hotmail.com • (Yahoo) carcotelul
dmih...@dginternational.org
www.developmentgateway.org

Information Tools. Global Partnerships. Effective Aid.

--

[Tim Davies]

Hello Dan,

You make a good point. It would be good to collect on that page a list of common queries that applications might want.

We brainstormed some of this with Government Digital Service in the UK a few months back. John: are there notes from that meeting available we could share?

All the best

Tim

[david C]

Under Candidate Patterns - Special parameters

(see http://wiki.iatistandard.org/roadmap/api_standardisation#candidate_patterns )

It looks like we have parameters that effect the query and/or the response, and these might need breaking up a bit.

So ?fields for example, could be used to restrict the fields searched (e.g for a free text search) AND to effect the response - i.e. determine which fields are returned in the response. So I'm just wondering if we need to set those up as query parameters and response parameters.

(I also wonder if fields as given needs All, Include, Exclude, and a couple of summary options)

Its not entirely straight in my head how we deal with e.g. 'title' fields that can be at various levels of an activity document (do I want to include/exclude title fields other than the main one), and similarly if I wanted to do attribute level searches, but I think that is partly because I don't have concrete examples of searches I'd really want to do. (So I think Dan's idea is good)

Anyway, hope that's of some use.

Cheers

David

[John Adams]

All

 

Here are the straight user stories captured from that GDS workshop that Tim mentioned. Feel free to put them onto Google Docs or somewhere more shareable. And I’d be really happy for others to chip in and improve them!

 

Thanks

 

John

 

From: tim.g....@gmail.com [mailto:tim.g....@gmail.com] On Behalf Of Tim Davies

Sent: 02 October 2012 00:25
To: iati-te...@googlegroups.com
Cc: John Adams

Subject: Re: [IATI Tech] API Standardisation - discussion starter

 

Hello Dan,

 

You make a good point. It would be good to collect on that page a list of common queries that applications might want.

 

We brainstormed some of this with Government Digital Service in the UK a few months back. John: are there notes from that meeting available we could share?

 

All the best

 

Tim

On Mon, Oct 1, 2012 at 7:07 AM, Dan Mihaila <danmi...@gmail.com> wrote:

Hello Tim,
This is quite interesting and as you said it is already a must since Cookham meeting and I admit that a common standard (IATI) must have a common interface (IATI API) in order to be accessible or to provide information in a widely accepted format no matter what is the IT system behind.
The starting draft looks very good, but I think it is missing the scenarios: 7-10 queries (samples or better used queries by different organizations) about IATI data stored in a system. I think we can collect this queries (with different grades of difficulty) quite easy and fast and then we can suggest what is the best approach regarding the "REST" style to be used.What do you think?

Cheers,
 Dan

 

Dan Mihaila, IT Consultant
Development Gateway International
49 Rue de Treves (Box 7) • 1040 Brussels, BELGIUM
(M) +40 722 502 304 • (GTalk) dan.m...@gmail.com • (Skype) carcotelul
• (MSN) dan.m...@hotmail.com • (Yahoo) carcotelul
dmih...@dginternational.org
www.developmentgateway.org

Information Tools. Global Partnerships. Effective Aid.

________________________________________
UK aid is helping the world's poorest people change their lives.
www.dfid.gov.uk/changinglives

[Siem Vaessen]

Hi Tim, all,

Thanks for setting that up. I believe finding a general consencus on how an (IATI) API should look is important.

I mainly relate from the API we have been building at OIPA, so my main reference and context resides within OIPA. Just a recap on some of the RESTfull the API in OIPA has:

___________________________________________

http://oipa.openaidsearch.org/api/v2/activities/?format=json&iati_identifier=NL-1-PPR-22163   IATI identifier

http://oipa.openaidsearch.org/api/v2/activities/?format=json&organisations=41120  IATI publisher (UN-HABITAT in this case)

http://oipa.openaidsearch.org/api/v2/activities/?format=json&organisations=GB-CHC-202918  (OXFAM) Receiving Org

http://oipa.openaidsearch.org/api/v2/activities/?format=json&organisations=NL-1&countries=BA (DGIS + country Bosnia Herzegovina)

http://oipa.openaidsearch.org/api/v2/activities/?format=json&query=AIDS  General query

http://oipa.openaidsearch.org/api/v2/activities/?format=json&sectors=410  Sectors

http://oipa.openaidsearch.org/api/v2/activities/?format=json&query=water&countries=YE|EG|MN&sectors=14010  Query + Countries + Sectors

http://oipa.openaidsearch.org/api/v2/activities/?format=json&statistics__total_budget__gte=1000000  Budgets

___________________________________________

Besides those queries we will be mapping other OIPA specific resources (which are mapped from the IATI namespaces) this quarter as well: 

http://oipa.openaidsearch.org/api/v2/docs/resources/  (list of resources stored, but not all of them implemented as a filter yet)

The ambition of the API in OIPA is to map any available resource from the IATI standard and make them available in a query and combine those queries. So from our point of view an API *must* be able to query anything a human or computer can come up with, otherwise the API wil limit information and/or a combination of information (retrieval).

The one main issue we will be dealing with the next few months is optimising speed, since querying 500K transactions records on some queries can be a pain. But I guess querying, storing (& precompiling) is not a specific API requirement, although it should be. 

Siem 

[Dan Mihaila]

Hello all,
This looks very interesting and probably we could collect more than 10 sample queries with different parameters.
I see that every example is about returning a collection of activities. I assume similar we could receive as result a collection of other IATI entities such as: list of sectors (pair: name-sector code) where DFID has projects for DRC.
What do you think?

Dan

Dan Mihaila, IT Consultant
Development Gateway International
49 Rue de Treves (Box 7) • 1040 Brussels, BELGIUM
(M) +40 722 502 304 • (GTalk) dan.m...@gmail.com • (Skype) carcotelul
• (MSN) dan.m...@hotmail.com • (Yahoo) carcotelul
dmih...@dginternational.org
www.developmentgateway.org

Information Tools. Global Partnerships. Effective Aid.

--

[Siem Vaessen]

Hi Dan,

Yes, OIPA will move towards supporting any query that we believe is necessary. We just have not had the time/resources to do this yet. 

I believe retrieving information from a systems should be just about that: retrieving anything that may be of interest to a user. 

The first scope of OIPA was indeed geared towards activities, but we'll have a drilldown to any other IATI entity. Or at least that is the ambition. If you have the time you may want to chip in some thinking here?

Siem

[Simon Whitehouse]

Hi Folks

Siem, do you mean that I could construct a query such as

"Show me the activities with transaction values of �10,000 to �15,000
for the second quarter of 2010 that have a sector category of Education,
group this by donor organisation"?

I suppose what I'm getting at is - what is the trade off between
allowing for all possible queries to be made through an API and its
performance?

The reason for me asking this is because I wonder if defining a smaller
subset of queries that can be handled more efficiently might be a more
optimal approach, at least initially.

I'm not a software developer, so am asking from a more theoretical point
of view.

All the best
Simon
@siwhitehouse
+44 7545 018285
skypeID: siwhitehouseuk

On 04/10/2012 15:44, Siem Vaessen wrote:
> Hi Dan,
>
> Yes, OIPA will move towards supporting any query that we believe is
> necessary. We just have not had the time/resources to do this yet.
>
> I believe retrieving information from a systems should be just about
> that: retrieving anything that may be of interest to a user.
>
> The first scope of OIPA was indeed geared towards activities, but we'll
> have a drilldown to any other IATI entity. Or at least that is the
> ambition. If you have the time you may want to chip in some thinking here?
>
> Siem
>
> On Wednesday, October 3, 2012 4:38:24 PM UTC+2, Dan Mihaila wrote:
>
> Hello all,
> This looks very interesting and probably we could collect more than
> 10 sample queries with different parameters.
> I see that every example is about returning a collection of
> activities. I assume similar we could receive as result a collection
> of other IATI entities such as: list of sectors (pair: name-sector
> code) where DFID has projects for DRC.
> What do you think?
>
> Dan
>

> *Dan Mihaila*, IT Consultant
> *Development Gateway International*
> 49 Rue de Treves (Box 7) � 1040 Brussels, BELGIUM
> (M) +40 722 502 304 � (GTalk) dan.m...@gmail.com <javascript:>�
> (Skype) carcotelul
> � (MSN) dan.m...@hotmail.com <javascript:>� (Yahoo) carcotelul
> dmih...@dginternational.org <javascript:>
> www.developmentgateway.org <http://www.developmentgateway.org/>
>
> */Information Tools. Global Partnerships. Effective Aid. /*

> <http://oipa.openaidsearch.org/api/v2/activities/?format=json&query=water&countries=YE%7CEG%7CMN&sectors=14010>

> http://bambuser.com/v/__2993098
> <http://bambuser.com/v/2993098>) there was some discussion

> of API Standardisation. Building on that, and discussion in
> Cookham and hosted by the UK Government Digital Service
> recently, I've put together some draft working notes on
> approaches to agree a standard pattern for REST APIs onto
> IATI data.
>
> The goal of this would be to have a common approach, so that
> front-end applications can be build agnostic about the
> backend that supplies them with data. This is becoming
> increasingly important as the amount of IATI data grows, and
> it may become necessary to use a range of backend tools to
> manage search and retrieval of activity information;
> aggregation and so-on.
>
> In the draft notes at

> http://wiki.iatistandard.__org/roadmap/api___standardisation

> <http://wiki.iatistandard.org/roadmap/api_standardisation>
> I've set out possible approaches. This is very much a first
> draft, and in need of comments and input. Please do either
> add to the wiki directly, or share thoughts here.
>
> Some of the main issues to work out are:
>

> * Approaches to determining parameters that a RESTFul API

> would use. E.g. to fetch a list of activities which have
> commitment transactions would you use URIs like

> /access/activities?condition=__transactions/transaction/@__type='C'
> or
> /access/activities?__transactions_transaction____type=C
>
> * Approaches to serialising IATI data as JSON

>
> All input welcome,
>
> All the best
>
> Tim
>
>
> --
>
>
>

> http://www.timdavies.org.uk <http://www.timdavies.org.uk/>

>
> 07834 856 303.
>
> @timdavies
>
>
> Co-director of Practical Participation:

> http://www.__practicalparticipation.co.uk
> <http://www.practicalparticipation.co.uk/>

>
> --------------------------
>
> Practical Participation Ltd is a registered company in
> England and Wales - #5381958.
>
> --
> You received this message because you are subscribed to the
> "IATI Technical" discussion list. Find out more at
> http://wiki.iatistandard.org/community/mailing_list
> <http://wiki.iatistandard.org/community/mailing_list>
>
> To post to this group, send email to iati-te...@googlegroups.com

> <javascript:>

>
> To unsubscribe from this group, send email to

> iati-technica...@googlegroups.com <javascript:>

>
> For more options, including the option to switch to a digest
> subscription, visit this group at
> http://groups.google.com/group/iati-technical

> <https://groups.google.com/group/iati-technical>

>
> Tickets for the IATI technical secretariat can be posted to

> http://support.iatistandard.org <http://support.iatistandard.org>

[Dan Mihaila]

Hi Simon,

I think you are right. Maybe we should not allow all kind of queries (or at least not guarantee a good performance). This is something that need to be discussed more probably.

 I assume after getting the projects filtered by a simple query (such as "all projects that have a sector category of Education, group this by donor organisation")  we could upload this feed to a (analyse) tool that can perform the other transformation we need such as:

- normalizing values

- transformation of the currency to a common currency

- checking the dates (to perform computation for the desired quarter)

- visualization :)

 
Dan Mihaila, IT Consultant
Development Gateway International
49 Rue de Treves (Box 7) • 1040 Brussels, BELGIUM
(M) +40 722 502 304 • (GTalk) dan.m...@gmail.com • (Skype) carcotelul
• (MSN) dan.m...@hotmail.com • (Yahoo) carcotelul
dmih...@dginternational.org
www.developmentgateway.org

Information Tools. Global Partnerships. Effective Aid.

On Oct 5, 2012, at 10:49 AM, Simon Whitehouse <s...@siwhitehouse.co.uk> wrote:

Hi Folks

Siem, do you mean that I could construct a query such as

"Show me the activities with transaction values of €10,000 to €15,000 for the second quarter of 2010 that have a sector category of Education, group this by donor organisation"?

   49 Rue de Treves (Box 7) • 1040 Brussels, BELGIUM
   (M) +40 722 502 304 • (GTalk) dan.m...@gmail.com <javascript:>•
   (Skype) carcotelul
   • (MSN) dan.m...@hotmail.com <javascript:>• (Yahoo) carcotelul

[Simon Whitehouse]

Hi folks

Going back to the first post as I don't think we've covered this so far.

How should an API handle requests that specify different languages?

Canada, for instance, require a bilingual approach with English and
French catered for. Somebody else might want to have all responses made
only in French, or Spanish.

A couple of considerations spring to mind.

Firstly, there is the question of codelist lookups. So, should a request
be able to specify a language other than English and have the Sector,
for example, returned in that language?

Secondly, if a language other than English is specified should an API
return data in that language where it exists. As an example, a Canadian
file could contain two descriptions, one in English and the other in
French. Should an API have the ability to only return the French version
if requested, or should everything be returned and the consumer left to
manage the languages?

All the best
Simon
@siwhitehouse
+44 7545 018285
skypeID: siwhitehouseuk

On 01/10/2012 11:07, Tim Davies wrote:
> Hello all,
>
> In the discussion part of the Open Aid Flows session at OKFest in
> Helsinki (video of presentations here http://bambuser.com/v/2993098)
> there was some discussion of API Standardisation. Building on that, and
> discussion in Cookham and hosted by the UK Government Digital Service
> recently, I've put together some draft working notes on approaches to
> agree a standard pattern for REST APIs onto IATI data.
>
> The goal of this would be to have a common approach, so that front-end
> applications can be build agnostic about the backend that supplies them
> with data. This is becoming increasingly important as the amount of IATI
> data grows, and it may become necessary to use a range of backend tools
> to manage search and retrieval of activity information; aggregation and
> so-on.
>
> In the draft notes at
> http://wiki.iatistandard.org/roadmap/api_standardisation I've set out
> possible approaches. This is very much a first draft, and in need of
> comments and input. Please do either add to the wiki directly, or share
> thoughts here.
>
> Some of the main issues to work out are:
>

> * Approaches to determining parameters that a RESTFul API would use.

> E.g. to fetch a list of activities which have commitment
> transactions would you use URIs like
> /access/activities?condition=transactions/transaction/@type='C'
> or
> /access/activities?transactions_transaction__type=C
>

> * Approaches to serialising IATI data as JSON

>
> All input welcome,
>
> All the best
>
> Tim
>
>
> --
>
>
>

> http://www.timdavies.org.uk <http://www.timdavies.org.uk/>

>
> 07834 856 303.
>
> @timdavies
>
>
> Co-director of Practical Participation:
> http://www.practicalparticipation.co.uk

> <http://www.practicalparticipation.co.uk/>

>
> --------------------------
>
> Practical Participation Ltd is a registered company in England and Wales
> - #5381958.
>

[Bill Anderson]

I think it is really important that all queries are acceptable. If not who is going to decide on behalf of unknown users what is an acceptable query, and who is going to re-engineer the service in future when, inevitably, needs change?

 

Bill

[Tim Davies]

Hello David, John,

I'm just catching up on contributions to this thread. The user stories John shared are now linked from the Wiki page http://wiki.iatistandard.org/roadmap/api_standardisation?&#user_stories and I would encourage anyone to add to these or integrate key user stories / cases into the wIki page.

On the question of how we might include/excluded fields with the same name, but that exist at different places in the heirarchy, the general idea is that each of the options for APIs would define some way to refer to the 'path' of fields from the XML, so there would be a different name in the API for iati-activity/description, and iati-activity/transaction/description. 

In the xpath style API this would be with the slash as above, in a django style API it would be something like 'description' or 'transaction__description' to distinguish the different fields. 

Hope that makes some sort of sense...

Tim

[Tim Davies]

Hello Simon,

This is something where different response types might require different approaches. At the moment the draft suggests clients should be able to request the language of the response, which I would suggest means:

• If codelist values are available in that language then return them;
• If field values are available in the language, return them;

The 'default' behaviour when the require language is not available might take some working out. 

In XML it would be fairly easy for an API to return all the languages it is aware of, and let the client choose (making use of xml:lang attributes). In json this is trickier, as the json would need to be structured with language tags.

Any thoughts from those working with json APIs about the best way to handle language in this case?

All the best

Tim

To post to this group, send email to iati-technical@googlegroups.com

To unsubscribe from this group, send email to

iati-technical+unsubscribe@googlegroups.com

For more options, including the option to switch to a digest
subscription, visit this group at
http://groups.google.com/group/iati-technical

Tickets for the IATI technical secretariat can be posted to
http://support.iatistandard.org

--
You received this message because you are subscribed to the "IATI Technical" discussion list. Find out more at http://wiki.iatistandard.org/community/mailing_list

To post to this group, send email to iati-technical@googlegroups.com

To unsubscribe from this group, send email to

iati-technical+unsubscribe@googlegroups.com

For more options, including the option to switch to a digest subscription, visit this group at http://groups.google.com/group/iati-technical

Tickets for the IATI technical secretariat can be posted to http://support.iatistandard.org

--


http://www.timdavies.org.uk

07834 856 303.
@timdavies

Co-director of Practical Participation: http://www.practicalparticipation.co.uk

[Dan Mihaila]

Hello,
I didn't have much experience with API JSON for multiple languages but when i performed a google search, I found a recommendation on GoogleCode site (http://google-styleguide.googlecode.com/svn/trunk/jsoncstyleguide.xml  - but I found the text in a cached page: http://webcache.googleusercontent.com/search?q=cache:37GWVA-sHLQJ:google-styleguide.googlecode.com/svn/trunk/jsoncstyleguide.xml+&cd=2&hl=en&ct=clnk&gl=ro&client=firefox-a  )

What do you think about this?

lang property and XML's xml:lang properties. The value should be a language value as defined in BCP 47. If a single JSON object contains data in multiple languages, the service is responsible for developing and documenting an appropriate location for the lang property.

Example:

{"data": { "items": [ { "lang": "en", "title": "Hello world!" }, { "lang": "fr", "title": "Bonjour monde!" } ]} } Property Value Type: string (formatted as specified in RFC 3339)
Parent: data


 And regarding the initial topic, I suggest to move a little the things and to fill the wiki with the some queries/scenarios. Tim, any suggestions

Dan

Dan Mihaila, IT Consultant
Development Gateway International
49 Rue de Treves (Box 7) • 1040 Brussels, BELGIUM
(M) +40 722 502 304 • (GTalk) dan.m...@gmail.com • (Skype) carcotelul
• (MSN) dan.m...@hotmail.com • (Yahoo) carcotelul
dmih...@dginternational.org
www.developmentgateway.org

Information Tools. Global Partnerships. Effective Aid.

To post to this group, send email to iati-te...@googlegroups.com

 
To unsubscribe from this group, send email to

iati-technica...@googlegroups.com

[Tim Davies]

Hello Dan,

Good pointer on the JSON approach to 'lang' 

In terms of next steps, I suggest people with an interest review what is currently on the wiki and propose changes / add use cases there if appropriate.

We could also plan for a skype call in the next few weeks, and see if it is worth forming a small working group on this. Any interest?

All the best

Tim