Codelist update for version 1.04 of the IATI Standard

[caprenter]

For the past couple of years, Kit Wallace has kindly hosted the codelist API, partly through a contract with IATI but he has also gone well above and beyond the terms of that contract to continue to provide a machine readable codelist service. We would like to take this opportunity to thank him for his valuable contribution.

The codelists used in each version of the standard are derived from different endpoints and we are finally taking in-house control of them here at the IATI Secretariat.

Therefore, anyone relying on the services at:

• http://datadev.aidinfolabs.org/data/, or

• http://data.aidinfolabs.org/data/

is strongly advised to migrate to the new endpoints given below by 6th May.

• v.1.01 of the standard - uses datadev - change to http://codelists102.archive.iatistandard.org/data/

• v.1.02 of the standard - uses datadev - change to http://codelists102.archive.iatistandard.org/data/

• v.1.03 of the standard - uses data - change to http://codelists103.archive.iatistandard.org/data/

Codelists in version 1.04 of the standard

In  v.1.04 of the standard a new codelists structure has been introduced for those that want to take advantage of its improved features (see below). We're calling this codelist version 2 (CLv2). This forms the 'single source of truth' from which codelists in the current/existing format (now known as codelist version 1, or CLv1) are derived.

To switch to the current versions of the codelists with the updates required for version 1.04 of the standard use http://dev.iatistandard.org/codelists/downloads/clv1/

(N.B. on 6th May we will move from dev to live - use http://iatistandard.org/codelists/downloads/clv1/)

If you rely on the codelist API please read the notes on "Codelist API compatibility in version 1.04 of the IATI Standard" below.

To switch to the new improved version (v.1.04 only) you can either:

1. Download the files using this endpoint http://dev.iatistandard.org/codelists/downloads/clv2/ (N.B. on 6th May we will move from dev to live - use http://iatistandard.org/codelists/downloads/clv2/ ), or

2. Work directly with GitHub to get the data (see below).

Working with GitHub directly

The codelist source is now on GitHub:

• https://github.com/IATI/IATI-Codelists/tree/version-1.04/xml

• https://github.com/IATI/IATI-Codelists-NonEmbedded/tree/master/xml

We use this source to create all derived versions (CSV, JSON and all forms of codelist v.1 files, as well as all the documentation on the iatistandard.org website).

If you wish to use the new style XML, or are prepared to run the supplied python scripts for converting to a different format of your choice, you can fetch the codelists from GitHub directly.

Why create a new structure for codelists?

As part of our move towards creating a Single Source of Truth (http://dev.iatistandard.org/developer/ssot/) for the IATI Standard, we have started to take a different approach to codelists. See background paper previously circulated: https://docs.google.com/document/d/1oeH-8BFB__2IYF4MLnUwx2LcXZCVd5e-iYsXtQ4ViTk/edit

As this work progressed, it became evident that, for the codelists to work in that environment, we would need to make a few changes. As a result:

1. The newer codelist files are more consistent:

• They don't include the element name as a tag name

• All have language information described the same way as IATI XML

2. There is also a codelist schema that all the source XML validates against - https://github.com/IATI/IATI-Codelists/blob/master/codelist.xsd

3. Finally, more metadata, including a description, is now included in the codelists.

Codelist API compatibility in version 1.04 of the IATI Standard

If you wish to stay with the current codelist API system in v.1.04 you will be able to do so but there are a few things you should be aware of:

1. The 'metadata' link/call is no longer available (instead, some of this is now available in the new style codelists and some is stored in the GitHub repository metadata to be consistent with the rest of the Single Source of Truth).

2. Only the latest versions (those required in v.1.04) will be available through the API. If you need an older version use the static archives detailed above.

3. URLs containing the version and/or language do not work. For example, in the past a URL like: /data/codelist/AidTypeFlag/version/1.0/lang/en was possible. This will not work using the /codelists/downloads/clv1/ endpoint. Instead, translations, where available, are maintained in the codelist version 2 (CLv2) files.

• The 'fields' element is no longer provided in the index XML/JSON. See http://data.aidinfolabs.org/data/codelist.xml and compare with http://dev.iatistandard.org/codelists/downloads/clv1/codelist.xml

• Version information is no longer provided in the XML.

• Specific changes to codelists:

• Budget Identifier (http://dev.iatistandard.org/codelists/BudgetIdentifier/) has different categories, and no 'sector' elements.

• Names in the File Format (http://dev.iatistandard.org/codelists/FileFormat/) list are no longer there as they make little sense, and the list now tracks the IANA source it is derived from directly.

If you have any questions about these changes, please don’t hesitate to contact us at co...@iatistandard.org.

Best,

The IATI Technical Team