DigitalNZ API review

[Chris McDowall]

Hello developers

The DigitalNZ team is currently undertaking a review of our API. We
are keen to get feedback from developers on things they would like to
see in a future version. Are there features you would like to see
added or changes to the syntax that would make the API simpler to work
with?

We are especially interested in hearing about pain points you might
have encountered while working with the API. What are things that
hindered your ability to work with the data?

Feel free to post suggestions to this thread or get in touch with me
directly ( http://digitalnz.org/about/contact-us ).

Please also note that the new API terms come into effect today. You
can read them here: http://digitalnz.org/about/terms-of-use/developer-api-terms-of-use

Many thanks,
Chris McDowall, DigitalNZ

[Elliott Young]

Hi Chris,

Great to have a chance to have input into the new API functionality.

Top of my list is Spatial Search - being able to search for items
within a certain radius of a point given as lat/long. Such a
capability would allow for interesting iPhone / Android apps as well
as regular websites. It'd be great if you could also *sort* the
results by distance from a point.

I'd also love to be able to search by any of the parameters being
returned by the Search Records API, by which I mean I'd like it if I
could include fields in the search text such as collection_root,
atl_free_download, name_authority, etc. The atl_free_download images
in beta have really great terms of use (https://beta.natlib.govt.nz/
terms-for-free-download) but there's no way to use specifically these
images in an application on the DigitalNZ API, which is a pity.

Thanks!
Elliott

[Jonathan Hunt]

On Jan 31, 1:10 pm, Chris McDowall <chris.mcdow...@gmail.com> wrote:

> Please also note that the new API terms come into effect today. You
> can read them here:http://digitalnz.org/about/terms-of-use/developer-api-terms-of-use

Thank you for pointing out the new API terms, Chris. Some feedback:

"5. Your API key is personal to you and cannot be used by or
transferred to any other person without DIA’s written permission.

6. You must keep your API key secure and confidential. If you believe
that the security and confidentiality of your API key has been
compromised, you must notify us as soon as possible at
in...@digitalnz.org"

The terms treat the API key as a secret but the requirements for
confidentiality preclude multiple useful development options. For
example, a pure javascript app that operates in browser would expose
the API key even though this is an attractive deployment option for
certain use cases. Also, an app deployed to iPhone or Android would
potentially expose the key unless additional steps are taken to
obfuscate the credential in the code. Would distributing an app
containing the API key contravene clause 5?

"7. If you are displaying DigitalNZ metadata retrieved via the
DigitalNZ API on or through a website or application that requires
registration you must, on request by DIA or any metadata contributor,
provide a logon to the website or application to DIA or the metadata
contributor, without charge."

What would DIA or any metadata contributor do with their access? What
are they checking for?

"12. You must not make more than 3000 API calls per day, per API key
issued to you, without written approval from DIA."

It would be nice if this was friendlier. 3000 API calls isn't much if
a particular app got popular. Also, an app "in the wild" might be
suddenly discovered or shared on social media causing a spike of
traffic completely unknown by the developer. Surely DigitalNZ can
implement some basic throttling or DDOS prevention to reduce the
impact of a traffic spike?

"(a) not to grant an API key to any person at its complete
discretion;"

I don't think a clause like this should be in a government API. There
should be solid grounds for disallowing online access, not simply
discretion.

"(ii) providing a website or application that is enabling others to
act in a manner that is either contrary to these terms of use or is
bringing or could bring DigitalNZ, the National Library, DIA or any
contributor of metadata to DigitalNZ into disrepute; and"

Ditto. Would a site that used Digital NZ data to critique
institutional practices be shut down by such a clause?

"19(d) to otherwise terminate your access to, or discontinue, the
DigitalNZ API at any time, for any reason and at its sole discretion."

Ditto. This type of clause was in the very first terms for DigitalNZ
(~June 2009) and was subsequently removed - why has it made a
comeback? I don't like the idea that DigitalNZ/DIA could unilaterally
terminate my use, especially if I have put significant time or money
into developing an application. There's no mention of any mechanism
for handling disputes or misunderstandings.

[Chris McDowall]

Hi Jonathon,

Thanks for your thoughtful response to the new API terms. I genuinely
appreciate you taking the time to respond. Please find my comments
inline.

> > Please also note that the new API terms come into effect today. You
> > can read them here:http://digitalnz.org/about/terms-of-use/developer-api-terms-of-use
>
> Thank you for pointing out the new API terms, Chris. Some feedback:
>
> "5. Your API key is personal to you and cannot be used by or
> transferred to any other person without DIA’s written permission.
>
> 6. You must keep your API key secure and confidential. If you believe
> that the security and confidentiality of your API key has been
> compromised, you must notify us as soon as possible at

> i...@digitalnz.org"

>
> The terms treat the API key as a secret but the requirements for
> confidentiality preclude multiple useful development options. For
> example, a pure javascript app that operates in browser would expose
> the API key even though this is an attractive deployment option for
> certain use cases. Also, an app deployed to iPhone or Android would
> potentially expose the key unless additional steps are taken to
> obfuscate the credential in the code. Would distributing an app
> containing the API key contravene clause 5?

I believe this aspect of the API terms warrants review. I agree that
both cases you raise are problematic. Your points have prompted an
internal conversation. I'll let you know how this turns out.

> "7. If you are displaying DigitalNZ metadata retrieved via the
> DigitalNZ API on or through a website or application that requires
> registration you must, on request by DIA or any metadata contributor,
> provide a logon to the website or application to DIA or the metadata
> contributor, without charge."
>
> What would DIA or any metadata contributor do with their access? What
> are they checking for?

This provision was added in response to our content partners. They
want to be able to verify that applications are using data as per the
terms of use without paying for the service.

> "12. You must not make more than 3000 API calls per day, per API key
> issued to you, without written approval from DIA."
>
> It would be nice if this was friendlier. 3000 API calls isn't much if
> a particular app got popular. Also, an app "in the wild" might be
> suddenly discovered or shared on social media causing a spike of
> traffic completely unknown by the developer. Surely DigitalNZ can
> implement some basic throttling or DDOS prevention to reduce the
> impact of a traffic spike?

This wording is a problem. It does not adequately convey our
intentions. How does something like the following sound?

"The default limit on API calls is 3000 API calls per day, per API key
issued to you. Please contact the DigitalNZ team to discuss high rate
limits if need be."

> "(a) not to grant an API key to any person at its complete
> discretion;"
>
> I don't think a clause like this should be in a government API. There
> should be solid grounds for disallowing online access, not simply
> discretion.
>
> "(ii) providing a website or application that is enabling others to
> act in a manner that is either contrary to these terms of use or is
> bringing or could bring DigitalNZ, the National Library, DIA or any
> contributor of metadata to DigitalNZ into disrepute; and"
>
> Ditto. Would a site that used Digital NZ data to critique
> institutional practices be shut down by such a clause?

We would only activate these clauses if the data was being used in a
completely inappropriate way. Unfortunately, it is difficult to
foresee all the possible inappropriate ways that data might be used.
Our track record to date is good. In the history of DigitalNZ we have
only ever terminated one API key. That was for commercial use.

> "19(d) to otherwise terminate your access to, or discontinue, the
> DigitalNZ API at any time, for any reason and at its sole discretion."
>
> Ditto. This type of clause was in the very first terms for DigitalNZ
> (~June 2009) and was subsequently removed - why has it made a
> comeback? I don't like the idea that DigitalNZ/DIA could unilaterally
> terminate my use, especially if I have put significant time or money
> into developing an application. There's no mention of any mechanism
> for handling disputes or misunderstandings.

Your comments here have prompted a closer review of this clause. We
are considering changing or removing this clause. We will still need a
clause to allow us to terminate access if usage contravenes our terms.
I will keep the list updated.

Thanks again, Jonathon. Please let me know if you have further
questions. (Seriously.)

Chris