Request advice on how to do Open Refine URL fetch using API key and HTTPS headers

[[•••••••]]

I'm trying to lookup Australian postcodes in Open Refine using the Auspost API and the URL fetching method, based on a column of suburbs. Can anyone explain to me how to implement this method described below, or suggest a reliable alternative? (Full documentation here https://auspost.com.au/devcentre/assets/pdfs/pac-pcs-technical-specification.pdf)

Many thanks, A

2.1 API Key 

A unique 32 character API key is used to authorise inbound requests to the APIs. To obtain an API key please visit 

http://auspost.com.au/devcentre and follow the link to create your Australia Post ID. After registration has been successfully 

completed you will be emailed an API key along with links to documentation specifying how to call the APIs and the terms and conditions 

associated with using the APIs. 

 

Australia Post requires customers to use these APIs via HTTP GET over SSL (Secure Sockets Layer) protocol so as to protect the API keys. 

The required design pattern is for customers to use AUTH-KEY and SSL for both the production and non-production websites. The API keys 

cannot be used in address line calls and MUST be sent within the HTTP HEADERS. 

2.1.1 Non-production API key 

During integration phase of your project, you might want to use non-production APIs available at https://test.npe.auspost.com.au, rather 

than live APIs. 

The following API key is required to access these APIs: 

28744ed5982391881611cca6cf5c2409 

...

5.1 Postcode Search API 

Search the collection of Postcode records for potential matches. The search criterion is location name or postcode with an optional state 

filter. Request must be a HTTPS GET. 

 

URL: 

https://auspost.com.au/api/postcode/search.{format} 

 

Formats: 

xml, json (Default to xml if format not provided) 

 

Requires Authorisation: 

True (API Key) 

 

Parameters: 

1. [q]: Required. The search criterion used to drive the search query. This parameter will hold either the location name or postcode 

2. [state]: Optional. Used to filter possible search results. Must be formatted using an Auspost accepted abbreviation (i.e. VIC 

for Victoria, QLD for Queensland) 

3. [excludePostBoxFlag]: Optional. Set value to true, if post boxes should be excluded from search results. That is 

useful for PAC. 

...

6.2 Service oriented API examples 

These examples return results in XML format, however note that JSON format is also permitted. 

 

Request type: HTTPS GET 

Header: AUTH-KEY: {Use your personal 32 character API key} 

 

Postcode search: 

https://auspost.com.au/api/postcode/search?q=melbourne 

[Thad Guidry]

The regular FETCH grel command does not do HTTPS (secure) connections or with any custom headers, which it appears you will need to support for that API.

What you can do is create a custom GREL expression built up with Jython or Clojure to handle it for you however...  something along this kind of expression in the Answer:

http://stackoverflow.com/questions/6999565/python-https-get-with-basic-authentication

If you require additional "hacking" support...any number of us can help you via payments through Paypal.  You might ask others if they have more time to create the custom Jython expression for you to do that work.

An alternative is to just use Scraperwiki.com and ask their community for hacking one up for you as well !

(I'm sorry that OpenRefine doesn't have a full featured FETCH dialog that would allow you to wire something up like this... we haven't had the time to implement one...but it is doable and we have an issue opened for it that I would probably assign a rough bounty of $600 (my estimate).

Best of Luck !

--
You received this message because you are subscribed to the Google Groups "OpenRefine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

-Thad

+ThadGuidry
Thad on LinkedIn

[[•••••••]]

Thank you for this, and apologies for the slow reply. So If I wrap the API into a python script that works on my computer, will I have to use special libraries to run it as jython in Open Refine, or does Open Refine access the python libraries on my computer? I'd rather learn how to fix the problem myself, if I can. Thanks again. 

[Mike Bestvina]

Maybe try Yahoo Pipes?

http://stackoverflow.com/questions/531179/using-a-https-resource-in-yahoo-pipes/2758422#2758422

So...

Source Data (HTTPS-JSON) -> Yahoo Pipes (JSON) -> Consume in Refine