API call sometimes forgets authentication?

[Gerard Schut]

I'm trying to access the data from the surveys through some simple VB and run into a strange problem.

When using the very simple code:

Dim httpReq As New MSXML2.ServerXMLHTTP60

Call httpReq.Open("GET", "https://kc.kobotoolbox.org/api/v1/data", False, "my_userid", "my_password")    <== these are obviously not my real username & password

Debug.Print httpReq.responseText

When I run this line of code, 

[{"id":55229,"id_string":"aRk84dxcTKZK5wsZegXC9d","title":"JAS - Joint Activity Survey","description":"JAS - Joint Activity Survey","url":"https://kc.kobotoolbox.org/api/v1/data/55229"}]

Which looks correct to me. So I know my data is under 55229.

However when I run the exact same code, but with the number added:

Call httpReq.Open("GET", "https://kc.kobotoolbox.org/api/v1/data/55229", False, "my_userid", "my_password")

It returns:

{"detail":"Not found."}

However, when I run the same from any webbrowser (and have authenticated during the first request), the second request returns the proper dataset.

If I logout from the browser and run the 2nd request without logging in, I get the "detail not found" error message, so it seems to me related to authentication, but that doesn't make too much sense as the first request seems to work just fine.

I've run this kobotoolbox.org and on humanitarianresponse.info with exactly the same result. 

I've searched around, but didn't see anyone having this problem. Any help is appreciated.

[Gerard Schut]

I didn't find a solution for the problem, but I've found a work around by using token authentication. 

I'm now able to get the dataset.

In case someone runs into the same issue, a simplified version of the VBA code below:

Dim httpReq As New MSXML2.ServerXMLHTTP60

Call httpReq.Open("GET", "https://kc.humanitarianresponse.info/api/v1/data/67602?format=xml", False)

httpReq.setRequestHeader "Authorization", "Token <all the digits of my secret token>"

httpReq.send

Debug.Print httpReq.responseText

[Kobo Users]

__________________________________

Hi Gerard,

I’m glad you found a workaround! I am curious, though, about the trouble you were having with username/password authentication.

When using the API, you do have to send the credentials with each request, whether you’re using username/password or token authentication. By contrast, a web browser perusing the API creates a session the first time you log in, where authentication details are stored in cookies etc., saving you the hassle of logging in again every time you load a different page.

I don’t have a VBA environment handy, but here’s a test using the program “curl” and an account whose username is “apitest” and password is “the password”:

1. First, querying without a specific ID:
   $ curl -u ‘apitest:the password’ https://kc.kobotoolbox.org/api/v1/data
   [{"id":55576,“id_string”:“ahudYAcPM9uivNnhv7N6Be”,“title”:“single question form”,“description”:“single question form”,“url”:“https://kc.kobotoolbox.org/api/v1/data/55576”}]

1. Next, querying for a specific ID (55576), and still providing the username and password:
   $ curl -u ‘apitest:the password’ https://kc.kobotoolbox.org/api/v1/data/55576
   [{"_notes":[],“meta/instanceID”:“uuid:4621196a-d851-44c8-9eca-67fd07f088c8”,“end”:“2016-05-23T18:25:44.000-04:00”,“_submission_time”:“2016-05-23T22:25:49”,“_uuid”:“4621196a-d851-44c8-9eca-67fd07f088c8”,“_bamboo_dataset_id”:"",“_tags”:[],“_attachments”:[],“start”:“2016-05-23T18:25:35.000-04:00”,_submitted_by,_geolocation],“_xform_id_string”:“ahudYAcPM9uivNnhv7N6Be”,“what_s_your_favorite_color_”:“chartreuse”,“_status”:“submitted_via_web”,_id,“version”:“1404633”,“formhub/uuid”:"f6ad759c000a46cc93b0745bba155544"}]

You’re right that an unauthenticated request leads to the “Not found” error you received, e.g.

$ curl https://kc.kobotoolbox.org/api/v1/data/55576
{"detail":“Not found.”}

The maintainer of the software library responsible for that behavior discusses his rationale at https://github.com/tomchristie/django-rest-framework/issues/1439#issuecomment-36012573.

Still, what’s curious to me is that your request:

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data/55229”, False, “my_userid”, “my_password”)

…does specify the username and password, and yet it still failed. If that continues to happen, please let us know so that we can investigate further.

Thanks,

John Milner
Developer, KoBoToolbox

On May 23, 2016 at 10:16AM EDT Gerard Schut <gerard...@gmail.com> wrote:

I didn't find a solution for the problem, but I've found a work around by
using token authentication.

I'm now able to get the dataset.

In case someone runs into the same issue, a simplified version of the VBA
code below:

Dim httpReq As New MSXML2.ServerXMLHTTP60
Call httpReq.Open("GET",
"https://kc.humanitarianresponse.info/api/v1/data/67602?format=xml", False)
httpReq.setRequestHeader "Authorization", "Token <all the digits of my
secret token>"
httpReq.send
Debug.Print httpReq.responseText

--
You received this message because you are subscribed to the Google Groups "Kobo Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kobo-users+...@googlegroups.com.
To post to this group, send email to kobo-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kobo-users.
For more options, visit https://groups.google.com/d/optout.

--
Any other questions? Visit support.kobotoolbox.org[[93b26426b69bb179b748affe40b335cfef074143-705150624]]

[Gerard Schut]

Hi John,

First of all, thank you very much for the reply. It's much appreciated.

I've done some more testing: I've installed curl and when I follow the same steps as previously described, it works all as it should. So that's the good news: I can reproduce what you write and it's not some proxy in between that is causing the problem I'm experiencing.

I did some more digging and find this article: https://zanstra.home.xs4all.nl/inTec/ServerXMLHTTP.htm

It seems that the MSXML2 object is a bit buggy and it doesn't send the username & password supplied in the parameters until the server sends back a 401. The only solution seems to be sending the username and password in the function as well as in the header.

I adjusted the code to this:

Dim httpReq As New MSXML2.ServerXMLHTTP60

Call httpReq.Open("GET", "https://kc.humanitarianresponse.info/api/v1/data/67602", False, "username", "password")

httpReq.setRequestHeader "Authorization", "Basic " & Base64Encode("username:password")

httpReq.send

Debug.Print httpReq.responseText

and now I get back the response that I was expecting (the same as in curl).

The link you send is exactly where the problem is:

api/v1/data --> This one returns a 401 when no username & password supplied

api/v1/data/12345 -> This one returns "nothing found" when no username & password supplied

Sorry, but that it not very consistent. I'm happy I got it to work because you pointed me in the right direction, but you have to agree that the function that gets the data set should work similar to it's "parent".

I hope I provided enough details that this maybe considered to be changed in a future release.

Kind regards, Gerard