File upload via python script

[edpr...@googlemail.com]

Hi,

I'm working on an automatic, daily QA for our MRI systems. I set-up qatrack successfully and everything is working so far (thanks to the QAtrack Wiki and Google Groups posts).
I already managed (using a code snippet from one of the posts) to send numerical, boolean and strings to QATrack. However, I do not know how to get the file upload working (images, text file).

Any help or insights would be appreciated.

Thanks,
Ed

[cacp...@gmail.com]

Hi, can you link to the post you are referring to? I'm very interested in this usage.

[edpr...@googlemail.com]

On Wednesday, November 18, 2015 at 3:06:14 PM UTC+1, cacp...@gmail.com wrote:
> Hi, can you link to the post you are referring to? I'm very interested in this usage.

Hi,
here is the link I was referring to:
https://groups.google.com/forum/#!searchin/qatrack/$23197/qatrack/vO5H-zsfgsc/jeWS4qpxKRkJ

and helpful as well:
https://groups.google.com/forum/#!searchin/qatrack/python$20list/qatrack/-IpMpiS6-1c/xZI1Tesf2nIJ

both links contain posts which include python snippets.

[Randle Taylor]

Hi Ed,

When you say that you can't get the file upload working does that mean that you get an error when trying to upload a file or problems when trying to analyze the file?  If the former, that may mean there is a permissions issue with the upload directories (located at qatrack/media/uploads & qatrack/media/uploads/tmp).  If the latter, I would suggest looking at the following test list on the demo site: http://randlet.com/qatrack/qa/utc/perform/11/?day=next&next=/qatrack/qa/ and the analysis procedure defined here:  http://randlet.com/qatrack/admin/qa/test/19/

RT

--
You received this message because you are subscribed to the Google Groups "QATrack+" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qatrack+u...@googlegroups.com.
To post to this group, send email to qat...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[edpr...@googlemail.com]

Hi Randle,

the file upload (pushing the file upload button and selecting an image from the file dialog) is working perfectly (in fact, in the beginning I had some issues regarding file permissions). Anyway, what I'm trying to do now is to upload the image from within a python script (using the requests module). I just don't know how to deal with files in general. Submitting simple numericals, strings , and booleans works so far.

[Randle Taylor]

Aha OK I see.

So the file upload gets posted to a different url, then your test list.  So what you will need to do is first post your file (use 'upload' instead of 'file' as the file name) to /qa/upload/ which will send you back a json response containing the file name (in key "temp_file_name") and any results that were calculated from that upload.  After posting your file, when you submit your data for the test list you need to set the value of the upload test to that temporary file name.  So roughly the sequence of events should look like:

1. Post file to /qa/uploads/ and get back a json response

2. Extract 'temp_file_name' from response.

3. Extract any calculated results from response (json key is "results")

4. Set the value of your upload test data to the temp_file_name

5. Post your test list data

Also, if you look at the code for the Upload view in particular the "handle_upload" method, you can see that the temp file name depends on the 'sessionid' cookie and the id of the upload test, so you need to include that info when posting your file.

Let me know if any of that isn't clear or you get stuck on a specfic test..If you can't get it working I can try to put together a demo when I have time.

Cheers,

Randy

[edpr...@googlemail.com]

Hi Randle,

thanks for the quick reply. I tried to get the image upload working but failed, in the very beginning. Basically I have two problems:

1. I get a forbidden (403) when I try to upload (doing this manually on the webpage works):

login_url = "http://myserver/qatrackplus/accounts/login/"
s = requests.Session()
s.get(login_url)
...
login_resp = s.post(login_url, data=login_data)

img = "image.png"
upload_url = http://myserver/qatrackplus/qa/upload/)
files = {"upload": open(img, "rb")}
resp = s.post(upload_url, files=files)

2. The responses I get (for instance from the numerical value post, which works) are not JSON. I think this is due to redirection.
How can I get in this case the needed information anyway?

Thanks for your help
Ed

[Randle Taylor]

I neglected to mention you'll need to send the csrfmiddlewaretoken as well when you post the data ) so that's probably why you're getting that error (It needs to be included anytime you post data).

Submitting the test list will give you back an html response (a redirect, the same behaviour as submitting a test list using the browser) but a successful upload file request will return JSON.  So you should be able to do `data = json.loads(resp.text)` to get a dictionary of all the data returned.

RT

[edpr...@googlemail.com]

Hi Randle,

thanks again. Still not working though. I googled around and found two suggestions:
1)
...
headers = {'Referer': upload_url)
request = s.post(upload_url, files=files, data={'csrfmiddlewaretoken': token}, headers=headers)

2)
...
headers = {'Referer': upload_url, "X-CSRFToken":token}
request = s.post(upload_url, files=files, headers=headers)

both variants give me an internal server error.
Any ideas?

Sorry to bother you again...
Ed

[Randle Taylor]

First version is what I think you would want although I wouldn't have thought you needed the headers.  I'll have to take a crack at it myself this evening and get back to you.

"Sorry to bother you again..."

No problem!

RT

[Randle Taylor]

Hi Ed,

So it turned out it was a little more complicated than I initially thought but here's how I've got it working using http://randlet.com/qatrack/qa/utc/perform/25/ as a demo:

import datetime

import json

import requests

def get_csrf_token_from_url(url, session):

    session.get(url)

    return session.cookies['csrftoken']

root = "http://randlet.com/qatrack/"

s = requests.Session()

work_started = datetime.datetime.now().strftime("%d-%m-%Y %H:%M")

test_list_url = root+"qa/utc/perform/25/"

upload_test_id = 44

# perform the login ################################################

login_url = root + "accounts/login/"

login_data = {

    'username':'demo',

    'password':'demo',

    'csrfmiddlewaretoken': get_csrf_token_from_url(login_url, s)

}

login_resp = s.post(login_url, data=login_data)

# upload data ######################################################

test_list_token = get_csrf_token_from_url(test_list_url, s)

img = "image.png"

upload_url = root+"qa/upload/"

files = {"upload": open(img, "rb")}

upload_data = {

    'test_id': upload_test_id,

    'meta': json.dumps({

        "work_started": work_started,

        "work_completed":"",

    }),

    'csrfmiddlewaretoken': test_list_token

}

resp = s.post(upload_url, files=files, data=upload_data)

data = json.loads(resp.text)

# encode your form data as strings

test_data = {

    'csrfmiddlewaretoken': test_list_token,

    "work_started": work_started,

    "work_completed":"",

    "status":"",

    "form-TOTAL_FORMS":"2",

    "form-INITIAL_FORMS":"2",

    "form-MAX_NUM_FORMS":"1000",

    "form-0-string_value": data['temp_file_name'],

    "form-1-value": data['result']['mean'],

}

# submit test data

resp = s.post(test_list_url, data=test_data)

A couple of notes:

1) The CSRF token for the upload has to be from the test list page (that's why you were getting the 403 on the upload)

2) You need to include the test_id and meta data when uploading

3) You need to set the 'string_value' key for the temp_file_name

4) I really need to write a proper api for doing this so it's far less clunky!

Cheers,

RT

To unsubscribe from this group and stop receiving emails from it, send an email to qatrack+unsubscribe@googlegroups.com.

[edpr...@googlemail.com]

somehow I didn't get an email regarding your last post. That's the reason for the delay.
Anyhow, today I checked the Ggroups and found your response.
Everything works now.

Thanks a lot.
Ed

[Dagmar S]

Hi,
we are using this upload via Phyton scripts for a while now. However, is it possible to skip one of the tests in the upload and how would I do that?

Thanks,
Dagmar

[Randle Taylor]

You should be able to add e.g. 'form-0-skipped': 'on'  in the test_data you send to the server to have a test skipped. RT

--

You received this message because you are subscribed to the Google Groups "QATrack+" group.

To unsubscribe from this group and stop receiving emails from it, send an email to qatrack+unsubscribe@googlegroups.com.

[Dagmar S]

Thank you very much. This was exactly I was looking for. It works now.

Dagmar