Broken uploads with latest XNATpy version (0.5.0)

[Duncan Smith]

Hi all,

We've run into errors after upgrading our XNATpy version when trying to upload .zip or .tar files with more than 1 file in.

We have previously been able to use commands like the examples given below to upload .zip files containing multiple DICOM files:

To upload to an existing Resource object:

resource.upload("dicom.zip", "dicom.zip", extract=True)

To upload to the prearchive:

prearchive_session = session.services.import_("dicom.zip", project=project_string, subject=subject_string, experiment=experiment_string, destination="/prearchive")

However when we use these same commands in XNATpy 0.5.0 we see this error for both:

Traceback (most recent call last):
  File "/home/duncan/Desktop/xnat-test/xnat-project.py", line 22, in <module>
    prearchive_session = session.services.import_("dicom2.zip", project=project_string, subject=subject_string, experiment=experiment_string, destination="/prearchive")
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/xnat/services.py", line 239, in import_
    response = self.xnat_session.upload_file(uri=uri, path=path, query=query, content_type=content_type, method='post')
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/xnat/session.py", line 769, in upload_file
    self.upload_stream(uri=uri, stream=file_handle, **kwargs)
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/xnat/session.py", line 899, in upload_stream
    response = self.interface.post(uri, data=stream, headers=headers, timeout=timeout)
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/duncan/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/duncan/anaconda3/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/duncan/anaconda3/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/duncan/anaconda3/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/duncan/anaconda3/lib/python3.9/http/client.py", line 1069, in _send_output
    for chunk in chunks:
  File "/home/duncan/anaconda3/lib/python3.9/http/client.py", line 1024, in _read_readable
    datablock = readable.read(self.blocksize)
  File "/home/duncan/anaconda3/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 10: invalid start byte

I've tested these two commands with XNATpy 0.4.3 against XNAT 1.8.6.1 and XNAT 1.8.7 and they work, and tested against the same 2 XNAT versions with XNATpy 0.5.0 and they fail (so I don't think this is an XNAT issue directly). We've tried changing the .zip and .tar files compression and other settings but this hasn't seemed to help.

I notice in the release notes for XNATpy 0.5.0 it mentions "Reworked upload functions to have more clear behaviour" so either how these functions are used have changed and we're now using them incorrectly, or errors have been introduced somewhere in the process.

Thanks,

Duncan

[Hakim Achterberg]

Hi Duncan,

Thank you for reporting the error. I think I found it and fixed it in the develop branch (I found another bug, so I release a new version very soon). Could you test if the develop branch works for you?

A quick way to install the develop branch to test would be:

pip install https://gitlab.com/radiology/infrastructure/xnatpy/-/archive/develop/xnatpy-develop.zip

Some background: basically I opened the file as bytes in 0.4.3 and in 0.5.0 I accidentally did as strings. For files containing only utf-8 compatible characters, this did not give an error, but for other data this gives an decoding error.

[Duncan Smith]

Hi Hakim,

Thank you for your quick response, and explanation of the issue. After testing with 0.5.1 "resource.upload()" now works as expected, and we no longer see error messages. I am however getting a new error when trying to use the archive command mentioned above:

prearchive_session = session.services.import_("dicom.zip", project=project_string, subject=subject_string, experiment=experiment_string, destination="/prearchive")

I receive the following response:

Traceback (most recent call last):

  File "/home/duncan/Desktop/xnat-test/xnat-project.py", line 40, in <module>

    prearchive_session = session.services.import_("dicom.zip", project=project_string, subject=subject_string, experiment=experiment_string, destination="/prearchive")

  File "/home/duncan/anaconda3/envs/test-xnatpy/lib/python3.10/site-packages/xnat/services.py", line 247, in import_
    if response.status_code != 200:
AttributeError: 'NoneType' object has no attribute 'status_code'

I've tested the same command in 0.4.3 and it works as expected. But I do see the upload in the prearchive, so the issue only seems to be related to the response.

Thanks for your help,

Duncan

[Hakim Achterberg]

Thank you for your feed back, I will check the new error as well and sort it out. 

--
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/VAo5v_U7CM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/0b9a9c8f-5e8b-4332-9873-a5340d774b04n%40googlegroups.com.