Upload of files via pyxnat is very slow

[David Just]

For internal reasons, I can’t use the built in xnat dicom receiver, I have to upload dicom files to each study/series using pyxnat.   
The code relevant code for this is:


    def storeFile(self, fullFilePath, overwrite=False, fid=None):
        if self.resource is None:
            self.store()
        if fid is None:
            fid = os.path.basename(fullFilePath)
        thefile = self.resource.file(fid)
        if overwrite or not thefile.exists():
            thefile.put(fullFilePath, self.type)


Where self.resource is a reference to an pyxnat resource object.

This works, but I’m finding that on fast box it’s taking about 1.5 hours to upload 6,000 dicom files. Is anybody else doing something similar, and if so what have your experiences been?

Thanks,
Dave.



 

[Archie, Kevin]

Dave,

 

The same XNAT machinery that accepts DICOM objects via C-STORE and puts them into the prearchive also accepts HTTP POST: either file-at-a-time, MIRC-style, or series-at-a-time zips containing DICOM. We haven’t done any benchmarks, but I would expect these to be faster than using the REST API because there’s less overhead (I think), especially for the series zips (which is what the upload applet uses now).

 

There’s no documentation for posting series zips, but there is Java code in the upload applet (http://hg.xnat.org/uploadassistant), particularly in org.nrg.dcm.Study and org.nrg.io.dcm.ZipSeriesUploader.

 

-          Kevin

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To post to this group, send email to xnat_di...@googlegroups.com.
To unsubscribe from this group, send email to xnat_discussi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/xnat_discussion?hl=en.

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Dattatray B]

Hi Dave, 

I am also trying to upload files using Pyxnat to our XNAT sever which is remotely located. We generally access it through ssh. 

In your case are you uploading data from your local pc(any storage) to XNAt installed on some server?

I have tried the follwing code but it gives me HTTP error. With this i am unsure if  PyXNAT can upload files from local pc to XNAt on sever. 

univ = Interface('https://hd-hni-xnat.cac.cornell.edu:8443/xnat')

learn_obj = univ.select.project('Learn') #project_obj

subject_list =  learn_obj.subjects().get()  #subject_no = accession number on XNAT web interface

subjectobj_HDHNI_S00163_ajsc = learn_obj.subject(subject_list[4])

experimenat_obj = subjectobj_HDHNI_S00163_ajsc.experiment('MR Session')

scan_obj = experimenat_obj.scan('3planeloc')

scan_obj.resource('NIFTI_upload').file('image.nii').put('/home/r100307_BOLD_SOCIAL1_RL.nii')

Error reposrted for the lasyt line:

<h3>The server encountered an unexpected condition which prevented it from fulfilling the request</h3><p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1">here</a>.<br>

Please continue your visit at our <a href="/">home page</a>.

</p>

</body>

</html>

Was wondering if you can share snippet of youe Pyxnat code ..? That would be of big help. 

I appreciate your time and consideration. 

Thanks!

DB

[dave J]

First, I don't see any authentication in that code. I'm assuming you deleted it when posting.   

the code i posted above is still in use today, so It should still work for you. 

But try this:

import os

import sys

def storeFile(xnat_resource, fullFilePath, fileType='Image', overwrite=False,fid=None):

    if not xnat_resource.exists():

        return None

    if not fid:

        fid =  os.path.basename(fullFilePath)

    thefile = xnat_resource.file(fid)

    if os.path.isfiile(fullFilePath):

        if overwrite or not thefile.exists() :

            thefile.put(fullFilePath, fileType, overwrite=True)

            sys.stderr.write("created file resource for: " + fullFilePath)

    else:

        sys.stderr.write("Cannot store file. File does not exist:  {file} ".format(file=fullFilePath))

univ = Interface('https://hd-hni-xnat.cac.cornell.edu:8443/xnat')

learn_obj = univ.select.project('Learn') #project_obj

subject_list =  learn_obj.subjects().get()  #subject_no = accession number on XNAT web interface

subjectobj_HDHNI_S00163_ajsc = learn_obj.subject(subject_list[4])

experimenat_obj = subjectobj_HDHNI_S00163_ajsc.experiment('MR Session')

scan_obj = experimenat_obj.scan('3planeloc')

scan_resource=scan_obj.resource('NIFTI_upload')

storeFile(scan_resource, '/home/r100307_BOLD_SOCIAL1_RL.nii', 'NII', True, 'image.nii')

[Dattatray B]

Thank you very much Dave. 

Yes i deleted the authetication. 

I will try as you are doing it in the shared code snippet. 

Many Thanks for sharing it. 

Dattatray 

[Dattatray Bhogawar]

HI Dave, 

Just a small clarification. 

xnat_resource stand for either project object/subject object/experiment obj? Right?

And what id fid?

Many Thanks again. 

Suyash

--
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/nrl_yV0RyP0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.

To post to this group, send email to xnat_di...@googlegroups.com.

Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

--

Thanks & Regards

Dattatray Bhogawar

[dave J]

xnat_resource can be any pyxnat object that can have files underneath it.   This is usually a resource object under scans but can be a resource object under project, experiment, or subject too.

fid is shorthand for File ID.   it's the "name" that xnat knows the file as.   in your example the fid was "image.nii"  and the file name was "r100307_BOLD_SOCIAL1_RL.nii" 

[Datta]

Thank a ton Dave. yes i am able to upload files on my server now. :)

but just that they are not visible when i login through browser. Will look into disucssion group. 

Thank you your help and sharing. 

Datta

[Dattatray Bhogawar]

Hi Dave, 

I am coming accross this wiered sitaution where i can upload data and download even using get method however the data doesnt seem to be there when i check through XNAT web browser.

Following is the code which i tried:

univ = Interface('https://hd-hni-xnat.cac.gwu.edu:8443/xnat')

learn_obj = cornell.select.project('Learn')

subject_list =  learn_obj.subjects().get()

subject_obj = learn_obj.subject(subject_list[4])

experimenat_obj = subject_obj.experiment('AFNI')

experimenat_obj.resource('NIFTI').file('image.nii').put('/Users/sdb99/Subject18/mprage-s18-obl.nii')

the above code does upload file successfuly as i can even download it with get method. However doesnt show up on XNAt website.

Any idea why this might be the case?

Many Thanks in advance.

Suyash

[dave J]

Have you checked under the "manage files" link on the study level?   

[Dattatray Bhogawar]

A big Thank you Dave.. :)

Yes its there. That just brought smile on my face. Wasn't able to figure about even after spedning 4-5 hours on that. 

Thanks again!

Datta