Re: Slow Dicom SCP transfers

[Fletcher Johnson]

I've also found that moving an archive is quite slow in line with the problem I have described.

On Tuesday, April 9, 2013 1:34:24 PM UTC-4, Fletcher Johnson wrote:

I've got a problem here involving slow dicom c-store transfers. Here's the scenario. I have two xnat installations present in two identical virtual machines. One of the installations is version 1.6 while the other is version 1.5. The version with 1.6 has around 2500 sessions while the 1.5 xnat is nearly empty (1-2 sessions). Now when I try to used the DicomRemap command that comes with the DicomBrowser package to do a c-store I get wildly different times for storing the same archive. It takes about 5x times longer to store an identical archive into the 1.6 xnat! I'm doing the transfers locally through the loop back device and activity on both machnes is near idle. The two machines are identical in the sense that they are clones of each other, so they have the same os etc. I'm working if this isn't an xnat scaling issue and I was looking to see if anyone has had circumstances like mine.

[David Gutman]

Is the storage the same as well.. I.e. both on the same class of storage within the vm h ost?

On Apr 9, 2013 1:34 PM, "Fletcher Johnson" <fjoh...@research.baycrest.org> wrote:

I've got a problem here involving slow dicom c-store transfers. Here's the scenario. I have two xnat installations present in two identical virtual machines. One of the installations is version 1.6 while the other is version 1.5. The version with 1.6 has around 2500 sessions while the 1.5 xnat is nearly empty (1-2 sessions). Now when I try to used the DicomRemap command that comes with the DicomBrowser package to do a c-store I get wildly different times for storing the same archive. It takes about 5x times longer to store an identical archive into the 1.6 xnat! I'm doing the transfers locally through the loop back device and activity on both machnes is near idle. The two machines are identical in the sense that they are clones of each other, so they have the same os etc. I'm working if this isn't an xnat scaling issue and I was looking to see if anyone has had circumstances like mine.

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Herrick, Rick]

Yeah, that was going to be my follow-up question as well. The type of drive, type of buss, and means of mounting (especially if one is, e.g., an NFS-mounted drive and the other eSata 10K RPM local storage) can have a huge impact on the transfer rate in the archive. The lion's share of work being done there is simply the transfer (really it's a copy and delete) of the data.

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

---

From: xnat_di...@googlegroups.com [xnat_di...@googlegroups.com] on behalf of David Gutman [dagu...@gmail.com]
Sent: Wednesday, April 10, 2013 4:06 PM
To: xnat_di...@googlegroups.com
Subject: Re: [XNAT Discussion] Slow Dicom SCP transfers

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Fletcher Johnson]

I guess I posted my reply directly to Rick instead of to the group. Thank you both for your replies, I'm very grateful for your help.

Here it is again. The 1.6 machine uses an nfs mount for the prearchive/archive/cache/ftp but uses its ext4 virtual hd (root fs) for the pipeline installation and the xnat installation. The 1.5 machine uses a virtual ext4 hd for all of its operations. I've tested transfers to the nfs mount and I haven't noticed anything strange. It does seem like a file system issue because the moving of an archive in the prearchive is affected as well. I've tried hdparm on both system's root fs and on the system hosting the nfs share and I have not noticed anything. I'm going to try using bonie++ to assess the file system performance next and I'll keep you updated.

[Fletcher Johnson]

Updates. Verbatim update I gave my group. Looks like an nfs issue.

Finally, some major break through has occurred. While running Bonnie++ on the nfs mount on brightstar I noticed it was taking an extremely long time. Checking htop I saw that the process was in state 'D' "uninterruptible sleep" _very_ frequently and as a result was consuming very little cpu. I stopped the test and when I ran the following command this is what I got.

<pre>
[root@brightstar ~]# time ls /rrinid_data/testupload/Bonnie.7245/
...
real    0m21.150s
user    0m0.193s
sys    0m2.180s
</pre>

As part of the test Bonnie++ will create many small empty files in the test directory. In this case 8192 directories. Running the command sometimes takes much less time, no doubt due to caching. When accessing the same directory on the nfs host (ninja) the results are instantaneous (0m0.061s) every time and so we can conclude for now that it is an nfs issue. I believe I didn't notice this before when monitoring Xnat's performance through htop because it was one of the many threads that was in state 'D' and so I missed it. Any ways, that's my excuse. It's also strange that this doesn't register as an IOWait % in either htop or iostat which is another reason why I was mislead.

I suspected that the issue was a file system problem even before I started this test because I confirmed with Ricky's help a week ago that even moving one archive in the prearchive from one project to another took forever. I also had two hints. 1) Xnat's filesystem dir is extensive and has _many_ entries in each dir and 2) The upload that Yasha did that contained many sub directories was significantly slower than his flat file upload. Actually there is a third hint and that is Anda's previous complaints about the file system performance.

[David Gutman]

One thing I've noticed and this is a property of nfs itself I believe...or ext4.. can't remember.     directories with more than several hundred files which is common with dicom cause nonlinear access issues... just try doing an ls in a directory with several thousand files.. and grab some coffee while it loads.

--

[Herrick, Rick]

One of the possible enhancements we’re looking at for future versions of XNAT is to move away from the file repository management business and use a JCR-compliant repository like Apache Jackrabbit or Sling. That should solve some of the file proliferation issues. As well, we should relatively soon (probably as a post-1.6.2 release enhancement, but for sure in 1.7) have the ability to store DICOM sessions in Zip files, meaning that DICOM sessions that may comprise hundreds or even thousands of files would be stored in one single archive.

 

I know this doesn’t help you now, but just to let you know that we are aware of these sorts of issues and have approaches in the works to help mitigate them!

 

From: xnat_di...@googlegroups.com [mailto:xnat_di...@googlegroups.com] On Behalf Of David Gutman

Sent: Thursday, April 18, 2013 5:25 AM
To: xnat_di...@googlegroups.com

[Fletcher Johnson]

Well, it turns out that even though there was some issue with our nfs setup, it wasn't actually the root cause of our c-store slowness. I even tried the 'vboxsf' filesystem that allows direct access to the host's file system. I was getting (faster) file system access but I didn't notice any significant boost in speed for storing dicoms with DicomRemap! Doing a 'du -s -h -c' on the prearchive went from 7 minutes to 2 min, but I only managed to shave 30s off the c-store (3min to 2.5min). So this issue is still up in the air. However, those sound like some good solutions to navigating massive directory trees Rick. Just not sure if that's our problem.