Inability to download large packages
Natalia C
Herrick, Rick
Whenever someone has an issue with downloads terminating after X GB, I usually recommend checking the configuration for the front-end proxy (e.g. nginx, Apache HTTPD, HAProxy, etc.), but at 500 GB I’m really not sure where that might be breaking.
That said, the thing about the desktop client is that it doesn’t download 500 GB, at least not in one chunk. Instead, it downloads a zip for each scan in each session that it’s downloading. If you look at the access logs for a download operation, you’ll see one request where the catalog is downloaded, then multiple requests to URLs like this:
http://server/data/experiments/XNAT_E00001/scans/1/resources/DICOM/files?format=zip
http://server/data/experiments/XNAT_E00001/scans/2/resources/DICOM/files?format=zip
…
http://server/data/experiments/XNAT_E99999/scans/60/resources/DICOM/files?format=zip
Those zip files aren’t written to disk but are instead written directly to the HTTP response stream, so it doesn’t seem like the issue would be running out of cache/work space. You’d see a lot of other problems as well if that were the case. The only thing I can think of off the top of my head is that there’s an issue releasing memory as the executors are churning through the catalogs and they become memory starved around the point where you get 500 GB, but I don’t really think that’s it either if the problem occurs consistently at the same amount of data being downloaded, because it’s likely that sometimes you’d download a bunch of small things and downloading would break at 400 GB and other times you’d download some large things and downloading would stop at 600 GB.
I’ll try profiling memory usage during download operations sometime soon and see if I can see anything that might point to a cause for this.
--
Rick Herrick
Sr. Programmer/Analyst
Neuroinformatics Research Group
Washington University School of Medicine
Phone: +1 (314) 273-1645
From:
xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Natalia C <nato...@gmail.com>
Date: Tuesday, March 22, 2022 at 9:54 AM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] Inability to download large packages
|
* External Email - Caution * |
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
xnat_discussi...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/xnat_discussion/0b48cb1e-19f3-4051-9111-60bc63397d4bn%40googlegroups.com.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.