Problem in downloading large zip files from XNAT (>1Gb)

48 views
Skip to first unread message

Konstantinos Ntatsis

unread,
Oct 21, 2021, 2:17:22 PM10/21/21
to xnat_discussion
Hi all,

We are storing non-dicom image data at the resource level as zip files (one zip contains multiple sub-files/images).  We have build a simple UI based on xnatpy for our clinical partners to be able to upload and download data from XNAT easily. Only a small proportion of these files reach a size above 1Gb (~1.2Gb), and only for these particular files some of our partners have reported that when they download them, they end up being corrupt and some of the sub-files are missing. This error occurs consistently, every-single time for them.

We also know the following:
- The files are properly uploaded in XNAT because I can download them myself correctly.
- It is not a problem of unzipping because when I send them one of the files downloaded by me with a file transfer website, they can unzip it without any issue.
- The downloaded files by the partners are indeed corrupt because when they send them to me again from the same file transfer website, I can also see that they are corrupt with missing sub-files.
- The problem has come up to now in both windows 10 and ubuntu. For me it works for windows 10, both at work and at home (different computers)
- It is not an issue of xnapy because the XNAT web UI also leads to this error

Any clues/directions that could explain this behavior? Let me know if there is more information that I could provide. Thanks in advance.

Kind regards,
Konstantinos



James Bartolotti

unread,
Nov 3, 2021, 1:59:21 PM11/3/21
to xnat_discussion
I had this exact issue when using the XNAT docker build (https://github.com/NrgXnat/xnat-docker-compose)

It was a problem with a default setting in the Nginx web server that caps downloads at 1Gb (see https://trac.nginx.org/nginx/ticket/1472)

The solution (see https://stackoverflow.com/a/22605784) is to edit your nginx.conf file and add the line "proxy_max_temp_file_size 0;" my file looks like this:

location / {
        proxy_pass                          http://xnat-web:8080;
        proxy_redirect                      off; 
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $remote_addr;
        proxy_set_header X-Forwarded-Host   $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_connect_timeout               150;
        proxy_send_timeout                  100;
        proxy_read_timeout                  100;
        proxy_buffers                       4 32k;
        client_max_body_size                0;
        client_body_buffer_size             128k;
        proxy_max_temp_file_size            0;
    }




 



a.m....@gmail.com

unread,
Nov 12, 2021, 10:57:09 AM11/12/21
to xnat_discussion
Dear James, Konstantino

Did you get this to work on the docker-compose XNAT? I'm using that, and I have just added the "proxy_max_temp_file_size" line to the location/ section of the nginx.conf file in the subdirectory "nginx" of the docker-compose tree. But the downloads are still chopped off at ~ 1Gb.

Are there any other lines that matter? Or is it not possible to change this in the docker-compose version?

bw
Alle Meije

a.m....@gmail.com

unread,
Nov 12, 2021, 11:04:51 AM11/12/21
to xnat_discussion
This is the /location section of my nginx.conf file (there is more before that, it's nested as  http { server { location / { ... }}}

    location / {
        proxy_pass                          http://xnat-web:8080;
        proxy_redirect                      http://xnat-web:8080 $scheme://localhost;

        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $remote_addr;
        proxy_set_header X-Forwarded-Host   $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_connect_timeout               150;
        proxy_send_timeout                  100;
        proxy_read_timeout                  100;
        proxy_buffers                       4 32k;
    proxy_max_temp_file_size        0;
        client_max_body_size                0;
        client_body_buffer_size             128k;
    }

bw
Alle Meije

Herrick, Rick

unread,
Nov 12, 2021, 9:52:15 PM11/12/21
to xnat_di...@googlegroups.com
This is the critical line:

client_max_body_size 0;

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of a.m....@gmail.com <a.m....@gmail.com>
Sent: Friday, November 12, 2021 10:04:51 AM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] Re: Problem in downloading large zip files from XNAT (>1Gb)
 

* External Email - Caution *

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/4c6af5ae-fff9-4bf6-b6bd-a33afc5c8853n%40googlegroups.com.

 


The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Reply all
Reply to author
Forward
0 new messages