upload assistant 1.1.2, One item not uploaded: java.io.IOException: too many bytes written
Christopher Grave
Christopher Grave
McKay, Mike
I haven't worked on the upload assistant code, so I'm not sure what is going wrong. One possibility that comes to mind is that maybe it's hitting the original 4 gig cap on zip file size (if our code is using code that enforces this). Another possibility is that your TurbineResources.properties file has this property (services.UploadService.size.max=1048576000), and that property is causing larger uploads to fail. The only other thing that comes to mind is maybe there's some issue with special character encoding like https://github.com/cloudant/java-cloudant/issues/155 . Sorry I couldn't be more helpful, but maybe one of those leads will be useful.
-Mike
Sent: Monday, October 1, 2018 9:16:38 AM
To: xnat_discussion
Subject: [XNAT Discussion] Re: upload assistant 1.1.2, One item not uploaded: java.io.IOException: too many bytes written
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/75e81ef0-43ba-4d37-a382-c2426149d3e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Herrick, Rick
It’s definitely possible that you’re running into the upload limit that Mike mentions. Note that it doesn’t really matter that the study in question is 6.5GB, since the upload assistant actually uploads by series. However if one of the series in your study is larger than the default 1GB limit, that could be the problem. That said, I don’t think the importer is using the Turbine upload service but is actually handling the upload on its own using classes from and based on the Apache Commons FileUpload library. The default for that library is -1, which means that the library itself imposes no size limit, which means that any size limit is imposed either by the application or by Tomcat.
Working from that, I think there are two places you could try to increase the size limit for your upload: the master web.xml for Tomcat in the conf folder of your Tomcat installation or in the web.xml of your XNAT application in the WEB-INF folder of your deployed app. Shut down Tomcat and edit one of these files, adding the following block:
<multipart-config>
<max-file-size>2097152000</max-file-size>
<max-request-size>2097152000</max-request-size>
<file-size-threshold>0</file-size-threshold>
</multipart-config>
You can put this anywhere in the file as long as it’s between the <web-app> and </web-app> elements. Save the file, restart Tomcat, and retry your upload.
Give that a try and see if it fixes your issue.
--
Rick Herrick
Sr. Programmer/Analyst
Neuroinformatics Research Group
Washington University School of Medicine
Phone: +1 (314) 273-1645
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/SN6PR02MB5102BADD2D954C9DD28C230EBFEF0%40SN6PR02MB5102.namprd02.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.
Christopher Grave
Herrick, Rick
Could the limit be in a proxy server in front of Tomcat, something like Apache HTTPD or nginx? For nginx the relevant setting is client_max_body_size and defaults to, I think, 1 or 10 MB. The XNAT Vagrant VM has this set to 0, which removes any size limit at the nginx level. The setting for Apache HTTPD is KeptBodySize and also takes 0 to specify no size limit.
--
Rick Herrick
Sr. Programmer/Analyst
Neuroinformatics Research Group
Washington University School of Medicine
Phone: +1 (314) 273-1645
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/75e81ef0-43ba-4d37-a382-c2426149d3e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/SN6PR02MB5102BADD2D954C9DD28C230EBFEF0%40SN6PR02MB5102.namprd02.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
xnat_discussi...@googlegroups.com.
To post to this group, send email to
xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/c6e9aed4-393f-43fa-9e54-c50b276f561d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Christopher Grave
It is behind a nginx proxy and that was one of my initial thoughts and something I have come across before, but changing that particular setting didn't help. If I recall correctly I think that particular error message manifests itself as a http error. I could be wrong.
Cheers
Chris.
John Flavin
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/df2c95c0-fc98-4eb3-b2a9-4df0c70b2698%40googlegroups.com.
Moore, Charlie
I took a look at the code for the assistant, and I think I agree with Flavin: https://bitbucket.org/xnatdev/upload-assistant/src/9f2d9a6b19afdbb921bb27e4b9e84293617d6b84/src/main/java/org/nrg/xnat/upload/io/dcm/ZipSeriesUploader.java?at=master&fileviewer=file-view-default . In particular, line 72 calls:
connection.setFixedLengthStreamingMode((int) zipSize);
where
final long zipSize = zipFile.length();
So, if the size of the zip for any series exceeds the maximum Java int size (which comes out to about 2.14GB), you’re going to have a problem, I would think. Presumably when you trimmed down the series to about 3GB, the zip compression got the file under the maximum.
Thanks,
Charlie
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/392C8087-0539-41DF-9EEE-92772FD0F43F%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Christopher Grave
Herrick, Rick
The only real way to fix it is to fix the code that’s choking on the long->int conversion. Which as it happens is basically pretty easy so I just did that. The signing certificate we use for the upload assistant has expired and the renewal is held up in bureaucratic entanglements between the certificate issuer and the general university mass. As soon as we get the renewed certificate, I’ll push the code up and get a new development build that you can test out to see if it fixes the problem.
--
Rick Herrick
Sr. Programmer/Analyst
Neuroinformatics Research Group
Washington University School of Medicine
Phone: +1 (314) 273-1645
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/51adf74a-8ef2-40a8-8c0f-9b5adf0ad6f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Noman Ghafoor
Regards