4.5G tar.gz upload -- No space left on device error

1,485 views
Skip to first unread message

landge

unread,
Apr 20, 2017, 3:26:24 AM4/20/17
to xnat_discussion
Hi, 

when trying to upload a 4.5G tar.gz archive to I get an error msg:


Processing of multipart/form-data request failed. No space left on device Your data may be available in the prearchive for your review

There should be plenty of space on our disks! 
Is this error msg referring to the space of the cache defined in the site administration cache path?

Thanks in advance,

landge

Benjamin Sørlie Ormset

unread,
Apr 20, 2017, 7:21:18 AM4/20/17
to xnat_discussion
Hi, guys!

I'm working with landge on this, and would just like to also submit the error produced in the /data/xnat/home/logs/restlet.log file when this occurs.

2017-04-20 10:27:00,991 [http-bio-8080-exec-6] ERROR org.nrg.xnat.restlet.services.Importer -
org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. No space left on device
       at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:351)
       at org.restlet.ext.fileupload.RestletFileUpload.parseRequest(RestletFileUpload.java:104)
       at org.nrg.xnat.restlet.resources.SecureResource.getFileWritersAndLoadParams(SecureResource.java:1128)
       at org.nrg.xnat.restlet.resources.SecureResource.getFileWriters(SecureResource.java:1051)
       at org.nrg.xnat.restlet.services.Importer.handlePost(Importer.java:156)
       at org.restlet.Finder.handle(Finder.java:357)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Router.handle(Router.java:504)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Router.handle(Router.java:504)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:130)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at com.noelios.restlet.ChainHelper.handle(ChainHelper.java:124)
       at com.noelios.restlet.application.ApplicationHelper.handle(ApplicationHelper.java:112)
       at org.restlet.Application.handle(Application.java:341)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Router.handle(Router.java:504)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Router.handle(Router.java:504)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:130)
       at org.restlet.Filter.handle(Filter.java:195)
       at org.restlet.Filter.doHandle(Filter.java:150)
       at org.restlet.Filter.handle(Filter.java:195)
       at com.noelios.restlet.ChainHelper.handle(ChainHelper.java:124)
       at org.restlet.Component.handle(Component.java:673)
       at org.restlet.Server.handle(Server.java:331)
       at com.noelios.restlet.ServerHelper.handle(ServerHelper.java:68)
       at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:147)
       at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:881)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:200)
       at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at org.nrg.xnat.restlet.util.UpdateExpirationCookie.doFilter(UpdateExpirationCookie.java:37)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
       at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
       at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.nrg.xnat.security.XnatInitCheckFilter.doFilter(XnatInitCheckFilter.java:51)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:48)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.nrg.xnat.security.XnatBasicAuthenticationFilter.doFilterInternal(XnatBasicAuthenticationFilter.java:143)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:133)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.nrg.xnat.security.XnatExpiredPasswordFilter.doFilter(XnatExpiredPasswordFilter.java:180)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:152)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
       at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
       at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
       at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
       at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
       at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
       at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
       at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
       at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
       at java.io.FileOutputStream.writeBytes(Native Method)
       at java.io.FileOutputStream.write(FileOutputStream.java:345)
       at org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:129)
       at org.apache.commons.fileupload.util.Streams.copy(Streams.java:107)
       at org.apache.commons.fileupload.util.Streams.copy(Streams.java:70)
       at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:347)
       ... 113 more


Hope this helps!

-- 
Best regards,
Benjamin
 

Herrick, Rick

unread,
Apr 20, 2017, 9:57:17 AM4/20/17
to xnat_di...@googlegroups.com

It’s possible that you ran out of disk space not on the storage where you have the prearchive mounted, but instead on the storage where the Tomcat working folders are mounted. Since you’re transferring data through Tomcat, it actually first stores the incoming multipart form data in a working folder before making it available to the application. This folder is set to something like /tmp/tomcat7- temp in the Tomcat default settings (looking at my current VM, it’s actually using /tmp/tomcat7-tomcat7-tmp, but can be changed by setting JVM_TMP in the Tomcat configuration file (e.g. /etc/default/tomcat7 on Debian-flavored Linux, /etc/tomcat7/tomcat7.conf on RH/Fedora).

 

Some configurations of Tomcat also use Catalina environment variables (I think mostly Debian-based). With these, you can set CATALINA_TMPDIR.

 

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Herrick, Rick

unread,
Apr 20, 2017, 10:02:29 AM4/20/17
to xnat_di...@googlegroups.com

Yeah, that would make sense. The code in there tries to retrieve the request body, in this case the 4.5 GB payload (actually probably more than that with overhead involved in marshalling the data from the multipart form body). As it’s doing that, it’s building the file somewhere in the Tomcat temporary space. It ran out of space there (or at the very least that’s what the OS told the server), with the resulting message passed back up to XNAT.

 

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

 

From: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com> on behalf of Benjamin Sørlie Ormset <benjamin...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

Benjamin Sørlie Ormset

unread,
Apr 21, 2017, 5:32:17 AM4/21/17
to xnat_discussion
Hi, Rick!

Thanks for the quick reply!
We're still having some trouble, unfortunately.

I specified the JVM_TMP path as you mentioned, and put it to a directory with about 200GB free, but we are still experiencing the same errors.
Is it a problem that the path for JVM_TMP is an nfs-mounted area?

Also, we are running tomcat as a different user. Could it be that tomcat will then use this users home area somehow? This is the only place where I can see that there might not be enough space.

-- 
Best regards,
Benjamin

Benjamin Sørlie Ormset

unread,
Apr 21, 2017, 6:23:57 AM4/21/17
to xnat_discussion
Ah, sorry. The home areas are not an issue either, this was fixed yesterday, and now have around 80GB available.
Is there a way for us to get the logs to indicate where it is running out of space?

-- 
Benjamin

Herrick, Rick

unread,
Apr 21, 2017, 11:21:26 AM4/21/17
to xnat_di...@googlegroups.com

You can look at the Tomcat logs and see if there’s more information there. The error occurs inside the Apache Commons file upload library, so XNAT only has the info that’s available in the exception that’s thrown from there. Maybe it logs it elsewhere but from what I can see in the code there’s no information about the resource that caused the error.

 

I just whipped up a simple plugin that we can expand on later, but for right now just basically dumps all of the system properties. You could try installing that and see if the values set for the temp space give you any clue as to what’s going on. The code is available here:

 

https://bitbucket.org/rherrick/xnat-sys-info-plugin

 

An already rolled plugin is available here:

 

https://bitbucket.org/rherrick/xnat-sys-info-plugin/downloads/

 

You can just drop that jar into your plugins folder, restart Tomcat, then go to:

 

http://server/xapi/sysinfo/environment

 

You’ll get a JSON map of all of the properties that we can find in your environment. Note that this REST call is only accessible by system administrators.

landge

unread,
Apr 22, 2017, 6:01:10 AM4/22/17
to xnat_discussion
In case it helps:

There seems to be a upper file size limit of 500MB. Above that size I've never
succeeded to upload anything.....

On Thursday, April 20, 2017 at 9:26:24 AM UTC+2, landge wrote:

landge

unread,
May 5, 2017, 7:07:11 AM5/5/17
to xnat_discussion
Hi,

 after installing the plugin, xnat does not start anymore !?
do you have any ideas why? Since I'm not having access to the log-files I can't 
give you anymore information concerning this issue.... 

thanks,
landge

Benjamin Sørlie Ormset

unread,
May 8, 2017, 10:28:11 AM5/8/17
to xnat_discussion
Sorry, access to all the log-files granted. :)

It looks like we're missing some dependency for the plugin to function properly.

2017-05-05 13:19:42,930 [localhost-startStop-1] WARN  org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanDefinitionStoreException: Failed to parse configuration class [org.nrg.xnat.plugins.sysinfo.plugin.SysInfoPlugin]; nested exception is java.io.FileNotFoundException: class path resource [org/nrg/xapi/rest/AbstractXapiRestController.class] cannot be opened because it does not exist
2017-05-05 13:19:42,937 [localhost-startStop-1] ERROR org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to parse configuration class [org.nrg.xnat.plugins.sysinfo.plugin.SysInfoPlugin]; nested exception is java.io.FileNotFoundException: class path resource [org/nrg/xapi/rest/AbstractXapiRestController.class] cannot be opened because it does not exist
        at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:182)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:321)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:242)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:273)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:98)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:682)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:523)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5116)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: class path resource [org/nrg/xapi/rest/AbstractXapiRestController.class] cannot be opened because it does not exist
        at org.springframework.core.io.ClassPathResource.getInputStream(ClassPathResource.java:172)
        at org.springframework.core.type.classreading.SimpleMetadataReader.<init>(SimpleMetadataReader.java:50)
        at org.springframework.core.type.classreading.SimpleMetadataReaderFactory.getMetadataReader(SimpleMetadataReaderFactory.java:98)
        at org.springframework.core.type.classreading.CachingMetadataReaderFactory.getMetadataReader(CachingMetadataReaderFactory.java:102)
        at org.springframework.core.type.classreading.SimpleMetadataReaderFactory.getMetadataReader(SimpleMetadataReaderFactory.java:93)
        at org.springframework.context.annotation.ConfigurationClassParser.asSourceClass(ConfigurationClassParser.java:641)
        at org.springframework.context.annotation.ConfigurationClassParser$SourceClass.getSuperClass(ConfigurationClassParser.java:805)
        at org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(ConfigurationClassParser.java:306)
        at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:232)
        at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:191)
        at org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(ConfigurationClassParser.java:272)
        at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:232)
        at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:199)
        at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:168)
        ... 22 more


Any tips on how we can fix this?

-- 
Best regards,
Benjamin

Herrick, Rick

unread,
May 8, 2017, 12:26:39 PM5/8/17
to xnat_di...@googlegroups.com

Sorry about that, I built against the XNAT 1.7.3 development code. I modified the source on a branch for 1.7.2 compatibility and pushed a build that should work for you here:

 

https://bitbucket.org/rherrick/xnat-sys-info-plugin/downloads/xnat-sys-info-plugin-1.7.2.jar

 

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

 

From: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com> on behalf of Benjamin Sørlie Ormset <benjamin...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages