[Lift] HTTP POST setting max fileUpload size

801 views
Skip to first unread message

soumik

unread,
May 16, 2010, 4:17:39 AM5/16/10
to Lift
Hi,
How do I configure the maximum file size that can be uploaded for a
HTTP POST method for jetty?
I've implemented a REST API with a POST method for users to post
binary files(~100MB), but when I'm making the POST request from the
client code using Apache Commons HttpPost method I'm getting the
following response from the server :

--------------------------------------------------------------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://
www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html> <body>Exception occured while processing/api/111111/a...@xyz.com/
123/video.mp4/addVideoPost<pre>Message:
org.apache.commons.fileupload.FileUploadBase
$SizeLimitExceededException: the request was rejected because its size
(69162761) exceeds the configured maximum (8388608)

org.apache.commons.fileupload.FileUploadBase
$FileItemIteratorImpl.&lt;init&gt;(FileUploadBase.java:914)

org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:
331)

org.apache.commons.fileupload.servlet.ServletFileUpload.getItemIterator(ServletFileUpload.java:
148)
net.liftweb.http.provider.servlet.HTTPRequestServlet$$anon
$1.&lt;init&gt;(HTTPRequestServlet.scala:110)

net.liftweb.http.provider.servlet.HTTPRequestServlet.extractFiles(HTTPRequestServlet.scala:
100)
net.liftweb.http.Req$$anonfun$3.apply(Req.scala:174)
net.liftweb.http.Req$$anonfun$3.apply(Req.scala:148)
net.liftweb.http.Req.x$17(Req.scala:343)
net.liftweb.http.Req.uploadedFiles(Req.scala:342)
net.liftweb.http.LiftSession.runParams(LiftSession.scala:306)
net.liftweb.http.LiftServlet.liftedTree1$1(LiftServlet.scala:207)
net.liftweb.http.LiftServlet.net$liftweb$http$LiftServlet$
$dispatchStatefulRequest(LiftServlet.scala:205)
net.liftweb.http.LiftServlet$$anonfun$3.apply(LiftServlet.scala:175)
net.liftweb.http.LiftServlet$$anonfun$3.apply(LiftServlet.scala:175)
net.liftweb.http.S$.net$liftweb$http$S$$wrapQuery(S.scala:969)
net.liftweb.http.S$$anonfun$net$liftweb$http$S$$_nest2InnerInit$1$
$anonfun$apply$25.apply(S.scala:1109)
net.liftweb.http.S$.net$liftweb$http$S$$doAround(S.scala:906)
net.liftweb.http.S$$anonfun$net$liftweb$http$S$$_nest2InnerInit
$1.apply(S.scala:1107)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$.net$liftweb$http$S$$_nest2InnerInit(S.scala:1106)
net.liftweb.http.S$$anonfun$net$liftweb$http$S$$_innerInit$1$$anonfun
$apply$28$$anonfun$apply$29$$anonfun$apply$30$$anonfun$apply
$31.apply(S.scala:1132)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$$anonfun$net$liftweb$http$S$$_innerInit$1$$anonfun
$apply$28$$anonfun$apply$29$$anonfun$apply$30.apply(S.scala:1131)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$$anonfun$net$liftweb$http$S$$_innerInit$1$$anonfun
$apply$28$$anonfun$apply$29.apply(S.scala:1130)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$$anonfun$net$liftweb$http$S$$_innerInit$1$$anonfun
$apply$28.apply(S.scala:1129)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$$anonfun$net$liftweb$http$S$$_innerInit
$1.apply(S.scala:1128)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$.net$liftweb$http$S$$_innerInit(S.scala:1127)
net.liftweb.http.S$$anonfun$_init$1$$anonfun$apply$38$$anonfun$apply
$39$$anonfun$apply$40$$anonfun$apply$41$$anonfun$apply
$42.apply(S.scala:1174)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$$anonfun$_init$1$$anonfun$apply$38$$anonfun$apply
$39$$anonfun$apply$40$$anonfun$apply$41.apply(S.scala:1173)
net.liftweb.http.CoreRequestVarHandler$class.apply(Vars.scala:325)
net.liftweb.http.RequestVarHandler$.apply(Vars.scala:248)
net.liftweb.http.S$$anonfun$_init$1$$anonfun$apply$38$$anonfun$apply
$39$$anonfun$apply$40.apply(S.scala:1172)
net.liftweb.http.CoreRequestVarHandler$class.apply(Vars.scala:325)
net.liftweb.http.TransientRequestVarHandler$.apply(Vars.scala:252)
net.liftweb.http.S$$anonfun$_init$1$$anonfun$apply$38$$anonfun$apply
$39.apply(S.scala:1171)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$$anonfun$_init$1$$anonfun$apply$38.apply(S.scala:
1170)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$$anonfun$_init$1.apply(S.scala:1169)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.S$._init(S.scala:1168)
net.liftweb.http.S$.init(S.scala:820)
net.liftweb.http.LiftServlet.doService(LiftServlet.scala:174)
net.liftweb.http.LiftServlet$$anonfun$doIt
$1$1.apply(LiftServlet.scala:73)
net.liftweb.http.LiftServlet$$anonfun$doIt
$1$1.apply(LiftServlet.scala:73)
net.liftweb.util.TimeHelpers$class.calcTime(TimeHelpers.scala:242)
net.liftweb.util.Helpers$.calcTime(Helpers.scala:30)
net.liftweb.util.TimeHelpers$class.logTime(TimeHelpers.scala:251)
net.liftweb.util.Helpers$.logTime(Helpers.scala:30)
net.liftweb.http.LiftServlet.doIt$1(LiftServlet.scala:72)
net.liftweb.http.LiftServlet.service(LiftServlet.scala:80)
net.liftweb.http.provider.HTTPProvider$$anonfun$service
$3.apply(HTTPProvider.scala:55)
net.liftweb.http.provider.HTTPProvider$$anonfun$service
$3.apply(HTTPProvider.scala:55)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.URLRewriter$.doWith(Req.scala:513)
net.liftweb.http.provider.HTTPProvider
$class.service(HTTPProvider.scala:54)
net.liftweb.http.LiftFilter.service(LiftServlet.scala:532)
net.liftweb.http.provider.servlet.ServletFilterProvider
$class.protected$service(ServletFilterProvider.scala:43)
net.liftweb.http.LiftFilter.protected$service(LiftServlet.scala:532)
net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun
$doFilter$1$$anonfun$apply$1.apply(ServletFilterProvider.scala:43)
net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun
$doFilter$1$$anonfun$apply$1.apply(ServletFilterProvider.scala:38)
net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$9$$anonfun$apply
$10$$anonfun$apply$11$$anonfun$apply$12.apply(Vars.scala:331)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$9$$anonfun$apply
$10$$anonfun$apply$11.apply(Vars.scala:330)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$9$$anonfun$apply
$10.apply(Vars.scala:329)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.CoreRequestVarHandler$$anonfun$apply
$9.apply(Vars.scala:328)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.CoreRequestVarHandler$class.apply(Vars.scala:327)
net.liftweb.http.RequestVarHandler$.apply(Vars.scala:248)
net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun
$doFilter$1.apply(ServletFilterProvider.scala:37)
net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun
$doFilter$1.apply(ServletFilterProvider.scala:37)
net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$9$$anonfun$apply
$10$$anonfun$apply$11$$anonfun$apply$12.apply(Vars.scala:331)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$9$$anonfun$apply
$10$$anonfun$apply$11.apply(Vars.scala:330)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$9$$anonfun$apply
$10.apply(Vars.scala:329)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.CoreRequestVarHandler$$anonfun$apply
$9.apply(Vars.scala:328)
net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:67)
net.liftweb.http.CoreRequestVarHandler$class.apply(Vars.scala:327)
net.liftweb.http.TransientRequestVarHandler$.apply(Vars.scala:252)
net.liftweb.http.provider.servlet.ServletFilterProvider
$class.doFilter(ServletFilterProvider.scala:36)
net.liftweb.http.LiftFilter.doFilter(LiftServlet.scala:532)
org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
388)

org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
182)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
765)
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:
230)

org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
114)
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
org.mortbay.jetty.Server.handle(Server.java:326)
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
536)
org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:930)
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)

org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
409)
org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:582)
</pre> </body> </html>

--------------------------------------------------------------------------------------------

I'm using Lift 1.1-M8 with jetty 6.1.24.

Thanks,
Soumik

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

Jean-Luc

unread,
May 16, 2010, 6:54:51 AM5/16/10
to lif...@googlegroups.com
Hi !

You can modify LiftRule's configuration in your Boot class.
Just look at maxMimeSize & maxMimeFileSize in LiftRule's documentation.
  /**
   * The maximum allowed size of a complete mime multi-part POST.  Default
* 8MB
*/
var maxMimeSize: Long = 8 * 1024 * 1024

/**
* The maximum allowed size of a single file in a mime multi-part POST.
* Default 7MB
*/
var maxMimeFileSize: Long = 7 * 1024 * 1024

I couldn't find the post, but I remember that David made a callback which let you monitor file uploads. As you have very big files you may find it interesting.

Jean-Luc




2010/5/16 soumik <sou...@gmail.com>



--
Jean-Luc Canela
jlca...@gmail.com

Marius

unread,
May 16, 2010, 11:57:10 AM5/16/10
to Lift
Right ... you have LiftRules.progressListener or
LiftSession.progressListener. The difference is scoping global vs/ per
session.

However due to reasons of API consistency I believe this needs to
change. We should only have this defined in LiftRules only as
FactoryMaker. This will induce breaking changes but IMO it would
worth. Any thoughts?

Br's,
Marius

On May 16, 1:54 pm, Jean-Luc <jlcane...@gmail.com> wrote:
> Hi !
>
> You can modify LiftRule's configuration in your Boot class.
> Just look at maxMimeSize & maxMimeFileSize in LiftRule's documentation.
>   /**
>
>    * The maximum allowed size of a complete mime multi-part POST.  Default
>    * 8MB
>    */
>   var maxMimeSize: Long = 8 * 1024 * 1024
>
>   /**
>    * The maximum allowed size of a single file in a mime multi-part POST.
>    * Default 7MB
>    */
>   var maxMimeFileSize: Long = 7 * 1024 * 1024
>
> I couldn't find the post, but I remember that David made a callback which
> let you monitor file uploads. As you have very big files you may find it
> interesting.
>
> Jean-Luc
>
> 2010/5/16 soumik <soum...@gmail.com>
>
>
>
> > Hi,
> >  How do I configure the maximum file size that can be uploaded for a
> > HTTP  POST method for jetty?
> > I've implemented a REST API with a POST method for users to post
> > binary files(~100MB), but when I'm making the POST request from the
> > client code using Apache Commons HttpPost method I'm getting the
> > following response from the server :
>
> > --------------------------------------------------------------------------- -----------------
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://
> >www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
> > <html> <body>Exception occured while processing/api/111111/...@xyz.com/
> > 123/video.mp4/addVideoPost<http://...@xyz.com/%0A123/video.mp4/addVideoPost>
> ...
>
> read more »

soumik

unread,
May 16, 2010, 12:15:22 PM5/16/10
to Lift
Thanks Jean-Luc & Marius,
That was very helpful.
However about the progressListener thing, since I'm making HTTP POST
method call from my client snippet service method using Apache HTTP
Components' HttpPost method, I'm not sure how the
LiftRules.progressListener callback on the server will be useful to
show file upload progress at the client end.
> ...
>
> read more »- Hide quoted text -
>
> - Show quoted text -

Marius

unread,
May 16, 2010, 1:35:10 PM5/16/10
to Lift


On May 16, 7:15 pm, soumik <soum...@gmail.com> wrote:
> Thanks Jean-Luc & Marius,
>  That was very helpful.
> However about the progressListener thing, since I'm making HTTP POST
> method call from my client snippet service method using Apache HTTP
> Components' HttpPost method, I'm not sure how the
> LiftRules.progressListener callback on the server will be useful to
> show file upload progress at the client end.

Well, if you are controlling the file upload you can easily do your
progress monitor from client side. Just FYI we support asynchronous
REST using COMET ... please see here:
http://www.assembla.com/wiki/show/liftweb/Asynchronous_REST_API_support
... correlating it with progressListener and actors you could achieve
this ... of course only if it makes sense.One possible usecase would
be if one application is uploading and another one would monitor
upload progress. But you know best what you need.
> ...
>
> read more »

soumik

unread,
May 17, 2010, 2:31:12 AM5/17/10
to Lift
So I modified the LiftRules.maxMimeSize & maxMimeFileSize to 100 MB.
Now with the same POST request I'm getting the following error:
--------------------------------------------------------------------
<pre>Message: java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOf(Arrays.java:2786)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
net.liftweb.util.IoHelpers$class.readOnce$2(IoHelpers.scala:95)
net.liftweb.util.IoHelpers$class.readWholeStream(IoHelpers.scala:99)
net.liftweb.util.Helpers$.readWholeStream(Helpers.scala:30)
net.liftweb.http.LiftRules$$anonfun$45.apply(LiftRules.scala:1101)
net.liftweb.http.LiftRules$$anonfun$45.apply(LiftRules.scala:1100)
net.liftweb.http.provider.servlet.HTTPRequestServlet$$anon
$1.next(HTTPRequestServlet.scala:116)
net.liftweb.http.provider.servlet.HTTPRequestServlet$$anon
$1.next(HTTPRequestServlet.scala:100)
scala.Iterator$class.toList(Iterator.scala:696)
net.liftweb.http.provider.servlet.HTTPRequestServlet$$anon
$1.toList(HTTPRequestServlet.scala:100)

net.liftweb.http.provider.servlet.HTTPRequestServlet.extractFiles(HTTPRequestServlet.scala:
118)
</pre>
------------------------------------------------------------------------------------
I've set the jvm heap size to 1024m by modifying the pom.xml as
follows:
<jvmArgs>
<jvmArg>-Xmx1024m</jvmArg>
</jvmArgs>

Any idea why this error might be coming?

-Soumik

Marius

unread,
May 17, 2010, 2:38:29 AM5/17/10
to Lift
In LiftRules there is the following:

var handleMimeFile: (String, String, String, InputStream) =>
FileParamHolder =
(fieldName, contentType, fileName, inputStream) =>
new InMemFileParamHolder(fieldName, contentType, fileName,
Helpers.readWholeStream(inputStream))

... which keeps the file in memory (by default). You have the option
of using

OnDiskFileParamHolder(name, mime, fileName, inputStream) or even
implement your own FileParamHolder

Br's,
Marius

Chris Hagan

unread,
Aug 22, 2013, 6:03:55 AM8/22/13
to lif...@googlegroups.com
For posterity (because I came across this while googling for a similar problem), one implementation of Marius's suggestion is, in your Boot class:

    LiftRules.handleMimeFile = net.liftweb.http.OnDiskFileParamHolder.apply

This has been tested to correctly buffer so that heap is not exceeded.  It's a good idea to also increase the maximum sizes referenced earlier in the thread, of course.
Reply all
Reply to author
Forward
0 new messages